AI agent skill · by @yogsoth-ai

Assess Construct Validity

Assess whether a benchmark measures its claimed capability using content, convergent, discriminant, and confound analysis.

Archived 2026-10-23 · Source updated 2026-09-29 · License: Apache-2.0. Check the original repository for the latest version.

SKILL.md instructions

# assess-construct-validity

## Purpose

Assess whether a benchmark or evaluation construct measures the claimed capability rather than content cues, confounds, or unrelated skill.

## Input contract

```yaml
required: [construct_claim, benchmark_specification, evaluation_records]
optional: [content_analysis, convergent_measures, discriminant_measures, confound_hypotheses]
constraints: [each validity judgment requires an observable indicator, comparison basis, and provenance]
```

## Procedure

1. State the target construct and map benchmark tasks, labels, and metrics to its intended components.
2. Check content coverage and plausible construct-irrelevant cues against the benchmark specification.
3. Compare convergent and discriminant evidence where available, preserving missing comparisons.
4. Test confound hypotheses with controlled contrasts or artifact probes and record residual uncertainty.

If construct validity depends on whether the operationalization covers the intended domain, consider `map-coverage-space` as the next tactic.

## Output contract

```yaml
produces: [construct_map, content_validity_assessment, convergent_discriminant_evidence, confound_report, validity_judgment]
delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions]
```

## Quality gates

- Every validity claim cites a task, measure, contrast, or artifact observation.
- Content coverage, convergence, discrimination, and confounds are reported separately.
- A missing diagnostic is marked unresolved rather than treated as evidence of validity.

## Failure and counterexamples

Do not infer construct validity from a high score, face validity, or agreement with another measure that shares the same confound.

## Provenance map

- `resolved: construct-validity-assessment`