AI agent skill · by @NeuroAIHub

Parameter Recovery Checker

Guides parameter recovery studies to validate model identifiability before trusting fitted parameter values

Archived 2026-10-23 · Source updated 2026-10-02 · License: AGPL-3.0. Check the original repository for the latest version.

SKILL.md instructions

# Parameter Recovery Checker

## Purpose

This skill encodes expert methodological knowledge for conducting parameter recovery studies -- a critical validation step before interpreting fitted model parameters. Parameter recovery determines whether a model's parameters are identifiable given the experimental design and sample size. A general-purpose programmer unfamiliar with computational modeling would not know that fitting a model is insufficient validation, or how to diagnose parameter tradeoffs and non-identifiability.

## When to Use This Skill

- Before trusting fitted parameter values from any computational cognitive model
- When developing a new model and assessing whether parameters can be distinguished from data
- When planning an experiment and determining the minimum trial count for reliable parameter estimation
- When a reviewer asks for evidence of model identifiability
- When comparing models and needing to ensure each model can be distinguished (model recovery)
- When fitted parameters produce suspiciously extreme values or hit bounds

## Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. **State the research question** -- What specific question is this analysis/paradigm addressing?
2. **Justify the method choice** -- Why is this approach appropriate? What alternatives were considered?
3. **Declare expected outcomes** -- What results would support vs. refute the hypothesis?
4. **Note assumptions and limitations** -- What does this method assume? Where could it mislead?
5. **Present the plan to the user and WAIT for confirmation** before proceeding.

For detailed methodology guidance, see the `research-literacy` skill.


## ⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please [open an issue](https://github.com/HaoxuanLiTHUAI/awesome_cognitive_and_neuroscience_skills/issues).

## Why Parameter Recovery Matters

Fitting a model to data and obtaining parameter estimates does NOT guarantee those estimates are meaningful (Wilson & Collins, 2019; Navarro, 2019). Common failure modes:

1. **Non-identifiability**: Multiple parameter combinations produce identical model predictions (e.g., drift rate and boundary in DDM trade off; Ratcliff & Tuerlinckx, 2002)
2. **Insufficient data**: Too few trials for the fitting procedure to recover true values
3. **Local minima**: Optimization converges to wrong parameter values
4. **Model misspecification**: The fitting procedure recovers parameters that do not reflect the assumed cognitive process

Parameter recovery is the standard diagnostic for these problems (Heathcote et al., 2015; Wilson & Collins, 2019).

## Step-by-Step Recovery Procedure

### Step 1: Define the Parameter Space

Choose ground-truth parameter values that span the plausible range for each parameter.

```
How many parameter sets to simulate?
 |
 +-- Minimum: 100 parameter sets (Wilson & Collins, 2019)
 |
 +-- Recommended: 500-1000 parameter sets for smooth recovery landscapes
 |
 +-- For publication: 1000+ parameter sets (Heathcote et al., 2015)
```

**Sampling strategy:**

| Strategy | When to Use | Source |
|----------|-------------|--------|
| Uniform grid | Few parameters (1-2), want complete coverage | Standard practice |
| Latin hypercube | 3+ parameters, want space-filling without excessive samples | McKay et al., 1979 |
| Random uniform | Simple, adequate for many parameters | Wilson & Collins, 2019 |
| Prior-based sampling | Have informative priors on parameter ranges | Palestro et al., 2018 |

**Range selection**: Use ranges from published parameter estimates in the domain. For example:
- DDM drift rate v: **0.5 -- 4.0** (Ratcliff & McKoon, 2008)
- DDM boundary a: **0.5 -- 2.5** (Ratcliff & McKoon, 2008)
- DDM non-decision time Ter: **0.1 -- 0.5 s** (Ratcliff & McKoon, 2008)
- ACT-R activation noise s: **0.1 -- 0.8** (Anderson, 2007)

### Step 2: Simulate Data

For each ground-truth parameter set:

1. **Match the experimental design exactly** -- Same number of trials, conditions, and structure as the real experiment
2. **Use the same model** -- The generative model must be identical to the model you will fit
3. **Include realistic noise** -- Use the model's noise mechanism (do not add external noise)
4. **Store the ground-truth parameters** for later comparison

**Critical**: The number of simulated trials per participant must match the actual experiment. Recovery with 10,000 trials tells you nothing about recovery with 100 trials (Wilson & Collins, 2019).

### Step 3: Fit the Model to Simulated Data

Apply the **exact same fitting procedure** you use for real data:

- Same optimization algorithm (e.g., MLE, Bayesian, chi-square minimization)
- Same parameter bounds and constraints
- Same starting values or initialization strategy
- Same convergence criteria

**Multiple starting points**: Run the optimizer from at least **5-10** random starting points per simulated dataset to avoid local minima (Heathcote et al., 2015).

### Step 4: Evaluate Recovery Quality

Compare recovered parameters to true (ground-truth) parameters using multiple metrics.

#### Primary Metrics

| Metric | Formula | Good | Acceptable | Concerning | Source |
|--------|---------|------|------------|------------|--------|
| Pearson correlation (r) | cor(true, recovered) | r > **0.9** | r > **0.8** | r < **0.7** | Heathcote et al., 2015; rough benchmarks |
| Bias | mean(recovered - true) | Near **0** | < **10%** of range | > **20%** of range | Wilson & Collins, 2019 |
| RMSE | sqrt(mean((recovered - true)^2)) | Small relative to range | -- | Large relative to range | Standard |
| Coverage | % of 95% CIs containing true value | ~**95%** | **85-100%** | < **80%** | Bayesian recovery |

#### Visualization (essential)

1. **Scatter plot**: Recovered vs. true for each parameter (identity line = perfect recovery)
2. **Bland-Altman plot**: Difference vs. mean (detect range-dependent bias)
3. **Parameter correlation matrix**: Off-diagonal correlations reveal tradeoffs

See `references/recovery-diagnostics.md` for visualization templates.

### Step 5: Check Parameter Tradeoffs

**Correlation between recovered parameters:**

```
Are any pairs of recovered parameters correlated |r| > 0.5?
 |
 +-- YES --> These parameters trade off. Consider:
 | - Fixing one to a theoretically motivated value
 | - Reparameterizing the model
 | - Collecting more data to improve identifiability
 | - Reporting the tradeoff and interpreting cautiously
 |
 +-- NO --> Parameters are identifiable given this design
```

**Common parameter tradeoffs in cognitive models:**

| Model | Correlated Parameters | Nature of Tradeoff | Source |
|-------|----------------------|-------------------|--------|
| DDM | Drift rate (v) and boundary (a) | Speed-accuracy tradeoff | Ratcliff & Tuerlinckx, 2002 |
| DDM | Non-decision time (Ter) and boundary (a) | Boundary absorbs timing variance | Ratcliff & Tuerlinckx, 2002 |
| ACT-R | Noise (s) and threshold (tau) | Both affect retrieval probability | Anderson, 2007 |
| RL models | Learning rate (alpha) and inverse temperature (beta) | Both control exploitation | Daw, 2011 |
| Signal detection | d-prime and criterion (c) | Criterion shift mimics sensitivity change | Macmillan & Creelman, 2005 |

## Model Recovery (Confusion Matrix)

Model recovery extends parameter recovery to test whether the correct model can be identified from data (Wagenmakers et al., 2004).

### Procedure

1. For each candidate model M_k (k = 1, ..., K):
 a. Simulate data from M_k with representative parameters
 b. Fit ALL candidate models to the simulated data
 c. Select the best-fitting model using your comparison metric (AIC, BIC, Bayes factor)
2. Construct a K x K **confusion matrix**: rows = generating model, columns = selected model
3. Diagonal entries should dominate (correct model selected)

### Quality Criteria

| Metric | Good | Concerning | Source |
|--------|------|------------|--------|
| Diagonal proportion | > **90%** correct | < **70%** correct | Wagenmakers et al., 2004 |
| Off-diagonal patterns | Symmetric confusion | Asymmetric (one model always "wins") | Wilson & Collins, 2019 |

**Warning**: If model A is selected when data are generated from model B more than **20%** of the time, those models are not distinguishable with your experimental design (Wilson & Collins, 2019).

## Sample Size Effects

### How Trial Count Affects Recovery

Recovery quality improves with more trials per participant. Test recovery at multiple trial counts:

| Trial Count | Expected Recovery | Recommendation |
|-------------|-------------------|----------------|
| < **50** trials | Often poor (r < 0.7) | Increase trials or simplify model |
| **50-100** trials | Marginal for simple models | May suffice for 2-3 parameter models |
| **100-200** trials | Adequate for most models | Standard for DDM (Ratcliff & McKoon, 2008) |
| **200-500** trials | Good for complex models | Recommended for models with > 4 parameters |
| **500+** trials | Excellent for most models | Required for hierarchical models |

Source: Wilson & Collins (2019); Ratcliff & Tuerlinckx (2002) for DDM-specific guidance.

### Recovery as a Function of N

Plot recovery metrics (r, RMSE) as a function of trial count to determine the minimum viable N for your specific model and paradigm.

## Landscape Analysis

### Parameter Sensitivity Surfaces

For 1-2 key parameters, compute and visualize the objective function surface:

1. Fix all parameters except the target parameter(s)
2. Evaluate the objective function (e.g., negative log-likelihood) at a grid of values
3. Plot the surface (1D: line; 2D: contour or heatmap)

**What to look for:**

| Surface Feature | Interpretation | Action |
|-----------------|---------------|--------|
| Single sharp minimum | Well-identified parameter | Proceed with confidence |
| Broad flat minimum | Parameter poorly constrained | Widen prior or collect more data |
| Multiple minima | Non-convex; local minima risk | Use multiple starting points; consider reparameterization |
| Ridge (elongated valley) | Parameter tradeoff | Two parameters are correlated; consider fixing one |

## Reporting Standards

### Minimum Reporting Checklist

When publishing a parameter recovery study:

- [ ] Number of simulated parameter sets (minimum **100**; Wilson & Collins, 2019)
- [ ] Sampling strategy for ground-truth parameters (uniform, LHS, prior-based)
- [ ] Range of ground-truth values for each parameter (with justification)
- [ ] Number of simulated trials per dataset (must match real experiment)
- [ ] Fitting procedure used (same as for real data)
- [ ] Number of starting points for optimization
- [ ] Recovery metrics for each parameter: correlation (r), bias, RMSE
- [ ] Scatter plots: recovered vs. true for each parameter
- [ ] Parameter correlation matrix (recovered parameters)
- [ ] Model recovery confusion matrix (if performing model comparison)
- [ ] Recovery as a function of trial count (if applicable)

### Where to Report

- **Main text**: Summary of recovery quality (r values, key plots)
- **Supplementary**: Full correlation matrices, all scatter plots, landscape analyses
- Parameter recovery is increasingly expected in top journals (Wilson & Collins, 2019; Navarro, 2019)

## Common Pitfalls

1. **Testing recovery with too many trials**: Simulating 10,000 trials when the experiment has 100. Recovery will look excellent but is irrelevant to your actual data (Wilson & Collins, 2019).
2. **Using different fitting procedures**: The recovery study must use the identical optimization pipeline as the real-data analysis. Different starting values, bounds, or algorithms invalidate the test.
3. **Ignoring parameter correlations**: High marginal recovery (good r for each parameter) can coexist with strong parameter tradeoffs that distort interpretation. Always check the cross-parameter correlation matrix.
4. **Reporting only correlation**: Correlation measures rank-order recovery but ignores systematic bias. A parameter can have r = 0.95 but be consistently overestimated by 30%. Report bias and RMSE alongside r.
5. **Sampling only near defaults**: If ground-truth values cluster around typical defaults, recovery may look good only in that region. Sample across the full plausible range.
6. **Neglecting model recovery**: Good parameter recovery does not guarantee good model recovery. Two models can have recoverable parameters individually but be indistinguishable when competing (Wagenmakers et al., 2004).
7. **Confusing identifiability with validity**: A model can have perfectly recoverable parameters and still be a poor model of cognition. Recovery is necessary but not sufficient (Navarro, 2019).

## References

- Anderson, J. R. (2007). *How Can the Human Mind Occur in the Physical Universe?* Oxford University Press.
- Daw, N. D. (2011). Trial-by-trial data analysis using computational models. In M. R. Delgado, E. A. Phelps, & T. W. Robbins (Eds.), *Decision Making, Affect, and Learning*. Oxford University Press.
- Heathcote, A., Brown, S. D., & Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In B. U. Forstmann & E.-J. Wagenmakers (Eds.), *An Introduction to Model-Based Cognitive Neuroscience*. Springer.
- Macmillan, N. A., & Creelman, C. D. (2005). *Detection Theory: A User's Guide* (2nd ed.). Lawrence Erlbaum Associates.
- McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). A comparison of three methods for selecting values of input variables. *Technometrics*, 21(2), 239-245.
- Navarro, D. J. (2019). Between the devil and the deep blue sea: Tensions between scientific judgement and statistical model selection. *Computational Brain & Behavior*, 2(1), 28-34.
- Palestro, J. J., Sederberg, P. B., Osth, A. F., Van Zandt, T., & Turner, B. M. (2018). Likelihood-free methods for cognitive science. Springer.
- Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: Theory and data for two-choice decision tasks. *Neural Computation*, 20(4), 873-922.
- Ratcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model. *Psychonomic Bulletin & Review*, 9(3), 438-481.
- Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. *Journal of Mathematical Psychology*, 48(1), 28-50.
- Wilson, R. C., & Collins, A. G. (2019). Ten simple rules for the computational modeling of behavioral data. *eLife*, 8, e49547.

See `references/` for diagnostic visualization templates and worked examples.