THE LAB / PARAMETER RECOVERY
You cannot check an estimator against real data, because with real data nobody knows the right answer. So you generate responses from parameters you chose yourself, hide them, and see whether the estimator finds its way back. Error should shrink as the sample grows. If it does not, the problem is the code — and you learn that before you put the instrument in front of anyone.
What this does not prove. Recovering an item’s difficulty here shows only that the arithmetic returns what was put in. It is not evidence about any real listener, it says nothing about whether a degradation is audible, and if real responses do not behave like this model — they will not, exactly — the estimates inherit that mismatch. Recovery bounds implementation error, not model error.
Each dot is one of 40 items. The dashed line is perfect recovery. The cloud should tighten onto it as the cohort grows — and it should stay centred on it, because a cloud sitting parallel to the line would mean bias rather than noise.
20 independent replications per row — a single cohort at n=50 can beat a single cohort at n=1000 on luck alone, so every figure is an average over repeated draws.
| N | REPS | r(p) | RMSE(p) | r(a) | r(θ) | α | TRUE REL | SE(β̄) | BIAS(β̄) |
|---|---|---|---|---|---|---|---|---|---|
| 50 | 20 | 0.838 | 0.056 | 0.252 | 0.827 | 0.673 | 0.674 | 0.037 | -0.067 |
| 200 | 20 | 0.942 | 0.029 | 0.447 | 0.805 | 0.661 | 0.666 | 0.013 | -0.046 |
| 1000 | 20 | 0.987 | 0.014 | 0.627 | 0.816 | 0.667 | 0.673 | 0.006 | -0.057 |
RMSE falls from 0.056 at n=50 to 0.014 at n=1000, with a correlation of 0.987 against the known values. Reliability (α = 0.667) lands within 0.006 of the model’s true reliability. This is the column that says the arithmetic is right.
Per-person error is set by how many trials that person answered, not by how many other people took the test. Sample size buys item statistics and cohort precision; test length buys individual precision. They are different currencies, and the table keeps them in different columns. At a fixed cohort of 400, lengthening the delicacy test from 10 trials to 60 lifts ability recovery from r = 0.62 to r = 0.86 — the same effort spent where it actually works.
SE(β̄) — sampling noise — falls from 0.037 to 0.006 as the cohort grows. BIAS(β̄) does not: it sits at roughly -0.057 points at every sample size, because the prestige instrument systematically understates susceptibility. Anchoring and the scale ceiling both push the same direction. Recruiting ten thousand people would not move it; only a design change would.
Computed at build time by src/analytics/recovery.ts with seed 4001; the item bank is 40 synthetic trials with drawn discrimination and difficulty. These are the same functions the test suite runs, so the figures above and the assertions in src/analytics/recovery.test.ts cannot disagree.
Methodology · The Prestige Test · The Delicacy Trials · The Threshold Test · The Ranking Test