Milestone 9 is about the part of the project that decides whether the final claim is credible. If CodeSelf eventually trains a coding model with GRPO, we need a way to compare the trained candidate against the base model on frozen held-out tasks.
The important word is compare. A single score is not enough. We need paired task outcomes, confidence intervals, and statistical tests that make the uncertainty visible.
Why Final Evaluation Is Different From Baseline Evaluation
Baseline evaluation asks, "How good is this model before training?"
Final evaluation asks, "Did the candidate actually improve over the base model on the same tasks?"
That second question is paired by nature. If both systems are evaluated on the same task set, each task gives us a before-and-after comparison. Some tasks are solved by both, some by neither, some only by the base, and some only by the candidate.
The interesting tasks are the discordant pairs:
- base passes, candidate fails
- base fails, candidate passes
Those are the task flips. They tell us whether the candidate is winning more often than it is regressing.
The final question is paired: the claim is not just whether one score is higher, but whether the candidate wins more task flips than it loses.
McNemar-Style Paired Testing
I added an exact McNemar/binomial test over discordant pairs. The test focuses on the base-only and candidate-only counts.
If the candidate improves many tasks and regresses few tasks, the p-value becomes small. If improvements and regressions are balanced, the p-value stays large. If there are very few discordant tasks, the result may be underpowered even if the pass rate changes a little.
This directly addresses one of the early critiques of the project plan: a small percentage-point gain on a small benchmark can be real but hard to prove.
Bootstrap Confidence Intervals
I also added paired bootstrap confidence intervals over tasks. The bootstrap resamples task outcomes and recomputes the pass-rate delta many times.
The output is a confidence interval in percentage points. This is useful because it shows the range of plausible deltas, not just the point estimate.
For example, a result like +4 pp with a [-2, +10] confidence interval is much weaker than +4 pp with [+1, +7].
What the Comparison Uses
The comparison tool takes two rollout JSONL files:
- base rollout file
- candidate rollout file
It matches common task IDs and compares the same sample index for pass@1. By default, that is sample index 0.
The report includes:
- base
pass@1 - candidate
pass@1 - delta in percentage points
- mean reward delta
- McNemar discordant-pair counts
- exact p-value
- bootstrap confidence interval
- improved task IDs
- regressed task IDs
- per-task outcome table
This makes final evaluation inspectable. If a model improves, we can see where. If it regresses, we can see that too.
Task flips are research evidence: aggregate deltas summarize the result, but improved and regressed task IDs start the qualitative error analysis.
Why This Milestone Comes Before Real Training
It might seem odd to build final evaluation before training a real adapter. But this is the safer order. If the final evaluation tooling exists first, the training process has less opportunity to quietly redefine success.
The project now has a workflow where checkpoint selection happens before final held-out testing, and final held-out testing is handled by a separate paired-comparison script. That separation helps avoid test leakage.
Decisions I Made
I kept the statistical implementation dependency-free. No SciPy is required. The exact McNemar p-value is computed as a two-sided binomial test over discordant pairs, and the bootstrap uses Python's standard random module.
I also made the report include task flips. Aggregate metrics are useful, but task-level flips are where qualitative error analysis starts.
Finally, I kept sample index explicit. The primary pass@1 claim should compare the same attempt number across models. pass@k is useful, but pass@1 is the cleaner paired primary metric.
What Comes Next
The project now has the full evaluation skeleton needed for a future real training run: baseline reports, experiment matrix summaries, checkpoint selection metadata, and final paired statistics.
The next path is either to implement the agentic strategy extension, or to set up real TRL/model integration and begin the headroom audit with an actual open-weight coding model.