Milestone 6 is about baselines. In a project about a self-improving coding agent, the baseline is not just a number at the beginning of a paper. It is the control condition for the whole project.
If the model later improves after reinforcement learning, we need to know exactly what it improved against. If it does not improve, we need enough instrumentation to understand whether the problem was the model, the reward, the task distribution, the parser, or the evaluation set.
Why Baseline Evaluation Needs Its Own Milestone
It is tempting to treat evaluation as something that happens after training. For this project, that would be backwards. The evaluation harness needs to exist before RL training so that we can freeze the baseline and avoid moving the goalposts.
The baseline should answer several questions:
- How often does generated code pass?
- How often does parsing fail?
- What is the reward distribution?
- Are sampled groups informative for GRPO?
- Is the evaluation set large enough to detect the improvement we hope to show?
Those questions are more useful than a single pass rate.
pass@k
I added the standard pass@k estimator used in code-generation evaluation. The idea is to estimate the probability that at least one of k sampled completions solves a task.
For pass@1, this is close to the ordinary success rate. For larger values of k, it captures whether a model can solve the task with multiple attempts. This matters because code models often improve substantially when allowed to sample several solutions.
The primary metric stays pass@1: it reflects the quality of the first answer, while pass@10 shows whether the model has knowledge but poor reliability.
Informative Prompt Fraction
One metric I added is specific to the GRPO plan: informative prompt fraction.
GRPO compares groups of completions for the same prompt. If all samples pass, there is no useful preference signal. If all samples fail, there is also no useful signal. The useful region is where some samples pass and some fail.
The evaluator now reports the fraction of tasks where sampled completions are mixed. This is a practical early warning metric. If the informative fraction is too low, training may look active while producing almost no meaningful gradient.
Parser and Reward Diagnostics
The evaluator also reports parser failure rate, execution pass rate, and reward statistics. These metrics help separate different failure modes.
For example:
- High parser failure means the generation format or parser needs work.
- High syntax success but low test pass means the model produces runnable but wrong code.
- Collapsed reward distribution means the reward may be too sparse or the task difficulty is wrong.
- Low informative fraction means the model and task pairing may be bad for GRPO.
This is why rollout records are verbose. Evaluation should be able to explain what happened, not only score it.
Baseline reports should be diagnostic: pass rate is useful, but parser status, reward shape, and group-level signal tell us why the number moved.
Power Analysis
I also added a simple power-analysis helper. It estimates a rough minimum detectable effect for pass@1 given a task count and baseline pass rate.
This is not the final statistical test. The final project should use paired tests and bootstrap confidence intervals. But the planning estimate is still useful because small benchmark sets can make small improvements impossible to prove.
For example, if a benchmark has around 164 tasks and the expected improvement is only a few percentage points, the result may be real but statistically weak. It is better to know that before spending GPU time.
What I Built
This milestone added:
pass@kestimator- rollout evaluation summary
- Markdown and JSON report writer
- informative-prompt fraction metric
- approximate minimum detectable effect helper
- evaluation CLI
- baseline report example
- unit tests for the evaluation path
The CLI takes rollout JSONL as input and writes a baseline report. This means the workflow is now:
- Prepare tasks.
- Generate rollouts.
- Evaluate rollouts.
- Inspect baseline metrics.
That is the first complete pre-training loop.
What Comes Next
Milestone 7 is the GRPO smoke test. We are not ready for a large training run yet, but we are ready to scaffold the training interface and run a tiny controlled experiment.
The most important thing will be to keep using the evaluation harness during training. If reward improves but held-out pass@1 does not, that is a warning. If informative prompt fraction collapses, that is also a warning. The baseline evaluator gives us the instruments we need before we start tuning the model.