Milestone 5 connects the pieces we built so far. We now have task schemas, a sandbox executor, dataset loaders, and a reward engine. The missing link is generation: given a task, render a prompt, ask a model for code, parse the model output, run the code, and store the result.
In reinforcement learning terms, this is the beginning of the rollout pipeline. A rollout is one sampled attempt from the policy model. Later, GRPO will need groups of these attempts for each prompt so it can compare better and worse completions. Before training, we need to make rollout generation boring, inspectable, and reproducible.
Why Parsing Matters
Coding models do not always return exactly what we want. Sometimes they wrap code in Markdown fences. Sometimes they explain the solution before the code. Sometimes they return multiple code blocks. Sometimes the code is syntactically invalid.
If the parser is too brittle, good completions get thrown away. If it is too permissive, invalid or irrelevant text reaches the executor. Both outcomes distort the reward signal.
The parser is part of the reward pipeline: parse failures, syntax failures, and successful extraction need to be logged explicitly so rollout quality can be audited later.
So I added a parser that prefers Python fenced blocks, falls back to plausible unfenced code, and records whether parsing succeeded. It does not silently hide syntax errors. Each rollout stores the parser status and parser error, which will let us compute parser failure rates later.
Prompt Templates
The first prompt templates are simple direct-solution prompts. They ask the model to solve the Python task and return only final code in a fenced block.
There are two versions:
- one that does not include public tests
- one that includes public tests
Hidden tests are never included. That distinction matters because public-test visibility is an experiment choice, while hidden-test leakage would invalidate evaluation.
Generation Backends
I added a backend abstraction instead of hard-coding one model library. Right now there are three backends:
mock: deterministic local completions for testing the pipelinestatic: always returns the same completion, useful for debuggingtransformers: optional local Hugging Face Transformers backend
The Transformers backend uses local_files_only=True, so it is meant for models that already exist on the machine. This keeps the project from unexpectedly downloading weights during tests.
The mock backend is not pretending to be intelligent. Its job is to make the pipeline testable. It produces a mix of passing and failing completions so we can verify parsing, execution, reward scoring, and rollout logging without needing a GPU.
Rollout Records
The rollout JSONL schema now stores:
- task ID
- sample index
- prompt template name
- rendered prompt
- raw completion
- parsed code
- parser status
- generation backend metadata
- execution result
- reward breakdown
- run metadata such as seed and hidden-test inclusion
This format is intentionally verbose. Early in the project, debuggability is more important than compact logs. If reward curves look strange later, these records should let us inspect exactly what happened.
Rollouts are research evidence: storing prompts, completions, parser status, execution results, and rewards makes later claims inspectable instead of just summarized.
CLI Smoke Test
I added scripts/run_rollouts.py, which runs the full loop:
- Load canonical tasks.
- Render prompts.
- Generate completions.
- Parse code.
- Execute public and hidden tests.
- Score reward.
- Write rollout JSONL.
- Print summary metrics.
Using the mock backend on the toy task produced three rollout records with zero parser failures, a mixed pass rate, and a non-collapsed reward distribution. That is exactly what the early GRPO pipeline needs from a smoke test.
Decisions I Made
I chose not to make a real model download part of this milestone. Model setup can become compute- and environment-dependent, while this milestone is about the software contract. The important thing is that the same rollout pipeline can run with a mock backend today and a local model tomorrow.
I also chose to store rendered prompts in rollout logs. This makes files larger, but it prevents ambiguity. Prompt templates are part of the experiment, and if a result matters, we should be able to reconstruct what the model actually saw.
What Comes Next
Milestone 6 is baseline evaluation. Now that rollouts exist, we can implement pass@k, evaluation summaries, parser failure reporting, reward statistics, and eventually the headroom audit.
The key research question is coming into view: before training with GRPO, does the base model produce enough partially successful groups? If all completions pass or all completions fail, GRPO has little to learn from. Rollout logging gives us the data needed to answer that.