Milestone 7 is the first training milestone, but I want to be precise about what was built: this is a GRPO smoke test, not a real fine-tuning run yet.
That distinction matters. It would be easy to import a trainer, point it at a tiny model, and call the project training. But before spending GPU time, I want to know whether the data path itself makes sense. For GRPO, that means checking grouped rollouts, rewards, group-relative advantages, informative prompts, and metrics logging.
Background: What GRPO Needs
GRPO stands for Group Relative Policy Optimization. The key idea for this project is that the model samples multiple completions for the same prompt, each completion receives a reward, and the update compares completions within that group.
This is why group variance matters. If a task has four sampled completions and all four pass, then the rewards are basically identical. There is no useful preference signal. If all four fail, the same problem appears. The useful case is mixed: some completions pass and some fail.
Informative prompts are the early training signal: a group helps GRPO only when sampled completions create a meaningful comparison.
That is the informative prompt idea we added earlier. A prompt is informative when sampled completions include both successes and failures. This milestone makes that concept executable.
What I Built
I added a dependency-free GRPO smoke trainer. It does not update model weights. Instead, it runs the parts of the pipeline that have to work before real training can be trusted:
- Generate grouped rollouts for each task.
- Execute and score each completion.
- Group completions by task ID.
- Compute mean reward and reward standard deviation for each group.
- Compute normalized advantages within each group.
- Track informative-prompt fraction.
- Track reward-variance prompt fraction.
- Write metrics JSONL.
- Write a checkpoint manifest.
The checkpoint manifest explicitly says it is not trained model weights. It is a plumbing checkpoint: proof that the rollout, reward, advantage, logging, and checkpoint-output contracts work together.
Why Not Jump Straight to TRL?
The plan is still to use TRL for real GRPO fine-tuning. But the current environment does not have model weights, GPU setup, or training dependencies installed. More importantly, the early risk is not the trainer API. The early risk is whether our reward and task distribution produce a useful learning signal.
This smoke trainer answers that question cheaply. If the informative-prompt fraction is near zero, real training would likely waste compute. If reward variance collapses, the trainer may run but learn nothing useful. If parser failures are high, the model may be punished for formatting instead of coding.
The smoke test gives us these diagnostics before real training.
Group-Relative Advantages
For each task group, the smoke trainer computes:
- rewards
- mean reward
- reward standard deviation
- normalized advantages
- number of correct samples
- total samples
- whether the group is informative
No variance means no relative signal: when reward standard deviation is zero, the smoke trainer sets all advantages to zero to mirror the practical GRPO issue.
Smoke CLI
I added scripts/train_grpo_smoke.py. It takes a canonical task JSONL file, samples grouped completions with a backend such as mock, computes GRPO diagnostics, and writes:
metrics.jsonllast_rollouts.jsonlcheckpoint_manifest.jsonsummary.json
Running it on the toy task with the mock backend produced mixed completions, nonzero reward variance, and an informative prompt fraction of 1.0. That is a tiny example, but it proves the machinery is connected.
Decisions I Made
I kept this milestone dependency-free. That makes the smoke test runnable anywhere the rest of the project runs. It also keeps the tests fast.
I separated checkpoint manifests from model checkpoints. This avoids pretending that a smoke run produced a trained adapter. The manifest records what was validated and what still remains to be done.
I also kept early stopping based on informative-prompt fraction. If a run produces too few useful prompt groups, the smoke trainer can stop immediately. That is the kind of guardrail we should carry into real training.
What Comes Next
The next major step is real experiment planning and TRL integration. Before that, we need to decide whether to install training dependencies, choose a local model, and run a proper model headroom audit.
The smoke trainer gives us the shape of the training loop. Real GRPO will need to replace the mock generator with a policy model and replace the manifest checkpoint with actual LoRA adapter checkpoints. But the surrounding contracts should stay the same: grouped rollouts, execution reward, advantage diagnostics, evaluation, and honest reporting.