Milestone 8: Planning GRPO Experiments Before Spending GPU

Milestone 8 is about experiment orchestration. The previous milestone gave us a GRPO smoke trainer that can compute group-relative rewards and informative-prompt metrics. This milestone turns that into an experiment matrix.

The project still does not claim real model fine-tuning yet. That would require installing training dependencies, choosing a model, and running on suitable hardware. But before doing that, we need a disciplined way to define ablations, run variants, summarize results, and select checkpoints without accidentally using held-out test results.

Why Orchestration Matters

Reinforcement learning experiments can become messy quickly. Small choices matter:

• random seed
• group size
• KL penalty
• reward version
• curriculum mode
• checkpoint selection metric

If those choices are made ad hoc, the final result becomes hard to trust. A good experiment runner makes the choices explicit and repeatable.

The Experiment Matrix

I added a matrix builder that crosses:

• seeds
• group sizes
• KL betas
• reward names
• curriculum modes

For now, the matrix runs through the dependency-free smoke trainer. That means it is testing the orchestration and metrics path, not updating model weights. But the shape of the output is designed to carry forward to real TRL runs.

Each variant has a stable slug, such as reward_v0_correctness_static_seed-20260601_g-4_kl-0.02_static. That slug becomes the output directory name for metrics, rollout logs, and checkpoint manifests.

Static Versus Informative Curriculum

The project plan called out a key GRPO issue: prompts with no within-group reward variance do not teach much. A model that passes all samples or fails all samples produces almost no relative signal for that prompt.

This milestone adds curriculum mode as an explicit experimental variable:

• static: keep the original task order
• informative: prioritize tasks that are more likely to provide useful learning signal

The current implementation is still a placeholder because we are using toy data and mock generation. But the experiment contract is now present. When real model rollouts exist, informative curriculum can use actual pilot-rollout statistics to prioritize partially solvable tasks.

Checkpoint Selection Without Test Leakage

I added best-run selection utilities. The important rule is that selection uses smoke or dev metrics only, not final held-out test metrics.

One interesting thing happened during the CLI smoke run: the highest mean-reward run had zero informative prompts. That means the toy model passed everything in the final group. High reward sounds good, but for GRPO it is not necessarily a good training candidate because there is no contrast inside the group.

So I added a minimum informative-prompt threshold for selection. With that threshold enabled, the runner selects a learnable mixed group instead of the all-pass run.

What the Runner Writes

The matrix CLI writes:

• per-variant metrics.jsonl
• per-variant last_rollouts.jsonl
• per-variant checkpoint_manifest.json
• global experiment_table.md
• global summary.json
• global selected_checkpoint.json

The selected checkpoint file records the selection policy: dev_or_smoke_metrics_only_no_heldout_test.

This is there to make the future methodology cleaner. If we later evaluate final held-out benchmarks, we can point to the selection rule and show that test results were not used to choose the checkpoint.

Decisions I Made

I kept KL beta in the matrix even though the smoke trainer does not use it for real policy updates yet. This is intentional. KL beta is part of the real GRPO experiment design, so the orchestration layer should already represent it.

I also kept reward names in the matrix. The MVP uses reward_v0_correctness, but later we can add shaped reward variants without changing the experiment runner.

Finally, I chose to write both Markdown and JSON summaries. Markdown is useful for humans; JSON is useful for scripts and future dashboards.

What Comes Next

The next major branch is real model integration. That means installing training dependencies, choosing a model with enough headroom, and running an actual GRPO job with LoRA adapters.

Another path is to build Milestone 9 scaffolding for final held-out evaluation and statistical testing before running expensive training. That would make the evaluation side even stronger before the first real adapter is trained.

Either way, the project now has the spine of an experiment system: tasks, execution, rewards, rollouts, baseline evaluation, GRPO diagnostics, and experiment orchestration.