Milestone 11 adds a PPO comparison path to CodeSelf. The goal is not to claim that PPO has been fully trained yet. The goal is to make the comparison structure explicit before real model fine-tuning starts.
GRPO is the preferred first algorithm in this project because it avoids a separate value model and works naturally with grouped samples. But PPO is still an important baseline. If the project eventually claims that GRPO is the right choice, the methodology should explain that with evidence, not just intuition.
Why Compare PPO and GRPO?
PPO and GRPO both optimize a policy using rewards, but they organize the learning signal differently.
PPO usually uses a value function to estimate a baseline. The policy update compares sampled rewards against that baseline and uses clipped probability ratios to avoid overly large updates.
GRPO uses groups of completions for the same prompt. Instead of a separate critic, it compares completions relative to the group. This can be attractive for code tasks because we can sample several solutions for the same problem and score them with tests.
In practice, the comparison should ask:
- Which algorithm uses memory more efficiently?
- Which produces better held-out
pass@1? - Which is more stable?
- Which makes better use of sparse execution rewards?
- Which has better throughput under the same rollout budget?
This milestone starts building the measurement layer for those questions.
The comparison should be planned before training: if GRPO wins later, the project should be able to show why under a fair rollout budget.
What I Built
I added a dependency-free PPO smoke trainer. Like the GRPO smoke trainer, it does not update model weights. Instead, it validates the rollout, reward, and metric path.
The PPO smoke trainer computes:
- mean reward
- reward standard deviation
- value baseline
- advantage mean and standard deviation
- mean absolute advantage
- clipped fraction
- policy objective placeholder
- value loss
- execution pass rate
- parser failure rate
Because the smoke runner does not have model log probabilities, it uses deterministic simulated importance ratios. That is enough to test clipping and report plumbing, but it is not a substitute for real PPO.
PPO Versus GRPO Reports
I added a comparison utility that reads the final metrics from GRPO and PPO smoke runs and writes a report table.
The report checks:
- rollout budget match
- mean reward delta
- pass-rate delta
- GRPO informative-prompt fraction
- PPO clipped fraction
This gives the future real-training implementation a report format to fill in with actual trainer outputs.
Smoke-only means smoke-only: the scaffold checks contracts and diagnostics, while real PPO still needs log probabilities, a value model, KL tracking, and checkpoint updates.
Decisions I Made
I kept PPO smoke separate from GRPO smoke. They share the same generation, execution, reward, and evaluation layers, but their diagnostics are different enough to deserve separate trainer modules.
I also made the report explicit that this is smoke-only. Real PPO needs log probabilities, a value head or value model, KL tracking, and actual checkpoint updates.
Finally, I matched rollout budgets in the tests and CLI examples. Algorithm comparisons are not meaningful if one method gets more samples than the other.
What Comes Next
At this point, CodeSelf has scaffolding for all major research components:
- task loading
- sandbox execution
- rewards
- rollouts
- baseline evaluation
- GRPO smoke diagnostics
- experiment orchestration
- final paired statistics
- agentic traces
- PPO comparison scaffolding
The next milestone is the publishable methodology package, unless we switch into real TRL/model integration first. The project is now ready to either become a polished reproducible research repo or start pulling in real models and training dependencies.