Milestone 12 is less glamorous than training a model, but it is one of the most important parts of the project. The goal is to make CodeSelf understandable and rerunnable by someone who was not sitting next to me while I built it.
In research projects, especially ML systems projects, it is easy to have a lot of impressive-looking scripts and still leave the actual methodology vague. This milestone is about closing that gap.
Why This Milestone Matters
The central claim of CodeSelf is that a coding agent can improve through execution feedback. That kind of claim needs more than a demo. It needs a clear answer to questions like:
- What was the base model?
- What tasks were used for training?
- What tasks were truly held out?
- What reward was optimized?
- How were checkpoints selected?
- How were final results tested statistically?
- Could someone else rerun the same pipeline?
Without those details, improvement curves can become storytelling instead of evidence.
Reproducibility is part of the method: a training curve is only useful if the surrounding protocol explains how it was produced and selected.
What I Built
I rewrote the methodology document so it now reads like a real project protocol instead of a placeholder. It explains the scope of the project, the dataset splits, the reward function, the GRPO and PPO plans, the agentic trace design, the evaluation protocol, and the limitations.
I also added four public-facing documents:
- a reproducibility checklist
- an adapter model card template
- a dataset card template for private coding tasks
- a limitations and safety document
These are templates for now because the project does not yet publish real adapter weights or private datasets. But the templates make the publication requirements explicit before training starts.
The Reproducibility Manifest
The code part of this milestone is a small manifest tool. It scans the public project surface, computes SHA-256 checksums, records environment metadata, lists git status, and writes the core commands needed to reproduce the smoke pipeline.
The command looks like this:
python3 scripts/make_reproducibility_manifest.py \
--output outputs/reports/reproducibility_manifest.json \
--markdown-output outputs/reports/reproducibility_manifest.md \
--archive outputs/reports/codeself_reproducibility.tar.gzThe archive is not meant to contain private hidden tests or model checkpoints. Those need controlled storage. But it does capture the public code, configs, docs, examples, and checksums that a reviewer needs to inspect the project.
A Decision I Am Happy With
I decided to separate smoke reproducibility from real-training reproducibility. The smoke pipeline can run locally with no heavy dependencies. It checks the engineering path: loading tasks, generating rollouts, executing code, scoring rewards, running GRPO/PPO diagnostics, comparing metrics, and producing reports.
Real training will need more metadata: model revision, tokenizer revision, hardware, seeds, adapter hashes, raw rollouts, and final checkpoint references. The documentation now says that directly, which should help prevent accidental overclaiming.
Smoke reproducibility is not real-training reproducibility: the local smoke path proves the contracts, while trained adapters will need model, hardware, seed, and checkpoint metadata.
What Comes Next
Now CodeSelf is ready for the next serious implementation jump: real model integration. The next milestone should connect the existing rollout and reward contracts to a local Transformers or TRL training path. If that works, the project can finally move from smoke diagnostics to actual learning curves.