When I first wrote down the idea for CodeSelf, the exciting version sounded almost magical: take a coding model, let it practice, score its attempts with tests, and use reinforcement learning so it gets better over time. That is the kind of project title that looks great on a portfolio page, but it also hides a lot of engineering and research traps.
This first milestone is about resisting the urge to start with the flashy part. Before training anything, I want the project to have a structure that makes future claims believable. If I eventually say "the agent improved," I need to know what model it started from, what tasks it trained on, what tasks it never saw, what reward it optimized, and whether the measured gain is larger than noise.
Background: Why This Project Needs Scaffolding First
Reinforcement learning for code is attractive because code has a rare property: a lot of it can be checked automatically. If a model writes a function, we can run unit tests. If it times out, we can penalize that. If it passes public tests but fails hidden edge cases, we can detect overfitting. Compared with open-ended writing tasks, coding gives us a more concrete reward signal.
But that reward signal is only useful if the surrounding system is careful. A weak sandbox can make generated code unsafe to run. A saturated benchmark can make improvement impossible to detect. A vague task schema can make results impossible to reproduce. A reward function with too many clever terms can accidentally teach the model to optimize style metrics instead of correctness.
Milestone 1 is deliberately foundational: create the repository, define the core task schema, set up config conventions, and start a methodology document before touching GRPO.
What I Built in This Milestone
The project now has a basic Python package called codeself. The first concrete piece is a task schema that represents a coding problem in a consistent way:
- task ID
- source benchmark
- natural language prompt
- split name
- entry point
- public tests
- hidden tests
- tags
- resource limits
- metadata
I also added a small in-memory task registry. It can hold task specs, reject duplicate task IDs, filter by split, and round-trip tasks through JSONL. This is not glamorous, but it matters because almost every later component depends on this format: prompt building, execution, reward calculation, training, and evaluation.
The repo also has example configs for:
- model selection and headroom audit
- dataset split manifests
- correctness-first reward
- a future GRPO smoke test
These configs are placeholders for now, but they establish the habit that experiments should be controlled by versioned files instead of scattered command-line choices.
A Key Research Decision: Audit Headroom Before Training
One important decision came from critiquing the initial plan. I should not automatically start from the strongest available instruct model. If the base model already scores near the ceiling on HumanEval or MBPP, then there may be no meaningful room to show improvement. A tiny gain might just be sampling noise.
So the methodology now includes a model headroom audit. Before training, I want to measure:
- how often the model produces parseable code
- greedy pass@1
- sampled pass@1 and pass@k
- per-problem rollout variance
- the fraction of prompts where some samples pass and some fail
That last point is especially important for GRPO. GRPO learns from differences within a group of sampled completions. If all samples for a task pass, there is no useful preference signal. If all samples fail, there is also no useful signal. The sweet spot is the partially solvable region where the model sometimes succeeds and sometimes fails.
This means the project needs to choose the model and training tasks together. The right model is not simply "the strongest model I can run." It is the model that leaves enough room for learning while still being capable enough to produce useful attempts.
Reward Design Decision: Correctness First
The initial project idea included correctness, code quality, and execution efficiency as reward components. That still makes sense as a long-term goal, but for the MVP I am keeping the reward simpler.
The first reward version is correctness-first:
- syntax/import success
- public test pass fraction
- hidden test pass fraction
- robustness test fraction
- penalties for unsafe code, timeouts, flaky behavior, and format failures
Quality and efficiency will be logged as metrics at first, not optimized directly. This is because they can be noisy or gameable, especially on short Python tasks. For example, runtime on tiny MBPP-style functions may be dominated by interpreter overhead instead of algorithm quality. Similarly, optimizing lint scores too early could reward neat-looking wrong code.
The philosophy: measure broadly, optimize narrowly, then add reward shaping only when ablations show it helps.
Why the Blog Series Starts Now
I also added a blog folder because I want this project to be understandable as it grows. Each milestone will get a post written in a technical but readable style. The goal is not just to present final results, but to show the research process: decisions, tradeoffs, failed assumptions, and what changed along the way.
For a project like this, the story matters because the final metric will never explain everything. If the agent improves, I want to explain why the result is credible. If it does not improve, I want the negative result to still be useful.
What Comes Next
The next milestone is sandboxed execution. That is where the project starts to become real: generated code must run safely, deterministically, and fast enough to support training. The executor will need resource limits, timeout handling, stdout/stderr capture, and eventually a distinction between fast training workers and stricter final-evaluation isolation.
Once the executor works, the project can start producing actual rollout data. That will tell us whether the reward has useful variance, whether the model is in the right difficulty band, and whether this idea is ready for the first GRPO smoke test.
For now, Milestone 1 gives CodeSelf a spine. The project has a plan, a package structure, a task format, config conventions, tests, and a methodology document that will grow with the experiments.