Milestone 4 is where CodeSelf starts to look like a reinforcement learning project. The executor from the previous milestone can run generated code and report what happened. Now the project needs to turn those execution results into a scalar reward.
This sounds simple: if the code passes, give it a high reward; if it fails, give it a low reward. But the details matter a lot. A reward function is not just an evaluation metric. It is the thing the model will optimize. If the reward is noisy, gameable, or poorly balanced, training can move in the wrong direction while still looking mathematically busy.
Background: Why Reward Design Is Dangerous
In coding tasks, correctness is the most natural reward source. We can run tests and see whether the generated program behaves correctly. That is one reason code is such an attractive domain for execution-based reinforcement learning.
The trap is adding too much too early. It is tempting to reward code quality, style, type-checking, and runtime efficiency immediately. Those are all useful things to measure, but they can also distort the learning signal.
For example, a lint-friendly wrong answer is still wrong. Runtime is also tricky on small Python tasks because interpreter startup and test overhead can dominate the actual algorithm. If runtime becomes part of the reward too early, the model may learn weird shortcuts instead of better programs.
For the MVP, the reward is correctness-first: measure other behavior, but avoid letting style or speed overpower whether the generated program actually works.
What I Built
The project now has reward_v0_correctness. It scores an execution result using:
- syntax and import success
- public test pass result
- hidden test pass result
- robustness test result, when available
- penalties for syntax errors, unsafe code, timeouts, and runtime errors
The reward engine returns a structured breakdown instead of a single bare float. A reward includes:
- final reward
- score before penalties
- total penalty
- component values
- component weights
- applied penalties
- diagnostic metrics
This is important because later, when training curves move, I want to know why. Did reward improve because more hidden tests passed? Because syntax errors went down? Because timeouts disappeared? A scalar alone cannot answer those questions.
Available-Weight Normalization
One design issue came up immediately: not every task has every type of test. Some tasks may have public and hidden tests but no robustness tests yet. If the robustness weight is always reserved, then a task without robustness tests could never receive a full reward.
For now, the scorer normalizes over available components. Syntax/import, public tests, and hidden tests can receive the full available score when robustness tests are absent. This makes the reward more practical during early dataset work while still leaving a slot for robustness tests later.
Quality and Efficiency as Metrics
I also added lightweight quality and efficiency diagnostics.
Quality metrics include syntax validity, line count, character count, max line length, and a simple quality score. Efficiency metrics include total execution duration, timeout status, and an optional relative runtime score when a reference duration exists.
The important decision is that these are metrics, not optimized reward terms. They will be logged so we can detect regressions and study behavior, but they will not push the model during the first GRPO experiments.
The current philosophy still holds: measure broadly, optimize narrowly, and only add shaping terms after ablations show they help.
Reward Audit Script
I added a small command-line audit script. Given a canonical task JSONL file and a solution file, it runs the solution through the sandbox and prints the full reward breakdown as JSON.
That gives the project a simple debugging loop:
- Pick a task.
- Write or inspect a candidate solution.
- Run the executor.
- Inspect the reward components and penalties.
Before training a model, this kind of manual audit is useful. It helps catch reward surprises while the examples are still small enough to understand.
Tests
The unit tests now cover:
- full reward for a passing solution
- partial credit when public tests fail but syntax/import succeed
- timeout penalties
- quality metrics for invalid syntax
- composite reward diagnostics
- audit script JSON output
This gives us confidence that the reward engine is deterministic and explainable.
What Comes Next
Milestone 5 is base model inference and parsing. That means loading or connecting to a model, generating candidate code, extracting executable Python from model output, and storing rollout logs.
Once generation exists, the reward engine can score real model attempts instead of hand-written toy solutions. That will let us measure the first critical GRPO signal: how often sampled groups contain both successes and failures.