The second milestone in CodeSelf is about execution. If the whole project depends on learning from code feedback, then we need a reliable way to run generated code, collect results, and avoid accidentally giving the model a dangerous amount of freedom.
This is the first milestone where the project starts to feel less like a plan and more like a system. A coding agent can now submit Python code, and the runner can check syntax, reject unsafe patterns, import the candidate, run public tests, and then run hidden tests.
Why Sandboxing Matters for Reinforcement Learning on Code
In many reinforcement learning projects, the environment is a simulator. In this project, the environment is a program executor. That makes the executor part of the learning signal. If it is flaky, the reward is noisy. If it is slow, training becomes expensive. If it is unsafe, the project is not worth running.
Generated code should always be treated as untrusted. Even if the model is not malicious, it can produce code that opens files, starts processes, imports network libraries, or enters infinite loops. A serious coding-agent project needs to handle those cases as normal inputs, not rare surprises.
The executor is part of the research environment: if execution is unreliable, every reward and every future training claim inherits that noise.
The Executor Design
For this milestone, I added two execution paths.
The first path is a dependency-free subprocess runner. It is meant for local development and future training rollouts. It writes candidate code and phase-specific test code into a temporary directory, runs Python with a wall-clock timeout, captures stdout and stderr, and cleans up afterward.
The second path is a Docker command builder for final evaluation. The Docker runner is not required for unit tests yet, but it defines the stricter contract I want later: no network, read-only filesystem, non-root user, CPU limits, memory limits, process limits, and a clean task directory.
This split is intentional. Training needs throughput, so a fresh Docker container for every candidate can become too slow. Final evaluation needs stricter isolation and reproducibility. The project plan already separates those two needs, and now the code does too.
Execution Phases
Each generated solution goes through phases:
- Syntax parsing.
- Static security scan.
- Import smoke test.
- Public tests.
- Hidden tests.
The static scan is conservative. It blocks obvious file, process, and network behavior such as os, subprocess, socket, open, eval, and similar calls. This is not a complete security solution, but it catches many bad candidates before they enter the executor.
The hidden tests are never shown to the model. For now, if public tests fail, hidden tests are skipped by default. That saves execution time and avoids spending hidden-test budget on obviously broken candidates. The runner can be configured differently later if a reward experiment needs full hidden-test fractions for every attempt.
Decisions I Made
I chose not to require Docker for the first passing tests. This keeps the project usable on a normal laptop and lets me iterate on the task schema, reward engine, and runner contract quickly.
I also chose to make result objects explicit. The executor returns phase-level results with status, duration, exit code, stdout, stderr, error text, and test counts. This structure will matter later when rewards need to distinguish syntax errors, timeouts, public-test failures, hidden-test failures, and unsafe code.
Finally, I kept the runner Python-only. Multi-language support would be interesting, but it would multiply the sandboxing and benchmark complexity before the core research question is answered.
The scope is narrow on purpose: one language, explicit result objects, and a fast local path before stricter final-evaluation isolation.
What This Milestone Gives Us
The project can now test generated Python code in a repeatable way. Unit tests cover:
- passing solutions
- public-test failures
- syntax errors
- security rejections
- timeouts
- Docker isolation command construction
This is enough to start building the reward engine in the next milestone. The reward function will consume these execution results and turn them into scalar feedback for GRPO.
What Comes Next
The next step is to connect execution to reward. The first reward should stay simple: correctness-first, with syntax/import success, public tests, hidden tests, robustness tests, and penalties for unsafe or invalid behavior.
Once the reward exists, we can generate base-model rollouts and ask the first serious research question: does the model produce enough partially successful attempts for GRPO to learn from?