Milestone 3 is about datasets. That sounds less exciting than reinforcement learning, but for this project it is one of the places where research quality is either protected or quietly lost.
CodeSelf will eventually train on coding tasks and evaluate on held-out tasks. To make that credible, every task needs to flow through one consistent representation. Otherwise the runner, reward engine, trainer, and evaluator all end up making slightly different assumptions about what a "task" is.
Why Dataset Loading Matters
Coding benchmarks are not all shaped the same way. MBPP-style tasks usually have a natural language prompt and a list of assertion tests. HumanEval-style tasks often include a prompt with a function signature, an entry point, and a test function named check. EvalPlus extends benchmarks with stronger tests. Private held-out tasks may use yet another local format.
If each benchmark is handled as a one-off special case, the project becomes hard to debug. Worse, evaluation leakage becomes easier. Hidden tests might accidentally enter prompts. Split decisions might change between runs. Task IDs might collide. A future result might be impossible to reproduce because the exact converted task set was never fingerprinted.
Milestone 3 adds a conversion layer before training: benchmarks can differ at the edges, but CodeSelf should only train and evaluate against one stable task representation.
What I Built
I added dataset loaders for three categories:
- MBPP-style JSON/JSONL files.
- HumanEval/EvalPlus-style JSONL files.
- Canonical CodeSelf task JSONL files.
The loaders produce the same TaskSpec objects used by the executor. That means a task has a task ID, source, prompt, split, entry point, public tests, hidden tests, tags, resource limits, and metadata.
I also added hidden-test sidecars. A sidecar is a separate JSON or JSONL file keyed by task ID. This is useful because sometimes the original benchmark file has public tests, and I want to attach additional hidden tests without editing the source dataset.
Finally, I added deterministic split assignment and dataset fingerprints. The split builder sorts and shuffles task IDs with a seed, assigns train/dev/test splits, and produces stable counts. The fingerprint gives us a hash of the converted task records, which will be important for reproducibility.
Hidden Tests and Leakage Control
One design rule is that hidden tests should never appear in prompts. This sounds obvious, but it is easy to violate accidentally when benchmark records contain tests, examples, prompts, and starter code in mixed fields.
The validator now checks for exact hidden-test leakage into the prompt or starter code. This is not a perfect contamination detector, but it catches a practical class of mistakes early.
For HumanEval-style records, there is another detail: the benchmark test often defines check(candidate) rather than directly calling the solution. The loader wraps that format by appending a call like check(entry_point), so the hidden test snippet can run in the same execution harness as other tasks.
Decisions I Made
I did not download benchmarks inside the project yet. Instead, I built local-file loaders. That keeps the code independent of network access and avoids mixing dataset acquisition with dataset normalization.
I also avoided adding a YAML dependency. The preparation script writes canonical JSONL and JSON manifests using only the Python standard library. This keeps early project setup simple.
The split logic is deterministic but intentionally basic. Later, when we load real benchmarks, we may want tag-balanced splits or difficulty-aware splits. For now, stable seeded splitting is enough to keep the pipeline moving.
The bias is toward reproducibility: local files, canonical records, seeded splits, and fingerprints before any claim-bearing training run.
What This Milestone Enables
The project can now convert benchmark-like files into canonical task JSONL, validate them, apply sidecar hidden tests, and produce split manifests. Unit tests cover MBPP conversion, HumanEval-style conversion, sidecar tests, deterministic splits, fingerprints, and the preparation script.
This gives the next milestone a clean input: execution results can now be connected to a reward function over a stable task format.
What Comes Next
Milestone 4 is the reward engine. It will take phase-level execution results from the sandbox and turn them into scalar feedback. The first version should stay correctness-first: syntax/import success, public tests, hidden tests, robustness tests, and penalties for unsafe code or timeouts.
Once reward exists, we can start generating base-model rollouts and measuring the most important GRPO precondition: whether the model produces enough partially successful groups to learn from.