Making ctf-lite a real training environment
July 20, 2026
A capture-the-flag eval, turned into an RL training environment — including the null result I kept instead of hiding, the diagnosis, and the fix I built and measured.
Qwen3-4B · GRPO + LoRA · prime-rl · 2×H100. The honest arc matters more than any single number.
The fix, in one picture
Base 4B pass-rate per difficulty rung on encoding-chain. Splitting difficulty across decoupled sub-knobs turned a cliff into a staircase.
Bright bars sit in the learnable band (~0.2–0.8) where GRPO has a gradient. v1 has one; v2 has three.
Act 1 — the null result
Trained a 4B on the tasks, then evaluated it properly — 16 rollouts per task, base vs trained, Fisher's exact test:
| scope | base | trained | Δ | p |
|---|---|---|---|---|
| overall (9 tasks) | 51/144 · .35 | 55/144 · .38 | +.03 | 0.71 |
| trained subset (3) | 35/48 · .73 | 39/48 · .81 | +.08 | 0.47 |
Nothing reaches p<0.05. Two measurement traps produced tempting-but-false positives:
- The harness artifact. Measuring before/after through different agent harnesses invented a gain that was pure instrumentation. Always eval through the same harness the model trained on.
- The small-sample mirage. An n=3 eval reported +0.11; at n=16 it collapsed to noise. An RL delta from a handful of agentic rollouts is not a result.
Act 2 — the diagnosis
GRPO only learns from a task whose rollouts contain both wins and losses — that difference is the gradient. The base 4B's difficulty was bimodal:
| task | base pass | gradient? |
|---|---|---|
| git-leaked-secret | 0.94 | — always solved |
| xor-vault | 0.75 | thin |
| hash-crack | 0.50 | ✓ learnable |
| gauntlet · multi-stage · +3 | 0.00 | — always fails |
Effectively one task carried a usable gradient. The null result was the expected outcome, not a surprise.
Act 3 — the fix
Each task sat at one fixed point on the difficulty axis. So I made difficulty a dial: each task exposes a ladder of rungs, and prime-rl's zero-advantage filter auto-selects the rungs in the model's learnable band. The fix that produced the staircase — split each task's difficulty across decoupled sub-knobs, so every rung is a small step:
| task | difficulty sub-knobs |
|---|---|
| multi-stage | chain length · per-stage crypto · final-seal key length |
| encoding-chain | nesting depth · encoding-menu size (easy→full) |
| repeating-key-xor | XOR key length · known-plaintext crib at easy rungs |
Every rung gold-validates and every encoder chain round-trips, so a real solver can win. A curriculum is one flag: --taskset.difficulties '[1,2,3,4,5]'.
What's honest about it
- The generator concept is validated — measured, not asserted. A fixed-difficulty task can be turned into a trainable ramp, and you can watch the learnable band populate.
- The tasks aren't broken — they're model-matched. multi-stage gold-validates and is solvable; its floor just sits above a 4B's frontier. It's a 7B+ task pointed at a 4B, not a bug.
- This is what a validation gate is for. I measured the difficulty response, so I know which rungs to train on and which to hold out.
- Still to do: the training run on the smoothed curriculum. The environment is now well-posed for it; the "after" number isn't in yet.