One Layer Deeper: learning a hidden recurrence, exactly
August 31, 2026
A competition about depth: watch a number get transformed a few times, then reproduce the transformation applied many more times — and on unseen numbers — with zero errors. A tiny learned recurrence swept Easy and every Medium. Hard did not fall, and the reason turned out to be worth writing down.
One Layer Deeper · Tilde Research × Core Automation · learn the one-step rule, reuse it recurrently, certify exact accuracy out to depth 64.
The task
Given a start x, a modulus N, and a step count T, output the result of applying one hidden operation T times. On the public tiers that operation is repeated modular squaring, x → x² mod N. Two demands make it hard: exactness — a depth certifies only if every example at that depth is right — and extrapolation — train on shallow T and small N, get graded on deeper iteration (OOD-T) and moduli you never saw (OOD-N). All under ≤500M params, a fixed H100 clock, and no hard-coded algorithm.
A recurrent Transformer just memorizes: it fits the training prompts with total confidence and sits near random on everything held out (~3–4% test). It never acquires an arithmetic inductive bias. The fix wasn't a bigger network — it was to stop predicting answers and learn the transition itself.
Learn the step, reuse it forever
If a model learns one exact transition and applies it recurrently, depth is free: running to T=64 is the same weights, more loops. The architecture that worked is a population of tiny arithmetic programs. Each expert is a few gates over just {1, x, N}; every gate learns which two inputs to read, add vs. multiply, an integer scale, and a bias. A learned selector concentrates on whichever program fits the labels — trained through final answers only. Nothing names squaring; the optimizer finds it, even from an init where zero experts start correct.
state = x
for step in range(T):
a, b = pick2({1, x=state, N}) # learned routes
gate = a + b if learned_op else a * b
state = (scale · gate + bias) mod N
# a learned selector picks the expert that fits the labelsOne 4,352-parameter file, unchanged, certified the three variable-modulus Mediums to depth 64 on both seen and unseen N:
| dataset | regime | test | max T | OOD-N max T |
|---|---|---|---|---|
| M3 | 11/13/15-bit N | 100% | 64 | 64 |
| M4 | 14/18/22-bit N · 6,074 moduli | 100% | 64 | 64 |
| M5 | 12/14/16-bit N · T=2/4/8 | 100% | 64 | 64 |
Hard, and where the wall was
Hard is a single hidden dataset that may change the recurrence itself — one graded attempt per day, on a rule you never see. Three attempts, and the first two answered the wrong question.
Attempts 1–2 came back 0.03% with a training curve flat on the floor. Easy to read as "wrong rule" — but the population provably contained the exact squaring program and still matched zero examples from step one. That's not a rule miss; it's a broken computation.
So I rebuilt the arithmetic — exact int64 modular multiply (exact to 40-bit), a 14-digit decoder, and a family covering 294/294 candidate quadratics. Re-validated on Medium: still 100%, still certified to 64. The final daily slot, spent on the fixed model, came back 0.03% — flat again, peak accuracy 0.008 over twenty thousand steps.
But now it meant something. With both confounds gone — arithmetic exact, population covering every small-integer ax²+bx+c — the model still matched zero examples. The conclusion isn't a tuning failure: Hard's rule lives outside that family entirely — larger coefficients, higher degree, or nothing polynomial at all. The method that swept ten Medium datasets was, for Hard, the wrong hypothesis class.