Can you pick the right architecture before training it?

Can you tell which architecture will solve a problem before training any of them? Amid et al. show that a network's frozen features at random initialization already carry much of the structure a trained model has. If that holds, it should work as an architecture screen. I tested it on a benchmark I had just competed in.

Neural Network Prior Kernel · applied to One Layer Deeper · 7 experiments, 24 documented limitations, one result that survived them.

Take an architecture, make ~48 copies with random weights, and never train them. Push data through all of them, concatenate the outputs, and fit a single linear readout. The score is how accurate that readout is.

The trick is the linear model: it can only take weighted sums of features that already exist, so it cannot invent structure. If it succeeds, the information was already sitting in the random features — put there by the architecture, before any learning.

The testbed is a good one because it admits no partial credit. Given x, a modulus N and a depth T, a model must output the result of applying a hidden operation T times, exactly, on moduli it never saw. Errors compound: a 99%-accurate step is right ~52% of the time after 64 applications. Only an exact rule survives, so identifying the rule works and approximating it does not.

The obvious objection to any positive finding here is that the probe simply likes one design. So I built two program families with an exact complementarity: one can express x² mod N but not x³; the other is the reverse. Each is correct for exactly one task. Changing only the task should flip which one the probe prefers — and it does, with no training anywhere:

LearnedProgramCell is the important row. It mirrors the architecture that actually scored 100% on the competition's public tiers: 3,392 trainable parameters, no frozen program buffer, learning which inputs feed each gate, add-vs-multiply, scales, bias and selector. It discovers the program 1*(x*x) from an initialization containing no correct expert — and trained, it reaches 100% on unseen moduli at every depth through T=64, certifying T=64 on the official M3, M5 and M4 protocols.

A positive result on a task-shaped architecture invites three obvious attacks. Each one became an experiment:

• "It just detects arithmetic-looking designs." I built a control identical in every respect — same 3,392 parameters, same routes, scales, bias, selector, same modular reduction — except its gates may only add, which places x² outside its span. One controlled change moved the probe from 20.7–37.5% down to 0.0–10.8%.
• "Your baselines were undertrained." Four learning rates at 2.4× the step budget. Best unseen-modulus accuracy across all five: 3.7%. They can fit training data (Neural-GPU reaches 88.8%) — they memorise and fail to generalise.
• "You gave your design mod N for free." True, so I gave a genuine neural net exact mod N and friendlier inputs. Still 0.2%. The barrier is precision, not the primitive: landing x² mod N exactly needs the pre-mod value exactly, and x² reaches N² — about 1e-6 relative accuracy at 10-bit moduli and 2e-10 at 16-bit, past where float32 lives.

It is not a general architecture-quality predictor. The designs it separates are arithmetic program families; the neural baselines all fail, but for a precision reason that has nothing to do with the probe. And it is a ranking instrument, not a calibrated score — it measures 26.8% ± 4.5 across seed schedules and swings 1.8% → 45.4% with the number of initializations, though its rank is first at every setting.

Two limits I could not close: the competition's Hard tier was never released, so the interesting case is untestable; and the winning code is private, so every architecture here is a reconstruction. Twenty-four limitations are catalogued in the repo, including three where an earlier version of this write-up was simply wrong and the experiment corrected me.