Teaching a transformer to write backwards

A decoder-only transformer that reads a normal forward-English question and answers in character-reversed English — emitted tail-first, so the model commits to how the sentence ends before it writes how it begins.

nanochat fork · custom BPE · direction tokens · trained from scratch. A full pretraining + SFT stack built to answer one question: can a transformer learn to generate coherent language tail-first?

Reversing a sentence at the character level is trivial for a two-line Python function and genuinely hard for a person to do fluently out loud. That gap is exactly what makes it a clean probe: it's a deterministic, checkable target with no ambiguity, but it forces the model to represent order in a way normal left-to-right training never asks for.

The property that makes this interesting is tail-first generation. Take the answer "The capital is Paris." Character-reversed, the target string is ".siraP si latipac ehT". The model produces that left-to-right — so the very first token it emits corresponds to the last character of the true answer. It has to decide the ending before the beginning. Normal autoregressive planning runs the other way, so this is a direct test of whether the architecture can plan in reverse.

The modeling was the easy part. The two things that cost real time were both in the data path — which is the honest lesson of most from-scratch training.

• Overlong reversed docs → NaN loss. The dataloader buffers conversations and packs them into fixed-width rows. When every buffered example was longer than the row capacity, the packer emitted an empty/degenerate batch and the loss went to NaN — not a bad learning rate, a data bug masquerading as one. Fix: filter reversed examples that exceed the row capacity before they reach the buffer, so a full buffer always contains something packable.
• macOS killed my checkpoints. The SFT post-step evaluation crashed on macOS (a process-spawn issue), and because the eval ran before the checkpoint save, a crash meant the run exited having written nothing to disk — hours of compute, no artifact. Fix: pass an explicit --save-every=N so checkpointing never sits behind the eval that's crashing.

The only quantitative number is validation bpb 1.2888 — there's no QA-accuracy metric, because nanochat's post-training eval crashes on macOS (the bug above) and I left it disabled. So the honest read is qualitative. Here is a real generation, verbatim:

Watch the two endpoints — this is the whole point of the project. The firsttoken it emitted was ".", the answer's final character. The last token it emitted was "ecnarF" — i.e. France, the first word of the readable answer, correctly echoing the subject of the question. The model wrote the entire sentence back-to-front and only committed to its opening word — the right one — at the very end. Grammatical English, generated tail-first, direction token firing, clean termination.

It is also factually wrong (the capital of France is Paris, not an energy story), with a repetition artifact ("energy, energy"). At 73M params and 125 SFT steps that ceiling is expected — and it's orthogonal to what the experiment tested. The structure is the result: coherent, tail-first English with the correct opening word chosen last. Content quality is a scale knob; whether the mechanism works is the question, and it does.

A reversed model earns its keep when it does something a forward model can't. The directions I'd take it next:

• Reverse model as a verifier. Score candidate answers with tail-first log-probs and see whether reverse-direction scoring reranks differently from — and complements — forward scoring.
• Forward/backward asymmetry. Probe where in the network positional/order information lives, and whether reversed training represents it differently. A small mech-interp study on a model I fully control.
• Infilling / plan-from-the-end. Condition on how a sequence should end and generate backward toward a given start.