A feedforward neural network distributes computation across many units and layers. The new result compresses that spatial structure into time. One neuron receives the input and its own previous outputs, emits a sequence of intermediate values, and eventually reproduces the output of the larger circuit. The chain of thought is not an explanation in natural language; it is the workspace in which the missing network is simulated.[1]
For a circuit of size s on n inputs, the construction uses a neuron with state dimension O(ns+s²) and O(s²) generated steps, independent of the original circuit’s depth. For threshold and ReLU activations, the simulation can be exact under the paper’s conditions. For activations such as sigmoid and GELU, it is approximate under decay and Lipschitz assumptions.[1]
One neuron does not mean one number
Where the original network goes
| Feedforward circuit resource | Autoregressive replacement |
|---|---|
| Many neurons and connections | A high-dimensional weight vector and state that encode the circuit and its intermediate values.Source 1 |
| Layer-by-layer parallel structure | A sequence of generation steps that performs pieces of the computation over time.Source 1 |
| Different weights at different circuit locations | One time-invariant neuron uses positional coding so the same rule can retrieve the correct operation at each step.Source 1 |
| Hidden activations inside the network | Intermediate tokens or states in the chain of thought.Source 1 |
| Final output | Read from the generated sequence after the simulation is complete.Source 1 |
The construction uses Sidon sets from additive combinatorics. Their distinctive pairwise differences give computational events unique temporal addresses. In effect, the neuron can recognize which simulated gate and input belong together from where they occur in the generated sequence, while reusing one fixed set of weights at every step.[1]
The theorem strengthens the computational case for chain of thought
Earlier work had shown that autoregressive models could gain substantial computational power from intermediate tokens, but some simulation bounds grew exponentially with circuit depth. The new threshold-neuron result removes that depth dependence, using polynomial state and time. As a consequence, an autoregressive threshold neuron can simulate polynomial-time Turing machines.[1][2]
Theoretical scaling
| Measure | Value | Context |
|---|---|---|
| Input | Circuit of size s on n inputs | The circuit uses the same activation family as the simulating neuron.Source 1 |
| Neuron dimension | O(ns+s²) | The model becomes high-dimensional even though the neuron count is one.Source 1 |
| Generation length | O(s²) | Computation is unfolded into autoregressive time.Source 1 |
| Depth dependence | None in the stated bounds | This improves the prior exponential-in-depth threshold simulation.Source 1Source 2 |
| Tightness | Tight under natural assumptions | The paper provides matching lower-bound reasoning for the relevant resources.Source 1 |
Training changes when the intermediate work is visible
The paper separates learning from final answers and learning from complete chains of thought. When the full intermediate trace is supplied, the autoregressive neuron can be trained efficiently to imitate the simulated computation. When only the final answer is observed, the authors prove hardness results under one-way-function assumptions. The distinction formalizes an intuition familiar from practice: supervision over intermediate work can reveal the algorithm that an endpoint label hides.[1]
The CIFAR-10 experiment tests imitation, not state-of-the-art vision
As an experimental illustration, the researchers trained a single autoregressive ReLU neuron on activation traces from a teacher network working on CIFAR-10. The student agreed with the teacher’s predicted labels on about 99% of test examples. That figure is teacher-student agreement. It is not 99% accuracy on the CIFAR-10 ground-truth labels, and the teacher is described only as having nontrivial test accuracy.[1]
What the result establishes
| Layer | Supported conclusion | Unsupported leap |
|---|---|---|
| Theorem | One autoregressive neuron can simulate broad feedforward circuits with polynomial resources. | A one-parameter scalar unit is as practical as every deep network.Source 1 |
| Learning theory | Complete intermediate supervision can make the simulated computation efficiently learnable. | Any natural-language rationale automatically teaches a correct hidden algorithm.Source 1 |
| Experiment | The trained neuron closely matched one teacher’s predictions in the reported setup. | The neuron independently achieved 99% CIFAR-10 accuracy or general intelligence.Source 1 |
| Interpretation | Time and state can substitute for architectural width and depth. | Spatial neural architecture no longer matters for efficiency, stability or hardware.Source 1 |
The result reframes “model size”. Counting units alone can be misleading when a system has a large state vector, precise weights and a long generated computation. A recurrent program can be small in one dimension and large in another. The theorem’s one neuron contains an encoded circuit and spends quadratic time unpacking it.
That is exactly why the finding is intellectually useful. Chain of thought is often discussed as text added after an answer or as a window into reasoning. Here it is a computational resource: a sequence that lets one fixed predictor emulate a much richer machine. The hidden architecture has not disappeared. It has moved from a graph of layers into the order of time.
Sources & further reading
- The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought arXiv · 12 Oct 2026 · Accessed 16 Oct 2026
- On the Power of Autoregressive Models and Chain-of-Thought Proceedings of Machine Learning Research · 1 Jun 2025 · Accessed 16 Oct 2026