A neural network’s performance can change abruptly as its training constraints change. A new paper proposes a common explanation for those jumps, for saddle points encountered during optimisation and for the surprisingly low-error paths that can connect separately trained models. The connection lies in the geometry of the space of possible network weights.[1]
Ibrahim Talha Ersoy, Andrés Fernando Cardozo Licha and Karoline Wiesner published the study in Communications AI & Computing on 21 October. They introduce Pathfinder, a method that uses a shifted L2 regularisation term to explore that geometry systematically. Its purpose is to reveal structure that a normal training run may pass through without mapping.[1]
The terrain is made of possible weight settings
Each configuration of a network’s weights is a point in a high-dimensional space. Its error supplies a height, and training seeks a low region. A saddle has directions that rise and others that fall. L2 regularisation changes the surface being optimised by adding a penalty related to the weights; in this study, the researchers move the penalty’s reference point and vary its strength.[1]
Their analysis links concave boundaries in the error landscape to regularisation-induced saddle points. As the strength of the penalty changes, an optimum can shift across a boundary into a region with different predictive performance. This provides a geometric account of the abrupt transitions observed in the experiments.[1]
Where Pathfinder found transitions
| Test | Observation | Setting |
|---|---|---|
| Sigmoid network on MNIST | Five transitions | Handwritten-digit classification |
| Sigmoid network on Fashion-MNIST | Five transitions | Clothing-image classification |
| ReLU network on MNIST | Five transitions | A different activation function on the digit task |
| Convolutional network on synthetic images | Two transitions | Three noisy pattern classes |
Good solutions can be connected
The principal MNIST experiment also examined five independently trained low-error models. Using one as a reference, Pathfinder traced connections from the others while error remained nearly constant. The models occupied different positions in weight space, yet were joined by low-error routes. This ties the transition picture to mode connectivity: the structure connecting different successful solutions.[1]
The additional experiments give the result breadth across two image datasets, different activation functions and a convolutional architecture. They do not yet establish the same structure in large Transformers. The method also requires repeated optimisation at successive regularisation settings, and its relationship to the path followed by an ordinary stochastic optimiser remains an open research question.[1]
The interpretive opportunity is to connect a change in geometry with a change in what a model can distinguish. In the digit experiments, transitions corresponded to changes in recognition of particular digits or groups. Mapping those boundaries offers researchers a way to study the organisation of learned abilities, alongside the usual question of how low the final error can go.[1]
Sources & further reading
- Phase transitions reveal hierarchical structure in deep neural networks Communications AI & Computing · 21 Oct 2026 · Accessed 25 Oct 2026