Older word vectors sharpen the test for AI’s internal maps

A preprint revised on 6 October adds a revealing comparison to research on AI’s internal maps. Elan Barenholtz finds substantial geographical information in fixed word vectors, an older and simpler form of language representation.[1]

The starting point is a striking result from Wes Gurnee and Max Tegmark’s ICLR 2024 paper. They fed place names into language models and trained a separate mathematical decoder to predict coordinates from the resulting internal activity. Their study covered locations worldwide, in the United States and in New York City, alongside three datasets about time. It found spatial and temporal structure across different kinds of entities, including cities and natural landmarks.[2]

A decoder makes the distinction between the model and the measurement important. In that experiment, researchers supplied coordinates while fitting the decoder, then tested it on held-out names. The model’s internal numbers were the input to the measurement. The map was produced by a learned translation of those numbers into latitude and longitude.[2]

What the words already carry

Word vectors were designed to preserve useful relationships in language. GloVe, introduced in 2014, learns from counts of words appearing near one another. Its authors illustrated the idea with ice and steam: solid is relatively more associated with ice, gas with steam, while water is associated with both. Those differences can be expressed numerically. A vector is a compact collection of such learned relationships, so “simple” does not mean empty of knowledge about the world.[3]

Barenholtz applied comparable decoders to static embeddings on the earlier study’s released place and historical-figure datasets. For multiword names, a basic version simply averaged the component word vectors. The representation stayed fixed across contexts.[1]

Llama-2-7B still outperformed those basic vectors. However, much of its advantage involved identifying the correct country or century; finer ordering within those groups was weaker. Static vectors for individual Wikipedia entities produced similar explained-variance scores to Pythia-2.8B on coordinates and Llama-2-7B on death years, on covered subsets. Similarity on this statistical measure did not imply equal geographical distance errors.[1]

Two different experimental questions

There is a useful everyday distinction here. A library catalogue might let someone reconstruct when books were published and which countries they concern. That arrangement would be valuable evidence about the information in the catalogue. Establishing how a particular reader uses that information requires observing or changing the reader’s behaviour. Recoverability and use invite different experiments.

The original ICLR paper also investigated individual neurons and performed interventions, going beyond its coordinate-decoding result. Its authors explicitly described a comprehensive causal world model as a conjecture. That qualification matters when judging what the later comparison challenges.[2]

Barenholtz’s baseline does not assess causal or behavioural findings, and his largest tested transformer was Llama-2-7B. The narrower contribution is useful: information recoverable from internal activity needs comparison with information already recoverable from word associations.[1]

Report an error or suggest a correction