Why predicting well lets AI compress information

A good prediction can save storage. If a sender and receiver share a useful model of what a message is likely to contain, the sender can describe it more economically. Information theory gives that intuition a precise basis: more probable messages can receive shorter codes while less probable messages receive longer ones.[1]

Probability becomes a length

The ideal contribution to code length is minus the base-two logarithm of the probability assigned to what actually occurs. For an illustrative symbol assigned probability one quarter, that is two bits; at one sixteenth, it is four. These are information quantities used across an encoded sequence, rather than a claim that each isolated character must occupy exactly that many physical bits.[1]

The condition is crucial: the shorter description comes from assigning probability to the actual data. Being confident about a prediction that turns out wrong can make the code longer. Compression therefore rewards useful expectations, including their treatment of exceptions, rather than confidence by itself.[1]

The receiver reconstructs the original

Arithmetic coding supplies the bridge from probabilities to a reversible bitstream. It repeatedly narrows an interval according to the probability of each symbol encountered, then represents a point within the final interval. More probable sequences leave a wider interval, which can be identified using fewer bits. The decoder follows the corresponding process to recover the symbols.[2]

Both sides must reproduce exactly the same probability distributions from the same context. The receiver is not asked to invent the most plausible message. An unusual name, a typo or an unexpected date still has to return unchanged. The probabilities affect how much information the sender must transmit; the transmitted code selects what really occurred.[2]

This separates the predictor from the coder. A better model can improve compression without changing the basic requirement of exact reconstruction. A broken agreement between encoder and decoder, however, breaks the system even if each model independently writes convincing prose.[2]

The language model has a size too

Language Modeling Is Compression, published at ICLR 2024, tested this connection using neural predictors and arithmetic coding. Its text results show why two different questions need separate answers: how small is the encoded data, and how large is the package when the model needed to decode it is included?[3]

Chinchilla’s smaller data stream came with much larger model storage. The adjusted percentage exceeds 100% because the counted package is bigger than the original input. These historical research results do not rank current compression products.[3]

A shared decoder changes the economics

Compression evaluations must decide what the receiver already possesses. The Hutter Prize rules count program size as well as the compressed archive, demand exact recovery of their test file, and forbid uncounted outside information. A tiny archive cannot claim all the credit if the missing information has simply been hidden in a large program.[4]

Consider a hypothetical two-gigabyte model that reduces files to a tenth of their original size. Using decimal units and ignoring other overhead, compressing 100 megabytes creates ten megabytes of data plus 2,000 megabytes of model: 2,010 megabytes altogether. For 100 gigabytes of data, the same accounting gives ten gigabytes plus two, or twelve gigabytes.[4]

The arithmetic is illustrative, not a measured compression rate. A receiver that already has the model faces a different marginal transfer cost, but the model has not ceased to exist. Reuse can spread a fixed storage cost over more data; it does not erase the resources required to run the decoder.[4]

Does that make compression a measure of intelligence?

A COLM 2024 study examined 31 publicly available base language models and twelve benchmarks covering knowledge and commonsense, coding and mathematical reasoning. Models that assigned shorter descriptions to suitable text corpora generally also performed better on the corresponding tasks. The researchers compared bits per character to accommodate different tokenisers and standardised context windows at 1,900 tokens.[5]

That is evidence of a relationship under specified conditions. The tasks were selected proxies for capability; the models were base models, rather than the full interactive assistants people use. The compression corpora were also chosen to match the capability domains. A result on programming text need not predict equally good performance on every kind of information.[5]

The study’s metric comes from model probabilities; it measures predictive description length rather than practical decoding speed. The finding is interesting because predictive structure can serve more than one purpose. It supports a research tool for studying capability, without establishing that one compression score measures every form of intelligence, understanding or independent judgement.[5]

Keeping the useful information is another problem

Lossless compression preserves the complete original. A task-oriented representation can deliberately discard information instead. The information bottleneck method formalises a trade-off: make a representation of an input compact while preserving information relevant to a specified target. The target matters because it defines what the representation should remain useful for.[6]

Imagine converting delivery orders into a representation for route planning. Addresses and time windows would be relevant to that purpose; decorative fonts might not be. If the task changes to checking whether an order form was altered, visual details could matter again. This original example illustrates why useful compression cannot be defined without asking useful for what.[6]

The approach does not require believing that every neural network follows the information bottleneck principle. It provides a way to state a design problem: which distinctions can be removed while retaining the information needed for the chosen prediction?[6]

Learned image compression makes a related trade-off visible. Ballé and colleagues’ 2018 system transformed images into a compact representation and optimised both the number of bits and reconstruction quality. It was lossy: recovering the encoded representation did not guarantee recovering every original pixel. Different choices of the quality objective could produce different reconstructed appearances.[7]

The system also sent side information describing properties of the representation, helping the decoder assign better probabilities. Those extra bits were worthwhile only if their cost was outweighed by savings elsewhere. The model learned that balance as part of its objective. Even information about how to compress the image had to earn its place in the file.[7]

Prediction, exact recovery and useful simplification therefore answer connected but distinct questions. A powerful compressor needs exploitable structure and honest accounting. A useful task representation also needs a clear decision about which information is allowed to disappear.[7]

Report an error or suggest a correction