Recommendation systems are beginning to borrow the machinery of language models. Instead of assigning a score to every film, song or product and then ranking the catalogue, a generative recommender can emit a short sequence of tokens that identifies the next item. Those tokens are called a semantic ID. Items with similar content may share part of the same code, much as related words can share linguistic structure.[1]
The attraction is scale: a model can generate a path through an item-code tree rather than score every catalogue entry. The code alone does not teach all the relationships in user histories. REGER, an arXiv preprint, adds training signals for semantic neighbours, item transitions and relative preference.[1]
The three relationships REGER trains explicitly
| Training component | Relationship it targets | How it is implemented |
|---|---|---|
| Generative semantic retrieval | Which items are similar in content | Given one item code, the model learns to generate codes for semantic neighboursSource 1 |
| Item transition structure attention | Which items tend to follow one another | Global attention is combined with local heads aligned to complete item-code boundariesSource 1 |
| Hierarchical preference distillation | How a user’s preference is distributed across competing items | A discriminative teacher supplies conditional probabilities over legal next tokens in the code treeSource 1 |
Why a semantic code is not enough
Semantic IDs are commonly produced by compressing an item representation into several discrete codebook choices. Shared prefixes can preserve coarse similarity, but the token embeddings and recommender are often initialized without direct knowledge of the original neighbourhood geometry. Training on next-item prediction then asks the system to infer product relationships indirectly from user histories. Sparse items and subtle transitions can be poorly served by that arrangement.[1]
REGER first creates a pretraining task in which one item’s code is used to generate the code of a nearby item in semantic space. This gives the model a reason to treat complete item codes as meaningful before it sees personalized histories. The authors then alter the history encoder so some attention heads can see the full sequence while others concentrate on the current item and its immediate predecessor. The design tries to preserve both long-term context and the local direction of behaviour.[1]
A ranking teacher is translated into token choices
The third component tackles a structural problem in generative retrieval. A conventional teacher model can assign preferences to complete items, whereas the generative model chooses one token at a time. REGER organizes item codes as a prefix tree and decomposes the teacher’s item-level distribution into conditional choices at each branch. Training can then tell the generator not only which final code is correct, but how attractive the alternative branches should be along the way.[1]
The paper derives a weighted decomposition of item-level divergence into prefix-level terms. Its implemented loss supervises prefixes on observed target paths, which the authors explicitly distinguish from the full-tree divergence. The practical aim is still a richer target: training indicates how plausible competing branches are, rather than supplying only one correct token.[1]
The evidence is comparative, not operational
On Amazon Reviews 2014 Sports data, REGER’s Recall@10 was 0.0545, compared with 0.0473 for ActionPiece, the strongest other generative model on that metric in the table. Recall@10 measures how often the held-out next item appears among ten recommendations. REGER did not lead every method on every metric: the discriminative DuoRec teacher scored slightly higher on Beauty Recall@10 and several Toys metrics. The reported results cover six datasets and support offline improvements, not a measured live-service gain.[1]
Offline metrics can reward accurate reconstruction of historical behaviour, including patterns produced by an older recommender. They may not capture novelty, diversity, fairness, long-term satisfaction or the business cost of a more complex training stack. A production test would also need to measure latency, catalogue churn, cold-start performance and whether the teacher model transfers its own biases into the generated token distribution.
REGER’s larger contribution is conceptual. Once a catalogue item is represented as a generated code, relations among complete items cannot be left entirely to token coincidence. The system needs to know what is similar, what tends to follow and what a particular user prefers among viable alternatives. The paper shows one way to put those relations back into the training signal. Whether that improves the recommendations people actually value remains the next test.[1]
Sources & further reading
- REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation arXiv · 27 Oct 2026 · Accessed 30 Oct 2026