Suppose a model is solving a new task and receives a growing stream of examples from neighbouring tasks. The intuitive expectation is monotonic: more relevant experience should help, or at worst do nothing. The preprint “Open-world AI: a Tale of Three Requirements” reports a less comfortable pattern. Some models improved as related data accumulated; others became systematically worse.[1]
The authors argue that open-world improvement depends on three properties appearing together: the system must learn at inference time, preserve a sufficiently rich representation of the incoming tasks and use a predictor capable of disentangling which parts of that representation matter for the present query.[1]
The three requirements
| Requirement | What it prevents | Failure pattern |
|---|---|---|
| Inference-time learning | Treating all new evidence as irrelevant after training | The model cannot incorporate a useful example even when it is shown repeatedly. |
| Rich representation | Compressing different tasks into the same insufficient description | New examples overwrite or blur distinctions needed by the target task. |
| Predictive disentanglement | Using every shared feature as though it has the same meaning everywhere | Transfer imports the wrong rule from a neighbouring task. |
Negative transfer is not the absence of learning
A system can respond to new examples and still move in the wrong direction. Imagine several forecasting tasks that share surface patterns but differ in one hidden causal relationship. If the representation drops that relationship, pooling more data strengthens the average shortcut. The model becomes more confident because it has learned more of the wrong commonality.
This is different from classic catastrophic forgetting, where learning a new task destroys performance on an old one. Here, the danger can appear even without sequential parameter updates. The model may receive more context at inference and use it badly because the context mixes task identity, evidence and prediction in a representation that cannot be cleanly separated.
What the preprint reports
| Measure | Value | Context |
|---|---|---|
| Model scale | Up to 8 billion parameters | Tests included language models across several sizes rather than one toy network. |
| Helpful regime | All three requirements present | Performance improved as related-task data accumulated. |
| Harmful regime | One or more requirements missing | Additional related data could degrade performance rather than merely plateau. |
| Simple repair tested | More scale or layer-type interleaving | The reported experiments did not recover the missing qualitative property through these changes alone. |
Why a larger context window does not solve the problem
A context window increases storage, not understanding. The model can hold more demonstrations while still failing to identify which task generated each one, which variables are stable and which relationships should transfer. Retrieval can make the same mistake at a larger scale if it selects examples that look semantically similar but obey a different rule.
The practical implication is that an open-world system needs a model of provenance and task structure, not just a memory buffer. It should be able to ask: Which environment produced this example? Which latent factors match the current case? What evidence would tell me that transfer is harmful? A model that cannot represent those questions may turn a richer memory into a larger source of interference.
A theory worth testing beyond the benchmark
The authors describe the three requirements as qualitative conditions and test them in deliberately controlled settings. That clarity is useful because it generates falsifiable questions for agents, personalisation systems and scientific assistants. But a preprint result on selected tasks does not establish how often the failure occurs in real deployments, where tools, retrieval, feedback and hidden system prompts interact.[1]
The counterintuitive lesson is precise. Open-world learning is not the ability to remember everything. It is the ability to preserve the right distinctions while deciding what to reuse. Without that, an AI can become worse not because it forgot the new examples, but because it forgot why they were different.
Sources & further reading
- Open-world AI: a Tale of Three Requirements arXiv · 14 Oct 2026 · Accessed 16 Oct 2026