How an AI can learn by being curious

An AI can learn from exploring before anyone rewards it for completing a particular job. Researchers do this by giving it an internally computed reward for experiences such as discovering an unfamiliar state or improving a prediction. Curiosity describes the training mechanism here. It does not establish that the system feels an urge to know.[1]

Imagine a simulated workshop containing a familiar switch, an unfamiliar lever and a panel that displays random patterns. The agent has time to investigate one. We would like its practice to teach something transferable, such as how the lever moves a platform. Simply counting how much stimulation each object provides would favour a different choice. This is an original illustration, not a reported robot experiment.

Make discovery count as a reward

The Intrinsic Curiosity Module, introduced at ICML 2017, rewards errors in predicting what follows an action. Its prediction concerns learned features of the scene, rather than every pixel. Another component learns to infer the action from the before-and-after observations, helping the representation emphasise changes connected to the agent’s behaviour.[1]

The designer still chooses this objective. The agent then learns which actions earn it, optionally alongside an external task reward. The interesting autonomy lies in selecting experiences without someone scripting every move. It comes from an optimisation process whose ingredients can be inspected.[1]

Surprise can be uninformative

Prediction error has a catch. A 2018 large-scale study added a television-like distractor to a simulated maze: changing its channel produced a randomly selected image. Agents driven by prediction error learned more slowly in its presence. The display remained difficult to predict even after repeated visits, so surprise could continue paying without supplying proportionate knowledge.[2]

This is often called the noisy-TV problem. It also explains the workshop panel’s appeal. Repeated exposure need not make a random next pattern predictable. The study did not show permanent entrapment in every run; with longer training, agents sometimes reached the external goal consistently. The broader study covered 54 environments, including games and simulated control tasks. Its deliberately introduced distractor isolated a weakness that aggregate scores could obscure. The useful lesson is about what the reward measures.[2]

Look for progress in learning

An earlier approach, Intelligent Adaptive Curiosity, tracked how prediction errors changed within different regions of experience. Its 2007 formulation used smoothed error histories to favour situations where learning was progressing. That creates a reason to move beyond something already mastered, and beyond something that remains persistently unlearnable.[3]

This changes what counts as an attractive activity over time. Once the lever’s behaviour is learned, its value for further learning can fall. Curiosity can therefore organise a moving curriculum rather than assign a permanent bonus to one object.[3]

Measure novelty without predicting the next picture

Random Network Distillation uses a different comparison. A fixed, randomly initialised neural network produces a target from the current observation. A second network learns to reproduce it. Their disagreement supplies the reward. Familiar observations tend to become easier for the predictor; an unfamiliar observation can still produce a large error.[4]

The target is deterministic for a given observation. That avoids requiring a predictor to guess an inherently random next frame in order to reduce its error. It does not make all environmental randomness harmless: endlessly different inputs can still be unfamiliar. The method changes the measurement of novelty, rather than proving that every novel experience is useful.[4]

A random target also explains the name. The system is not receiving expertise from a knowledgeable teacher network. It is using a learnable, repeatable target as a way to estimate familiarity.[4]

Ask where competing predictions disagree

The 2019 Disagreement method instead trains several models to predict the consequences of actions. Its reward measures how much their predictions differ. With enough examples of a random process, the models can converge on the same average outcome. Their shared uncertainty about the next sample need not keep generating a disagreement reward.[5]

The distinction is between variability in the world and differences between the models’ current accounts of it. In the workshop, several models might initially disagree about how far the platform will move. An experiment can resolve that disagreement. Once their predictions settle on the same average outcome for the random panel, watching another pattern may teach much less.[5]

That is the method’s rationale, under its modelling assumptions. An ensemble can share a mistaken prediction, and disagreement is still an estimate rather than a direct measurement of everything an agent could learn.[5]

Plan a journey towards useful experience

A promising experiment may require several familiar actions first. Plan2Explore, presented at ICML 2020, learns a model of an environment and imagines possible trajectories through it. It trains an exploration policy using anticipated disagreement in that model, rather than only rewarding the immediately encountered observation.[6]

In our workshop, reaching a new control room might require crossing a corridor whose behaviour is already predictable. Thinking about later opportunities makes those otherwise uninteresting steps worthwhile. This is an illustration of planning for future information, not an environment used in the paper.[6]

The researchers evaluated twenty visual control tasks. After exploration without an external task reward, they supplied a task’s reward function and trained a policy inside the learned model. Their zero-shot evaluation meant no additional interaction with the environment for that task. It still involved computation and policy training; the agent did not acquire a finished new skill instantaneously.[6]

The benefit can be faster learning

A robot-development preprint revised in June 2026 offers a useful recent example. Ten randomly initialised agents controlled a simulated wheeled robot with an arm. They trained on 60 of 180 combinations of verbs, colours and objects, with goal rewards and feedback. Curiosity could cover sensory outcomes and feedback voice. The remaining 120 combinations tested generalisation.[7]

The longer-trained comparison matters. These results support a benefit in learning speed, while showing that much of the gap can close with additional training. The curiosity terms did not establish an ability that the comparison could never acquire.[7]

For the fictional workshop, the meaningful outcome would be a growing ability to use its mechanisms in new combinations. How busy the agent looked would tell us much less. Useful exploration leaves it better prepared for a task that was not spelled out at every step. That is the practical attraction of making learning itself part of the objective.

Report an error or suggest a correction