An assistant can remember that you like early flights and still book a journey that violates a more important constraint: perhaps you cannot travel on a particular date, or the preference belonged to someone else. Personal AI becomes useful only when it can tell the difference between a remembered fact, a changed circumstance and an instruction that should not be followed automatically.[1]
That distinction is the premise of P-ContextBench, a preprint released on 29 October. Its creators generated 100 synthetic digital lives, embedding 10,088 hidden facts in 18,261 fragments resembling records from mobile apps. They retained 4,810 verified questions spanning straightforward recall through decisions in which an agent must respect a person's current constraints. The benchmark is synthetic and researcher-designed, not a covert audit of real people’s phones.[1]
Five questions, not one memory score
The first category tests recovering a fact from one source; the second combines evidence from more than one. The next categories ask what is true now after circumstances change, what higher-level pattern follows from routines, and whether an agent should comply, warn, clarify or refuse when a request conflicts with the person's constraints. Those last three are closer to what a practical assistant needs to do while arranging appointments, purchases or travel.[1]
The researchers report that GPT-5.4-mini achieved 78.98% overall accuracy on single-source retrieval and 66.22% on multi-source recovery. Its scores fell to 49.05% on temporal tracking, 54.83% on pattern inference and 46.05% on constraint-aware action decisions. Those are task-specific measurements within this benchmark, not general reliability percentages for every deployment of the model.[1]
Selected GPT-5.4-mini results in P-ContextBench
| Test capability | Overall accuracy | Pass across three runs |
|---|---|---|
| Single-source facts | 78.98% | 50.89%Source 1 |
| Across multiple sources | 66.22% | 22.22%Source 1 |
| Current-state tracking | 49.05% | 2.63%Source 1 |
| Higher-level pattern inference | 54.83% | 3.37%Source 1 |
| Constraint-aware actions | 46.05% | 0.85%Source 1 |
The reliability result is more troubling than a single low score. The benchmark also asks whether the same case is answered correctly across three runs. For GPT-5.4-mini, the three-run pass rate was 2.63% for temporal questions, 3.37% for higher-level inference and 0.85% for constraint-aware decisions. Even where an assistant sometimes gets the answer right, a user may not know which attempt to trust.[1]
Why a life story is harder than a profile card
Most product demonstrations put preferences in a neat box: favourite airline, location, dietary restriction, perhaps a calendar. Actual personal context accumulates through scraps of communication. A restaurant preference might be in a chat; an allergy in a health note; a temporary schedule restriction in email; and a later cancellation in a calendar. The assistant needs an evidence trail with dates and scope, not simply the most recent sentence that contains a keyword.[1]
P-ContextBench deliberately constructs fragmented, evolving records rather than a single explicit profile. Its creators also tested whether filler information and decoy material confused performance. The paper reports that increasing the filler-to-evidence ratio from one-to-one to 2.5-to-one had little statistically significant effect on the harder tasks in the experiments they ran. The main difficulty, the authors argue, was not just retrieving a relevant log but reconstructing the right present state or translating it into a safe action.[1]
That suggests useful product design choices. One is to distinguish what the model has seen from what it has confirmed is still true. Another is to carry source provenance and timestamps into decisions. A third is to treat uncertainty as a reason to ask a targeted clarification before a costly or irreversible action. These are design implications of the study, not fixes proven by its authors to eliminate the measured errors.[1]
The researchers used several validation steps: they rejected ungrounded generated test cases, checked whether evidence survived assembly of the synthetic logs and compared automated judging with human annotators on a sample. They report over 95% agreement in that sample. That improves confidence that the cases are answerable, but it does not establish that the synthetic lives fully represent the complexity or privacy stakes of real people.[1]
A realistic product evaluation would add actual longitudinal users who consent to testing, measure what happens after corrections, and separately audit mistakes that could cause harm. The current paper does not provide a customer injury rate, prove that a particular assistant has unsafe access to accounts, or reveal anything about private chats outside the test.[1]
The real advance is conceptual. 'It remembered me' is a weak acceptance test for an agent that can act. A stronger one is: did it recover the right current evidence, recognise which constraints governed this decision, and do so consistently when asked again? That is the threshold personal AI will have to clear before convenience can safely become delegation.[1]
Sources & further reading
- P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior Brown, Bari, Tadikonda and Song, arXiv · 29 Oct 2026 · Accessed 2 Nov 2026