PM-Bench: Evaluating Prospective Memory in LLM Agents

Prospective Memory: A Critical Gap in Agent Reliability

Prospective memory—the ability to remember and execute an intention upon encountering a future cue or state while engaged in ongoing activities—is essential for real-world autonomous agents (e.g., personal assistants, DevOps bots), yet lacks standardized, cognitively grounded evaluation.

PM-Bench Design: A Seven-Day Cognitive Load Simulation

• Built on the established Virtual Week paradigm from cognitive psychology, simulating a text-based week-long environment;
• Requires LLM agents to concurrently manage primary tasks (e.g., email triage, calendar coordination) while dynamically determining whether pre-specified deferred tasks (e.g., ‘remind about meeting when CEO’s email arrives’) are triggered;
• Explicitly measures three core capabilities: intention maintenance, delayed execution, and latent cue monitoring.

Rigorous Empirical Findings: Fundamental Bottlenecks Persist

• Evaluated across eight state-of-the-art LLMs—including GPT-5.4, Claude-3.5-Sonnet, Llama-3.1-405B, and Gemini-2.0-Pro—and eight agent configurations (e.g., ReAct, Reflexion, Plan-and-Execute);
• All settings show substantial failure: top result is GPT-5.4 with adaptive monitoring, achieving only 65.1% F1;
• Key insight: Interventions for improving prospective memory (e.g., explicit timestamp embedding, external memory caching, self-reflection triggers) exhibit strong model- and architecture-specificity—no universally dominant strategy exists.

Open Value: A Controlled Testbed for Diagnosis and Intervention

• PM-Bench is fully open-sourced, including standardized prompt templates, environment simulator, evaluation scripts, and baseline results;
• Designed as a controlled testbed to enable reproducible research into both training-time interventions (e.g., PM-aware instruction tuning) and inference-time mechanisms (e.g., cue-aware planning loops).