A new benchmark exposes a critical gap in robotic AI: modern systems can perceive and reason, but they struggle to remember. For developers of generalist and humanoid robots, that gap is becoming the central challenge.
Researchers from the University of Michigan, Stanford University and Figure AI have released RoboMME — the Robotic Memory Manipulation Evaluation benchmark — the first large-scale, standardised framework designed to measure how well robots retain and use information across extended tasks. The results are sobering: even advanced AI systems fail at memory tasks that humans handle with ease.
The benchmark arrives at a pivotal moment. As Vision–Language–Action (VLA) models — which combine visual perception, natural language understanding and motor control — move from research labs into commercial deployment at companies such as Figure AI and Tesla, the field has assumed that perception and reasoning are the binding constraints on progress. RoboMME challenges that assumption directly.
Why Memory Is Different
Most existing robot benchmarks permit policies to rely almost entirely on the current camera frame. If a robot can see the scene, it can act. RoboMME deliberately breaks that assumption. Every task in the benchmark is non-Markovian: the correct action cannot be determined from the current observation alone. The robot must remember.
The researchers ground this design in cognitive science, organising robotic memory into four dimensions drawn from human memory theory:
Temporal memory — tracking event sequences and repetition counts (e.g., how many objects have already been placed).
Spatial memory — maintaining object locations across occlusion and scene change.
Object memory — preserving consistent identity for objects referenced earlier in instructions.
Procedural memory — reproducing previously demonstrated manipulation strategies.
These four dimensions map to four task suites — Counting, Permanence, Reference and Imitation — each comprising four purpose-built tasks. The full benchmark spans 16 tasks, 1,600 demonstrations and approximately 770,000 high-quality training timesteps.
"No single memory representation consistently dominates across all tasks." — RoboMME researchers
The Performance Gap
The gap between human and machine performance is stark. Human participants — assessed via a video question-answering protocol with an oracle motion planner — achieved an overall success rate of 90.5 per cent. The strongest AI policy, FrameSamp+Modul, reached 44.5 per cent. Most systems scored between 17 and 38 per cent.
The hardest tasks were time-sensitive and motion-centric. StopCube — which requires pressing a button at a precise moment during a moving cube's oscillation — defeated nearly every system. Even humans struggled with it, losing count of passes as the video was revealed incrementally. InsertPeg, which demands replicating both the choice of peg and the direction of insertion from a video demonstration, proved similarly resistant.
Recurrent memory approaches — which compress history into a fixed-size neural state — performed worst overall. The researchers attribute this to training instability: shallow recurrent layers grafted onto a pretrained VLA backbone disrupt the original representations without providing sufficient compensating capacity.
What Works, and Where
The benchmark's central finding is that memory effectiveness is highly task-dependent. No single approach generalises across all four task suites.
Perceptual memory — which stores selected visual tokens from past frames — performed best in aggregate, particularly when combined with a modulator integration strategy (adaptive LayerNorm conditioning of the action expert). This combination, FrameSamp+Modul, excelled on motion-centric and time-sensitive tasks because it preserves low-level visual continuity across frames. It also scaled efficiently: performance improved steadily as the memory budget increased, with only modest additional compute.
Symbolic memory — which distils history into natural language subgoals generated by an auxiliary vision-language model — excelled on counting and event-salient tasks where high-level semantic summaries suffice. However, it degraded on tasks requiring fine-grained visuomotor control, and introduced substantial computational overhead: grounded subgoal prediction with a fine-tuned Qwen3-VL-4B model required roughly three times the compute of the base pi-0.5 policy.
MemER, the strongest prior method, combined perceptual keyframe retention with symbolic subgoal generation, achieving 42.4 per cent overall and outperforming all other approaches on tasks with dynamic scene change — specifically the ButtonUnmask family. Its advantage comes from maintaining keyframe images during execution, which preserves visual context that symbolic summaries cannot adequately encode.
Perceptual memory best balances performance and compute efficiency — but no method fully solves any single task suite.
Real-World Transfer
The simulation trends transferred to physical hardware. The researchers tested their strongest symbolic and perceptual policies on a 7-DoF Franka Panda robot across four real-world tasks mirroring the benchmark: fruit counting (analogous to BinFill), cup-tracking with swaps (VideoUnmask/Swap), block re-identification (VideoRepick) and trajectory drawing (PatternLock).
Perceptual memory dominated on DrawPattern — the motion-centric task — while symbolic memory led on the counting task. Both approaches showed room to improve on tracking and re-identification tasks, consistent with simulation results. The physical experiments used 350 demonstrations collected via Oculus Quest 2 teleoperation at 15 Hz.
Implications for the Industry
For robotics developers, the RoboMME findings reframe the product roadmap. The dominant research investment in recent years has targeted perception quality, model scale and data diversity. RoboMME suggests that for real-world deployment — where robots must execute long-horizon tasks in dynamic environments — memory architecture is now an equivalent constraint.
The benchmark also underscores that memory is not a monolithic problem. Counting tasks need different mechanisms than motion imitation; dynamic scene tracking needs different mechanisms than both. The researchers conclude that future high-performance systems will likely require hybrid architectures that integrate multiple memory representations — a direction that MemER already begins to explore, at the cost of significantly higher inference overhead.
RoboMME is available at robomme.github.io, including code, task descriptions and demonstration videos. The paper was published on arXiv in March 2026 (arXiv:2603.04639). Author: Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, Joyce Chai
Leave a comment