Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the robot should reuse that knowledge rather than start each task from scratch. Yet today's robots often treat each task as new: compact scene representations omit interaction-derived knowledge, raw video histories are difficult to query, and VLM planners reason at inference time without persistently updating what the robot knows.
We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks. It maintains a spatially grounded 3D scene graph of objects and locations, augments it with properties and outcomes learned through interaction, and links visual observations for fine-grained recall. We evaluate MessyMem in simulation and on a real mobile manipulator. In a continuous 25-task simulation spanning over 3 hours, MessyMem achieves 80.0% task progress, outperforming the strongest ablation by 14.8 percentage points and the strongest external baseline by 28.9 points, while retrieving task-relevant evidence from thousands of stored keyframes and over an hour into the past.
MessyMem's memory structure allows it to be deployed in diverse real-world environments seen below. See more at Real-World Results.
Put away the coriander
Put away Italian seasoning
Put away the hand sanitizer
Exploration
Get the orzo pasta
Get me a healthy drink
Put the sock away
Find the scissors
Find John’s cup
Find the newly placed game controller
A TidyBot++ with a 6-DoF YAM arm and holonomic mobile base. Two ZED 2 RGB-D cameras, one on the base and one pole-mounted for a higher view, build the scene graph and localize objects in 3D. A wrist-mounted fisheye camera adds keyframe views, and wall-mounted fiducials support localization. Skills are dynamic movement primitives fit from a few teleoperated demonstrations and retargeted to each object with VLM 3D pointing.
Memory for mobile manipulation must maintain three core properties. Prior work delivers them separately. MessyMem combines all three in one persistent store.
Queries must work across rooms, floors, and sessions, in a form compact enough to actually search. Each query returns not only what the robot has seen but where to go to find it, so the planner can navigate directly instead of re-exploring.
Many properties, such as weight, emptiness, articulation, and whether a cabinet is locked, cannot be inferred from passive observation and must be learned by acting. Memory has to record what each interaction reveals, including why an attempt failed.
Text summaries alone are often insufficient. The robot must retrieve visual context on demand: the exact shelf where coffee was last seen, the name printed on a particular cup, or how cluttered one cabinet is compared to another.
Memory is built from RGB-D observations and manipulation outcomes, and used in a closed loop: every planning step retrieves against the latest memory.
the next planning step retrieves against the newly updated memory
SAM3 segments each RGB-D frame, masks are lifted to 3D with depth, and same-label detections merge into persistent object nodes. Each node keeps an id, a 3D centroid, and a property dictionary, and stays in memory after leaving view.
After each manipulation, a VLM reads a multi-frame collage of the action and writes
structured updates to the target node: properties like openable, fixture contents,
and success or failure with a reason. Newer evidence overwrites stale values.
Saves the frames before, after, and most informative in each manipulation, plus a frame whenever the graph changes. Each is linked to its scene-graph nodes with metadata and analyzer labels, and near-duplicates are filtered out.
The retriever pools candidate keyframes from three sources: frames linked to scene-graph nodes, frames whose analyzer labels match the task, and recent interaction frames. Per-source caps keep recently seen objects from crowding out the rest.
Retrieval then runs in two stages: a text-only pass shortlists frames from their metadata, and a multimodal pass inspects thumbnails to keep a small, non-redundant set. The planner combines these with the scene graph and current view.
A walkthrough of the three memory properties, followed by the robot building and querying memory during a task.
All methods share the same planner, perception, and primitives, so differences come from memory alone.
Each trial starts by viewing fixture locations. Their contents stay hidden until the robot interacts.
Tasks arrive one at a time, so what the robot learns early must persist.
Shared across methods, with a budget of twelve planner calls per task.
Task progress and action count, over 50 trials per method (in simulation) with 95% CIs.
Scenarios run in the MuJoCo-based RoboCasa365 simulator, with a Franka Panda arm on an Omron mobile base.
25 consecutive household tasks in a 10-cabinet kitchen with more than 60 objects and fixtures, run for over three hours without resetting memory or the environment. Three cabinets are locked. An initial teapot search encourages exploration; later tasks reuse earlier observations across four task families.
Locate objects using remembered locations, neighboring items, or accessibility.
Example finding a stored mug among countertop distractors
Identify a specified object instance or match a countertop reference using visual details from earlier observations.
Example distinguishing cereal brands, even when Corn Flakes is stored in the fruit cabinet
Select a storage destination for bread, a mug, or a bowl using remembered cabinet contents, without transporting the object.
Example choosing the pantry cabinet for bread based on its dry goods
Recall earlier contents and spatial relationships. These questions can be answered from existing memory without further exploration.
Example recalling which objects were stored with the mayonnaise
First Find bananaThen Find ketchup
MessyMem stores the lock state → goes to cabinet 3.
SG + KF retries the locked cabinet → task failure.
First Find mustardThen Pick & place ketchup
MessyMem compares stored keyframes → picks from cabinet 1.
SG + IA relies on item counts → picks the obstructed bottle.
MessyMem achieves the highest task progress in all three simulated evaluations, and no single memory component suffices on its own.
Mean task progress (%) over 50 trials per method; whiskers show the 95% confidence interval. Bar shade encodes how many of MessyMem's three memory components a method has; external baselines are shown in grey.
| Method | Cluttered Pick | Locked / Unlocked | 25-Task Long Horizon |
|---|---|---|---|
| MessyMem (Ours) | 84.0 [77.0, 90.0] | 99.0 [97.0, 100.0] | 80.0 [77.4, 82.6] |
| SG + IA | 58.0 [53.0, 63.0] | 88.0 [82.0, 94.0] | 65.2 [63.5, 67.0] |
| RoboEXP | 53.0 [48.0, 58.0] | 70.0 [63.0, 76.0] | 51.1 [49.1, 53.1] |
| SG | 49.0 [45.0, 53.0] | 50.0 [50.0, 50.0] | 49.1 [47.4, 51.0] |
| SG + KF | 51.0 [44.0, 58.0] | 43.0 [35.0, 51.0] | 47.9 [45.5, 50.4] |
| MemER-style | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | 1.8 [1.0, 2.5] |
| MemER + Fixtures | 8.0 [3.0, 13.0] | 27.0 [16.0, 39.0] | 0.7 [0.3, 1.2] |
On Cluttered Pick, MessyMem reaches 84% vs 58% for SG + IA and 51% for SG + KF. Without keyframes, the robot knows where ketchup was seen but not which bottle is reachable.
On Locked / Unlocked, MessyMem reaches 99% and SG + IA 88%, vs 43% for SG + KF. A saved image cannot tell a locked cabinet from a failed grasp; the analyzer can.
RoboEXP's predefined relations miss fine-grained appearance and learned state like a locked cabinet. MemER-style memory loses track once fixtures leave view; adding fixture positions only lifts it from 0% to 8% and 27%.
Over three hours of continuous operation, the goal and scene-graph links narrow thousands of keyframes to a small candidate set before visual reranking, so older experience stays reachable as memory grows.
All six paired comparisons are significant after Holm correction (pHolm < 10−4). MemER's recency bias buries older evidence, and RoboEXP's relations cannot tell Froot Loops from a visually similar Trix box.
MessyMem reaches 1.00 task progress in both scenarios, while each ablation fails in its own way.
Three drawer units, one locked. The robot finds scissors, moves to John's cup, then finds a controller newly placed in an unlocked drawer. MessyMem is the only method to complete all three goals, in 10.8 steps with an SPL of 0.92, the closest to the optimal path.
| Method | 1: “Find Scissors” | 2: “Move to John's cup” | 3: “Find placed controller” | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Prog. ↑ | Steps ↓ | SPL ↑ | Prog. ↑ | Steps ↓ | SPL ↑ | Prog. ↑ | Steps ↓ | SPL ↑ | Prog. ↑ | Steps ↓ | SPL ↑ | |
| MessyMem (Ours) | 1.00 | 6.4 | 0.85 | 1.00 | 1.2 | 0.90 | 1.00 | 3.2 | 1.00 | 1.00 | 10.8 | 0.92 |
| SG + IA | 1.00 | 6.6 | 0.90 | 0.60 | 1.4 | 0.60 | 1.00 | 3.0 | 0.96 | 0.87 | 11.0 | 0.82 |
| SG + KF | 0.80 | 7.4 | 0.67 | 0.60 | 3.6 | 0.34 | 0.40 | 6.0 | 0.19 | 0.60 | 17.0 | 0.40 |
| SG | 0.00 | 9.4 | 0.00 | 0.20 | 1.0 | 0.20 | 1.00 | 5.0 | 0.81 | 0.40 | 15.4 | 0.34 |
SPL is success weighted by path length, against human-annotated optimal step counts of 6, 1, and 4 for the three goals. Overall steps are summed across the trial.
Two drawers of socks and a loose sock on a chair. After inspecting both drawers, the robot must put the loose sock with its match, which differs from the others mainly in color or pattern. The match is randomized each trial, so the answer cannot be memorized.
Task progress on Sock Pairing.
@inproceedings{banwasi2026messymem,
title = {MessyMem: Learning-from-Doing Memory for Mobile Manipulation},
author = {Banwasi, Anuva and Muckelroy III, William and Sundaresan, Priya
and Zhao, Linfeng and Bohg, Jeannette and Ho, Cherie},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}This work is supported by the Toyota Research Institute. We thank Marlowe for robust infrastructure for training robot learning policies. Priya Sundaresan is supported by an NSF GRFP. Cherie Ho is supported by the Croucher Postdoctoral Fellowship. Thank you to Satvik Sharma and Francis Engelmann for helpful discussions. We thank Alberta Longhini, Tyler Lum, and Carlota Parés-Morlans for feedback on earlier drafts of the paper.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.