What Should a Robot Remember?

A coding agent that forgets something can go back and read the file again. A robot that forgets where it saw the mug has to go and look. It may have to cross three rooms, open a cupboard, and move things to do it. And if someone has tidied up since, the information is gone for good.

That is why memory matters more for embodied agents than for the chat and coding agents most memory systems were built for. It is also why nobody had a clean way to compare the options. Large language model agents usually store their past as text. A robot has more choices: it can keep the camera frames it saw, write descriptions of them, record a structured map of objects and their relations, or combine these. Each choice decides what survives, and each costs a different amount of model inference to build and to use.

A new paper from Dexmate and Carnegie Mellon, What Should Embodied Agents Remember?, introduces MeMoREval, a benchmark built to answer exactly that question. The code and tasks are on the project page.

Experience First, Questions Later

The benchmark works in two phases. In the experience phase, the agent carries out a sequence of ordinary tasks without knowing what it will be asked afterwards. It writes to memory as it goes. In the request phase, it receives new instructions and has to answer or act from memory alone, with no fresh observations. Some requests are questions, such as which room the satchel is in. Others are actions, such as fetching the orange mug from a shelf of look-alikes or restoring a shelf to the way it was.

That split is the point. Because the agent does not know the future task while it is writing memory, the benchmark measures how well a memory design preserves evidence it did not know it would need. It also measures cost separately for writing memory and for reading it, since a design that is cheap to build can be expensive to use, and the reverse.

There are 53 tasks across four simulated indoor environments: an office, a photography studio, a home and a clinic, each with seven to nine rooms and a few hundred objects. The simulated robot is a Vega 1 Pro bimanual mobile manipulator, controlled through navigation, open-and-close and pick-and-place skills. Tasks fall into seven categories: object location, spatial configuration, appearance binding, set aggregation, user preference, occluded state and skill outcome, which is remembering which way of doing something actually worked.

Keep the Frame, Not the Caption

The headline result is simple. With no memory at all, the agent solves 3 of the 53 tasks. With hand-written facts handed to it, it solves all 53, so every task is solvable when the right evidence survives. In between, what the agent keeps decides what it can do.

Keeping only a text log scored 39 percent. Having the model write a description of each camera frame, the approach most text-based memory systems would take, scored 56 to 60 percent depending on how long the descriptions were allowed to be. Storing the camera frames themselves scored 70 percent, and cost less to write, because nothing has to be generated at write time. The gap is not a summarisation problem; descriptions and frames were stored and retrieved the same way. A description written before the task is known simply leaves things out. Which mug was next to which can, what colour the plate was, what was behind the cupboard door.

Adding a scene graph on top of the frames, a structured record of objects and their spatial relations, gave the highest overall success for every model tested, at a higher read-time cost. Frames helped in six of the seven task categories, with the clearest gains in appearance binding and object location, the two categories where a caption is most likely to have dropped the detail you later need.

The paper runs these comparisons across five memory backends, from a plain rolling transcript and a wiki-style writer to three published agent memory systems adapted to the robot setting, and across three base models. The pattern holds, with the size of the effect varying by backend and by task category. It also tries deployment-oriented variants: letting experience accumulate across tasks, keeping only selected frames to save storage, and giving the memory writer images as well as text. Each of those changes the trade-offs in its own way, which is the argument for having a benchmark at all.

The Same Protocol on a Real Vega 1 Pro

The team then ran the protocol on a physical Vega 1 Pro, the same embodiment as in simulation, using the robot's existing navigation and manipulation controllers. A SLAM map of the room provided the spatial record, annotated with objects and support relations. Observations came from the head camera, with objects segmented by an off-the-shelf segmentation model.

Two demonstrations tested memory of object arrangements. In the first, the robot throws away a soda can and is later asked to bring the fruit that had been next to it. In the second, it moves a lemon to another table, then has to recall that the lemon's original plate was orange and fetch the apple that had been to its right. In both, the robot used its memory, with frames plus scene graph on the wiki backend, and completed the delivery without help.

Two runs are a feasibility check, not a study. What they show is that the protocol and the memory interfaces carry over from simulation to a real robot with real camera observations and a real map, which is what you need before memory designs can be compared on hardware rather than only in simulation.

Why This Matters for Deployed Robots

A robot working a shift accumulates hours of experience that no single task needs and some later task will. The practical questions are what to keep, in what form, and at what cost in inference. MeMoREval gives those questions a controlled setting and a shared yardstick: common experiences, common interfaces, common grading. Its first answers are that frames are worth keeping, that a spatial map pays for itself, and that writing captions to save space costs you tasks later.

The work was done on the same platform we ship today as Vega 1 Pro and Vega 1 U, in simulation and on the robot.

Takeaway

For an embodied agent, forgetting is expensive and sometimes irreversible, so memory design deserves its own benchmark. MeMoREval provides one: 53 tasks, four environments, an experience-then-request protocol, and cost measured separately for writing and reading memory. Storing camera frames beats describing them, adding a scene graph helps most, and the protocol runs on a real Vega 1 Pro.

Related Resources

• MeMoREval: What Should Embodied Agents Remember? (paper page)
• Project page and code
• All research published on Dexmate robots