A memory system created by MIT researchers is designed to identify entries in a robot’s memory that might be wrong and corrects the descriptions of mislabeled objects. This method could help robots better perform complex tasks that require multiple steps.

The system, which is called UQ-DAAAM (Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment), compares multiple descriptions of the same object, seen from different viewpoints, to quantify the robot’s uncertainty about what it saw.

It then selects additional views to reduce the uncertainty and combines these captions into a more accurate description of the object, which it saves to memory. The researchers also employed a special technique to streamline this process so a mobile robot can continually update uncertain memory entries in real time.

Amending memory and remembering

This method builds on the researchers’ prior work, DAAAM, a robot memory system that uses a vision-language model (VLM) to richly describe objects in scenes a robot captures with its onboard cameras. These descriptions are stored as detailed captions in a 3D map-based representation of the robot’s environment that it can call upon later to answer questions.

However, the robot sometimes might see objects from a distance or the camera might capture blurry or poorly lit images. UQ-DAAAM works to resolve this by integrating uncertainty quantification into the DAAAM framework. It allows the robot to measure its confidence in the accuracy of object descriptions and fix incorrect captions as it travels through an environment.

While on the move, the robot constantly collects and stores images of the world around it. Many of these images show the same objects captured from different angles. UQ-DAAAM uses these additional viewpoints to identify and correct unreliable descriptions.

Once it has identified additional captions that will reduce its uncertainty about the object, the system uses a large language model (LLM) to fuse them into a single object description. To boost accuracy, the LLM puts more weight on new captions and those with less ambiguity.

This refinement step can resolve ambiguities for most of the objects in the robot’s environment, while only requiring two additional captions per object. UQ-DAAAM also provided more accurate answers to questions about the objects it observed, where it saw them, and when it last encountered them than baseline methods.

As the robot moves across an environment, it takes multiple photos with onboard cameras. On left are three views of a dark object on a rooftop. The system initially states it can’t give an accurate description and then identifies the object incorrectly as a squirrel. As it continues moving, the robot integrates uncertainty quantification and it refines the view and states the object is “a dark gray, cylindrical chimney with a slightly tapered top and a smooth surface.” Courtesy: Massachusetts Institute of Technology

The researchers are planning to use UQ-DAAAM to guide different types of robots, like mobile robotic manipulators, and apply their uncertainty quantification techniques to other domains.

They also are studying the use of UQ-DAAAM to identify areas of high uncertainty the robot could revisit to improve its understanding of the environment. They are also exploring uncertainty quantification in systems that predict the future state of a robot’s environment.