A robot capable of picking up objects and placing them in another location in an unfamiliar room has been created by roboticists at New York University (NYU).

To create such a robot, the team outfitted a system from manufacturer Hello Robot with visual language models (VLMs), which enabled the robot to recognize objects based on language prompts.

Source: arXiv (2024). DOI: 10.48550/arxiv.2401.12202

The robot, dubbed OK-Robot, features wheels, a pole and retractable arms with claspers that serve as hands, according to the NYU team. Further, the system was not trained in the environment in which it was working in, but instead trained via a zero-shot algorithm.

OK-Robot was trialed in 10 volunteer homes where the NYU team created 3D videos using an iPhone and fed them to the robot so that it could get a feel for the layout of these spaces. Once inside, the robot was instructed to perform a series of 170 tasks including, for example, simple moving tasks wherein it was instructed to move a bottle on the shelf to the trash can.

Of those 170 tasks, the NYU team reported that the robot successfully completed them roughly 58% of the time. However, the team determined that the success rate might be improved to more than 80% by decluttering the workspace.

An article detailing the robot, “OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics,” appears in the journal arXiv.

To see OK-Robot perform simple moving tasks in an unfamiliar environment, watch the accompanying video, which appears courtesy of NYU.