A Carnegie Mellon University research team in Pennsylvania is teaching robots how to recognize and grasp objects by much the same techniques as babies learn.

Manipulation is a challenge for robots and a bottleneck for many applications. For example, during the Defense Advanced Research Projects Agency's 2015 Robotics Challenge, some of the world's most advanced robots had difficulty opening doors or unplugging and re-plugging an electrical cable.

A robot learns about its world by poking  and pushing objects.A robot learns about its world by poking and pushing objects.Key to the university researchers' approach is allowing robots to spend hundreds of hours poking, grabbing, and otherwise physically interacting with a variety of objects. In their findings, the CMU researchers show that robots gain a deeper visual understanding of objects when they are able to manipulate them.

In the past, visual perception and robotic control have been studied separately, says Abhinav Gupta, leader of the research team. Visual perception has been developed with little consideration of physical interaction. Most manipulation and planning frameworks can't cope with perception failures, says Gupta. He predicts that allowing the robot to explore perception and action simultaneously, like a baby, can help overcome these failures. The learning process, however, will require hundreds of hours of interaction.

"We will use dozens of different robots, including one- and two-armed robots, and even drawings, to learn about the world and actions that can be performed in the world," says Gupta.

The cost of robots has come down in recent years, enabling researchers to unleash lots of robots to collect an unprecedented amount of data on physical interactions, he says. A past problem has been the difficulty of gathering sufficient amounts of data. The team will scale up the learning process to help address this data shortage.

According to Lerrel Pinto, one of the researchers, much of the work has been done using a two on Baxter robot with a two-fingered manipulator. Using more and different robots, including those with more sophisticated hands, will enrich manipulation databases.