A robot was able to copy a sketch of the Mona Lisa just by looking at it. Source: Brown University Computer scientists at Brown University have created an algorithm that enables robots to write words and produce sketches — a development that may open the lines of communication between robots and their human coworkers and collaborators.

Using deep learning networks that evaluate images of handwritten words or sketches, the algorithm can interpret the probable pen strokes used to create them. Based on that information, the algorithm replicates the words and sketches by mimicking the pen strokes it observed.

This is accomplished using two different models of an image the algorithm has been tasked with replicating. The first model is a global model that looks at the image as a whole. With that model, the algorithm locates a probable starting point for issuing the first stroke. As the first stroke is issued, the algorithm then begins looking at the second image model pixel by pixel to measure where the stroke should go and how long it needs to be. When the stroke is completed, the algorithm again references the global model, starting the process over again until the image is completed.

"Just by looking at a target image of a word or sketch, the robot can reproduce each stroke as one continuous action," said Atsunobu Kotani, a Brown undergraduate who led the algorithm's development. "That makes it hard for people to distinguish if it was written by the robot."

To demonstrate their work on the algorithm, researchers enabled a robot to write “hello” in 10 different languages and to reproduce a rough sketch of the Mona Lisa.

Hoping to improve communication between robots and people, the developers envision that robots will be able to leave Post-it Note messages, take dictation or sketch diagrams for human collaborators and coworkers in the future.

“I want a robot to be able to do everything a person can do," said Stefanie Tellex, an assistant professor of computer science at Brown and Kotani's advisor. "I'm particularly interested in a robot that can use language. Writing is a way that people use language, so we thought we should try this."

To see the robot in action, watch the accompanying video that appears courtesy of Brown University.