Guided by user hints, a new system uses AI to generate multiple plausible colorizations. Image credit: Library of CongressSuppose you had an old black-and-white photograph and you wanted to bring it into the 21st century by adding realistic color.

Well, you could do it manually, which would likely be an exceptionally time-consuming process. Professionals can work on a single image for days to get it just right. But researchers at UC Berkeley, led by electrical engineering and computer sciences professor Alexei A. Efros, have developed a new technique that leverages deep networks and artificial intelligence and allows novices to quickly produce reasonable results—with no artistic ability required.

In previous work, the team trained a deep network on big visual data to colorize grayscale images automatically, without user intervention. Its Colorful Image Colorization microservice is an algorithm trained on a million images that can do some pretty impressive brushwork. But there are limitations; when the system encounters an ambiguous color, it decides on a single possibility—and the user is stuck with the computer’s choice.

"The goal of our previous project was to just get a single, plausible colorization," says co-author Richard Zhang, a Ph.D. candidate. "If the user didn't like the result, or wanted to change something, they were out of luck.”

The new system, however, enables the user to correct and customize the colorization in real time. The system takes user input in the form of colored points, or “hints,” and propagates them to the rest of the image. Conversely, the system learns common colors for different objects and makes recommendations to the user. And it will allow for “out-of-the-box” thinking: If a user decides to make elephants pink, the system won’t argue—even though it knows what color they are in the natural world.

The team tested their interface on novice users, challenging them to realistically colorize a randomly selected grayscale image. Even with a time constraint of just one minute per image, the users produced colorizations that fooled human judges in a test scenario.

The research, entitled "Real-Time User Guided Colorization with Learned Deep Priors,” will be presented at SIGGRAPH 2017, which showcases some of the most innovative in computer graphics research and interactive techniques worldwide.