Rice University researchers have developed technology to provide computers with continuous vision—a step toward allowing the devices to see what their owners see and keep track of what they need to remember.

“The concept is to allow our computers to assist us by showing them what we see throughout the day,” says Lin Zhong, professor of electrical and computer engineering. “It would be like having a personal assistant who can remember someone you met, where you met them, what they told you and other specific information like prices, dates and times.”

Zhong says "RedEye" is the kind of technology the computing industry is developing for use with wearable, hands-free, always-on devices that are designed to support people in their daily lives. The trend, which is sometimes referred to as “pervasive computing” or “ambient intelligence,” centers on technology that can recognize and even anticipate what someone needs and provide it right away.

RedEye is the kind of technology the computing industry is developing for use with wearable, hands-free, always-on devices designed to support people in their daily lives. Image credit: Pixabay.“The pervasive-computing movement foresees devices that are personal assistants, which help us in big and small ways at almost every moment of our lives,” Zhong says. “But a key enabler of this technology is equipping our devices to see what we see and hear what we hear. Smell, taste and touch may come later, but vision and sound will be the initial sensory inputs.”

According to Zhong, the bottleneck for continuous vision is energy consumption because today’s best smartphone cameras, though relatively inexpensive, are battery killers, especially when they are processing real-time video.

Zhong and former Rice graduate student Robert LiKamWa began studying the problem in 2012 by measuring the energy profiles of off-the-shelf image sensors and determined that existing technology would need to be about 100 times more energy efficient for continuous vision to become commercially viable. In a 2013 paper, LiKamWa, Zhong and colleagues showed they could improve the power consumption of commercially available image sensors tenfold simply through software optimization.

“RedEye grew from that because we still needed another tenfold improvement in energy efficiency, and we knew we would need to redesign both the hardware and software to achieve that,” LiKamWa says. The energy bottleneck was the conversion of images from analog to digital format, he adds.

“Real-world signals are analog, and converting them to digital signals is expensive in terms of energy,” he notes. “There’s a physical limit to how much energy savings you can achieve for that conversion. We decided a better option might be to analyze the signals while they were still analog.”

The main drawback of processing analog signals—and the reason digital conversion is the standard first step for most image-processing systems today—is that analog signals are inherently noisy, LiKamWa says. To make RedEye attractive to device makers, the team needed to demonstrate that it could reliably interpret analog signals.

They attacked the problem using a combination of the latest techniques from machine learning, system architecture and circuit design. In the case of machine learning, RedEye uses a technique called a “convolutional neural network,” an algorithmic structure inspired by the organization of the animal visual cortex.

“Conventional systems extract an entire image through the analog-to-digital converter and conduct image processing on the digital file,” LiKamWa says. “If you can shift that processing into the analog domain, then you will have a much smaller data bandwidth that you need to ship through that ADC bottleneck.”

LiKamWa said convolutional neural networks are the state-of-the-art way to perform object recognition, and the combination of these techniques with analog-domain processing presents some unique privacy advantages for RedEye.

“The upshot is that we can recognize objects—like cats, dogs, keys, phones, computers, faces, etc.—without actually looking at the image itself,” he says. “We’re just looking at the analog output from the vision sensor. We have an understanding of what’s there without having an actual image. This increases energy efficiency because we can choose to digitize only the images that are worth expending energy to create."

It also may help with privacy implications because a set of rules can be defined whereby the system will automatically discard the raw image after it has finished processing. So, if there are times, places or specific objects a user doesn’t want to record—and doesn’t want the system to remember—mechanisms can be designed to ensure that photos of those things are never created in the first place.

Zhong says the team is now working on a circuit layout for the RedEye architecture that can be used to test for layout issues, component mismatch, signal crosstalk and other hardware issues. Work is also ongoing to improve performance in low-light environments and other settings with low signal-to-noise ratios.