German researchers are developing a system to monitor vehicles' interior environment with the aim of helping achieve a more fluid passenger experience as autonomous technologies are adopted.

“Using depth-perception cameras, we capture the vehicle’s interior, identify the number of people, their size and their posture. From this, we can deduce their activities," says Dr. Michael Voit, group manager at Fraunhofer Institutes for Optronics, System Technologies and Image Exploitation. Fraunhofer is collaborating with Volkswagen Group Research, Bosch and other companies on the “Intelligent Car Interior” project, funded by the German Federal Ministry of Education and Research.

Such functionality has relevance to a number of assistance systems, such as partially autonomous driving situations, the researchers say. For example, if the driver needs to attend to children sitting in the back seat, the system could display a video image of the rear of the vehicle to enable the driver to keep his or her eyes on the road, yet still see what the children are up to.

Assistance systems will soon be able to detect what vehicle occupants are doing—an important feature for autonomous driving. Image credit: Fraunhofer IAO. Based on information regarding where people are sitting and how big they are, airbags could also be adjusted to individual body sizes. And by analyzing the position of a passenger’s limbs, airbags could recognize special situations, such as when a passenger has his feet on the dashboard.

According to the Fraunhofer researchers, the main challenges in implementing such a system lie in evaluating the recorded data. Software can already detect people and their limbs, and a type of image overlay can also trace their movements. But how can the computer be taught to recognize what passengers are doing?

“One challenge is to reliably identify the objects people are using. If one considers that, in principle, any object could find its way into the vehicle, we must somehow limit the number of detectable possibilities," says Voit. "We therefore set basic parameters and tell the computer where the sun visor and glove compartment are, for example.”

The researchers first tested and refined the cameras and the associated evaluation algorithms in Fraunhofer’s driving simulator. Next, the system will be integrated into a Volkswagen Multivan. The results will form the basis for new vehicle concepts for the next five to ten years.