Virtual medical appointments have enabled health professionals to minimize in-person contact while still providing care to patients, however, they lack the ability for doctors to reliably obtain a patient’s pulse, respiration and other important vital signs in real time. University of Washington (UW) researchers led a team that has developed a way to use a smartphone or computer camera to take a person’s pulse and respiration signal from a real-time video.

The researchers presented this state-of-the-art system in December at the Neural Information Processing Systems conference and are now proposing improvements to better measure these important physiological signals.

“Machine learning is pretty good at classifying images. If you give it a series of photos of cats and then tell it to find cats in other images, it can do it. But for machine learning to be helpful in remote health sensing, we need a system that can identify the region of interest in a video that holds the strongest source of The method uses the camera on a person’s smartphone or computer to take their pulse and breathing rate from a real-time video of their face. Source: Cristina Zaragoza/Unsplashphysiological information — pulse, for example — and then measure that over time,” said lead author Xin Liu, a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering.

The system has to adapt quickly to each person’s unique physiological signature, apart from other variations, said Liu.

To preserve privacy, the system runs on the person’s device, not in the cloud. Machine learning captures subtle changes in light reflecting off the person’s face, which indicates changing blood flow, and converts the changes into pulse and respiration rate.

A dataset of videos of people’s faces and each person’s pulse and respiration measured by standard instruments in the field was used to train the system initially. Next, the system used spatial and temporal information from the videos to calculate the two vital signs. This approach performed better than machine learning systems using videos where subjects were moving and talking. However, the team had to overcome the problem of “overfitting,” difficulty with datasets that included different people, backgrounds and lighting.

The system generated a personalized machine-learning model for each individual to help it look for the areas in the video frame containing the physiological features that correlate with changing blood flow in a face under different contexts, for example, skin tone, lighting conditions and environment. This allowed the system to focus on the right area.

Try the researchers’ demo version that can detect a user’s heartbeat over time. Doctors can use this to calculate heart rate.

“We acknowledge that there is still a trend toward inferior performance when the subject’s skin type is darker,” Liu said. “This is in part because light reflects differently off of darker skin, resulting in a weaker signal for the camera to pick up. Our team is actively developing new methods to solve this limitation.”

The researchers are also collaborating with doctors to understand how this system performs in the clinic.