Researchers from Carnegie Mellon University have created a drone system that can detect and film an artistic shot without scripted scenes, GPS tags or maps. It does this by learning and understanding human visual preferences. The team’s goal was to free people to focus on matters at hand, rather than directing a drone.

A new drone system can film scenes without human direction.

The team trained the drone with deep reinforcement learning using data from a user study, allowing it to detect which viewpoints make the most visually interesting shot for a scene. In the user study, participants viewed scenes on a photo-realistic simulator. The simulator displayed varied perspectives, shots and scales, distances and actor positions for the same scene. Users were asked to score scenes based on how visually appealing and artistically interesting they were.

The drone system learned how to shoot the best scene from the user study data, and learned that there are some movements and angles that are more visually interesting.

Other drone systems used in film are not capable of this. These systems will shoot a continuous backshot because it allows the drone to follow a clear and safe path behind the actor, even if the shot is not right for the scene. During the study, participants reported that long backshots can be boring and monotonous. To make these shots more interesting, the drones would have to quickly and frequently switch angles, but drones do not have the ability to switch angles often enough for this to work.

In the future, the team wants to explore more parameters and create customized artistic preferences for directors. They also want to enable the system with motion planners that can anticipate the actor’s movements and map the environment with lidar.

Drone systems like this would be useful for movies, but could also be used in sports, government and police surveillance. Police departments are already using drones to monitor crowds and understand traffic patterns, but their drone systems require a lot of focus to fly, which means there is less time to monitor what the drone is recording. With the new system, police would not have to spend time flying the drone and could focus on more pressing matters.

A paper on the research will be published in the Journal of Field Robotics and will be presented at the 2019 International Conference on Intelligent Robots and Systems.