As robots become increasingly common and collaborative in manufacturing, interacting with them safely has also become an increased priority.

Ideally, robots work alongside humans safely, but do not require a degree in robotics or programming to operate. While modern physical human-robot interaction (pHRI) systems still fall short of that vision, they are continually improving.

Keeping people safe when operating around cobots currently focuses on collision detection. The logic is simple: if a robot runs into something it doesn’t expect, it needs to stop or reset, rather than potentially crush somebody. While that logic keeps workers safe, it also limits some of the ways humans would want to interact with a robot. This includes hand-guiding, where an operator intentionally moves the robot through a given path and the robot trains itself to repeat the motion for production.

To better enable these capabilities, engineers have turned to electronic skin (e-skin) matrices and tactile sensor arrays, as well as wrist-mounted torque sensors. Obtaining higher touch sensitivity has been a driving force in the development of many of these systems for years. However, they also have downsides. They add mechanical complexity, increase the system weight and fail often.

Collaborative robots have been gaining in popularity recently. They still have collision avoidance mechanisms in place, primarily through “stop-and-go” routines, where they execute a preset task unless some sort of trigger — such as a light curtain or "physical touch” would make them stop and require further instruction before proceeding. Crucially, they are unable to understand why the interrupt happened, and are only capable of continuing their pre-programmed paths.

So, as modern engineers look to alternative technologies or ideas, one of the most intriguing ways is to use systems that are already in place.

Reading force at the joint

Understanding natural physical interaction with robots requires an understanding of how they perceive their internal states. Standard industrial robots come equipped with high-resolution torque sensors directly integrated into each of their joints. While originally designed for collision detection and to compensate for gravity, they are also capable of measuring micro-deflections and slight changes in position that might be caused by either an external force or an internal problem.

Some of those external forces could be intentionally created by humans. When a human pushes against an arm, that force propagates throughout the mechanical structure, creating measurable variation in torques at joints throughout the robot. Using those variations, a controller can dynamically calculate the external force on the end-effector by using a Jacobian matrix.

While the forces introduced by these intentional movements might be minuscule by industrial robotics standards, likely only 1 or 2 newtons according to a recent study, they are large enough to distinguish themselves from both gravity and the inertial noise. Therefore, the controller can recognize when the arm is being trained to do something without relying on intricate external systems like e-skin or capacitive touch sensors.

Actually training them based on that input requires adoption of a different kind of technology though.

From motion to meaning

Robots typically operate under very rigid regimes, which can be a hindrance more than a help. Different users interact with robots in different ways. They use varying speeds, different pressures and have behavior quirks. To transform these nuances into standardized, semantic commands that robots can actually use, engineers regularly use recurrent neural network architectures, such as bidirectional long short-term memory (Bi-LSTM) networks.

Recurrent neural networks are particularly good at dealing with sequential data, such as a series of joint positions that a robot moves through. Adding a bidirectional component allows the system to operate with an even higher level of pattern recognition, because understanding what the purpose of a specific movement is depends on where the motion is going, rather than simply where it started.

A critical feature of this type of system is how adaptable they are to multiple users. Training data supplied from multiple different users results in highly accurate movements and positioning. However, when trained on a single user, the accuracy of future movements drops precipitously. This stresses a fundamental rule of machine learning when implemented in physical robotics: It’s critical to capture human variability in the training set.

The types of industrial robots that would benefit most from these training systems aren’t housed in pristine laboratories. The chaotic environment of the manufacturing line means that sometimes users will approach a robot from an awkward angle, creating inputs that might be completely different than those contained in a neural network's training set.

One option to deal with this inherent variability is to force engineers to attempt to generate thousands of inputs manually in order to train a model on enough of a variety of inputs that it won’t be completely unfamiliar with any new one. But that is time consuming and expensive, so instead engineers use a technique called data augmentation.

This involves simulating input data and positioning scenarios to ensure that rare inputs cases are not unfamiliar. Performing advanced mathematics on the simulated data, such as axis sign inversions, can generate remarkably resilient training data sets, and allows engineers to plan for situations that even their most creative colleagues wouldn’t come up with.

Putting the model to work

Training is only useful if it translates into meaningful behavior. And in many types of modern robots, that behavior is still governed by a state machine. One of the most common in pHRIs is a hierarchical finite state machine (HFSM). These programming tools attempt to create a frictionless, intuitive loop where a user can control a robot arm with a few simple touches and potentially a voice command too.

To move the HFSM from one state to another, some thresholds have to be reached, which are defined by the training data. For example, if a user touches an arm while it’s executing a task, it will stop immediately, similarly to a regular collision detection system. However, if the system is waiting in an idle state, a touch command or series of movements could send it into a different motion regime. Advanced systems could even integrate voice-activated controls to allow a user to confirm the change without ever requiring them to touch a computer screen or other traditional human-machine interface (HMI).

Summary

Turning a robot’s physical structure into its control interface still has a long way to go. Implementing such a system on larger robots, such as autonomous vehicles, might not even be feasible.

But as different types of machines become more integrated into manufacturing, how engineers interact with them becomes increasingly important.

From e-skins to neural networks, the technologies underpinning this shift are constantly evolving. There are many ways to interact with a robot, but engineers seem to have an eye on the end-game. It might be a while before we arrive at a standardized pHRI, but continual experimentation and feedback, which are similar to how the robots themselves experience the world, are the most efficient way to develop them.