Figure 1. Optical character recognition for license plate recognition was among the first inspection methods to be accomplished in a laboratory. Source: Adobe stock

Since the early 1980s, computer-based vision systems have been used to identify conditions in factory automation, as well as in the broader world. As computing power and programming techniques have advanced, new applications for this technology have followed, with Teledyne DALSA at the forefront of these advances.

Traditionally, machine vision has worked — and still largely works — based on a set of rules explicitly set up by engineers and integrators. For example, these rules might specify that a hole in a part has to be within a plus or minus range of a dimension, that there are a certain number of objects (or blobs — groups of pixels) visually present, or that only “X” number of vehicles are allowed in an area before turning on a stop light to deny further entry.

However, a vision system would not necessarily know that a shade of paint is not normal, or that a certain line looks a little off unless it is explicitly specified. While an experienced operator might question an unexpected anomaly, such a condition makes no difference to a machine if the inspection can still run.

Unlike a human operator, a machine may lack understanding when it encounters something that humans might notice as “off,” whether the machine is brand new, or if it has inspected hundreds or even millions of parts or scenes. To state the obvious, machines do not normally learn from their experience like a human, unless a programmer is directly involved.

On the other hand, they do not get tired, inspecting part after part, scene after scene, without complaint or drowsiness. So, while machines are spectacular at catching specific errors under laboratory and well-defined conditions, they do not pick up on unforeseen anomalies that humans would classify as “bad,” or at least “odd.”

However, the addition of artificial intelligence (AI) to visual inspections is quickly changing this, as AI improves with experience.

Machine vision case study: AI enhances traditional ANPR

Optical character recognition was among the first inspection methods to be accomplished in a laboratory. As this technology has advanced, image processing has been used to identify vehicles in the form of automatic number plate recognition (ANPR) for many years (Figure 1). However, with the advent of AI, such ANPR systems can reach a higher level of robustness, especially in challenging conditions such as darkness, rain and snow, with a lower engineering effort.

In addition, AI could be used to monitor for other automotive “conditions.” This would enable such abilities as monitoring for passengers in the car or even the use of seat belts. While drivers today may still stop at toll booths, and — hopefully not — receive a ticket from a real-life police officer, expect both processes to become more automated in the near future.

AI expands vision capabilities

With the emerging field of AI, where a machine can be taught to recognize and analyze conditions based on a data set – not well-defined rules – it is possible to imbue a vision system with “experience.” In the context of a vision system, this data is a set of images that allows a neural network to literally learn from the observed environment.

With the input dataset properly distilled into an AI model, a vision system can perform a number of tasks. This could mean detecting anomalies that would not normally appear, indicating a bad part. Or a system can be taught to recognize different “types” and sort them into groups — i.e. “This is a bald eagle, while that is, in fact a lizard.” Such concepts are elementary to humans, but much harder for a machine to define programmatically.

These abilities allow for automated vision inspection to extend into areas that were not previously possible, as a pure rules-based approach can be very difficult in some scenarios. AI setups are of course not perfect, and typically classifies how sure it is of its conclusions with a confidence level rating. If this does not meet a certain threshold, the part/scene could then be flagged for further human inspection, or simply rejected outright depending on the situation.

Figure 2. AI plus vision inspection methodologies are still in their infancy. Source: Adobe stock

Traditional machine vision meets AI

The drawback to an AI approach is that a “feeling” of whether something is correct or not is often not good enough. Take the measurement example mentioned earlier, where a part needs to be within a certain dimensional range. If it is close, the AI may be somewhat confident that it is an OK part, and perhaps when combined with other inspection aspects that are spot-on, may decide to pass an assembly that is not fully “good.”

In a world where one bad measurement out of 100 can mean a critical failure, this is not acceptable whatsoever. What is, in fact, needed is a way to merge the benefits of the AI’s “feelings,” with the cold calculating precision of a numerical-based pass/fail approach. In this way, AI can be implemented as just one segment of a larger vision-based inspection, leaving more well-defined criteria to more traditional machine vision methods.

AI inspection application potential use: Chip leads

While AI plus vision inspection methodologies are still in their infancy, one example of where this tandem approach could be expedient is for chip lead inspection. Here, AI could be implemented to detect the presence and location of any bent or damaged leads — an error which could take a near-infinite number of forms.

Traditional image processing methods could also be implemented in an inspection to measure the pitch between leads with high precision, a task that is not well-suited to AI. Such an approach would satisfy well-defined pitch specifications, while allowing AI to analyze potential “damaged lead” issues.

Astrocyte™ AI vision model training

The Sapera Vision Software Suite with Astrocyte™ by Teledyne DALSA is designed to integrate these two worlds on a wide range of camera and frame grabber hardware. Unlike other solutions, Astrocyte™ and the Sapera Vision Software Suite as a whole is not cloud-based, with models that are taught and implemented locally. This can be extremely important for users that need high data security for themselves as well as their customer’s data.

While previously discussed largely in the context of inspection and classification, Astrocyte™ supports multiple deep learning architectures, including:

• Classification, which is used to predict what an item is. Such a classification can be a yes/no distinction such as “Is that a person?” Or it can involve several categories in applications such as food sorting and character recognition.
• Anomaly detection, which is used to identify rare instances that are different from the majority of the dataset, often indicating defects. This functions as a binary classifier (normal/abnormal) and can be trained with unbalanced datasets, which involve a large number of normal samples and a small number of abnormal samples.
• Object detection, which finds a particular object or objects in an image and classifies it or them. It works well for presence detection, object tracking, locating defects and other similar operations.
• Segmentation, which sorts pixels into larger components for further analysis.
• Noise reduction, which reconstructs a high-quality image from a degraded image. This operation is an important building block for advanced analysis and sensing tasks.

AI integration with Sapera Vision Software Suite

Once trained, the resulting model or models can be integrated into an overall vision process using the Sapera Vision Software Suite. The Sapera Processing API lies at the heart of such inspections, allowing the use of the Astrocyte™-trained neural network as one element, along with other non-AI machine vision tools such as filtering, geometry and point-to-point operations. Other Sapera Processing tools include OCR and barcode reading, lens distortion correction and measurement/blob analysis.

Running on top of Sapera Processing, Teledyne DALSA’s Sherlock is a powerful graphical user interface for inspection setup. This allows system integrators, who may or may not be programmers, to focus on the end results rather than having to write code themselves.

The final element of this package is Sapera LT, Teledyne DALSA’s free image acquisition software development toolkit (SDK) for 2D/3D cameras and frame grabbers. APIs are available for C++, .NET and standard C, and it supports Windows as well as some Linux implementations. This software is hardware-independent, meaning it can be used with not only Teledyne DALSA’s cameras but a wide variety of other compatible devices.

Sapera LT features built-in diagnostic and monitoring tools, facilitating the resolution of issues for integrators and end-users. Extensive documentation and example source code is available, allowing users to quickly implement new inspection setups. Astrocyte™, Sapera Processing and Sapera LT are modular, so while they work well together and different elements can be integrated as needed.

Teledyne DALSA

Teledyne DALSA is part of the Teledyne Imaging group and a world leader in the design, manufacture and deployment of digital imaging components for the machine vision market. Teledyne DALSA image sensors, cameras, smart cameras, frame grabbers, software and vision solutions are used in thousands of automated inspection systems around the world and across multiple industries including semiconductor, solar cell, flat panel display, electronics, automotive, medical, packaging and general manufacturing.

Ready to upgrade to an integrated AI/machine vision approach? Teledyne DALSA is ready to help. They support customers by helping them configure built-in solutions into their specific environment, and can provide vision expertise for the integration of low-level hardware/software building blocks into their applications. More information is available at teledynedalsa.com.