Editor's Note: It's NFL Super Bowl time again, and this article first published in March 2015 on Engineering360 may be just the thing to distract you and your friends if the big game doesn't go your way.

The addition of an on-screen graphic to show TV audiences the first-down line in U.S. football is now so routine that televised games that do not offer a yellow first-down line look like they are missing a standard part of the game.

Though the "1st and Ten" system from Sportvision Inc. was introduced without any pre-promotion or fanfare in September 1998 for National Football League games, it revolutionized televised U.S. football with an impact comparable to that of instant replay. The technology has been so successful that its use has been extended to motorsports, baseball, sailing and what the world outside of the U.S. knows as football.

The basic concept is simple: draw a superimposed straight line in the right place on the playing field. However, the development work that made this yellow line possible was complex, both in the conceptual details and its on-site execution. The technology's success required more than just advanced graphic-image algorithms running on powerful graphics processors. Delivering this feature was complicated by the fact that sporting venues change regularly. Perhaps more important — sports broadcasters and game fans are unforgiving of even minor glitches or mistakes. (For proof, look no further than the blown call at the end of the New Orleans Saints game against the Los Angeles Rams.)

Follow the Puck

The idea behind 1st and Ten began with a request from management at media and entertainment giant News Corp in 1994 to Stan Honey, who at the time was executive vice president of technology. (Honey earned his Electrical Engineering degree from Stanford University in 1983.) News Corp managers wanted a system to track and highlight fast-moving hockey pucks during televised games via a "glow" and puck-trailing tail. Honey pulled together a team, many of whom, like him, had extensive backgrounds in military projects at SRI International and other defense-oriented firms. They delivered the hockey system at a price that was below the $2 million budget and within the promised 18-month schedule.

(Click to enlarge.) Stan Honey was technical director for the America's cup yachting race and is an accomplished sailor. Source: Stanford University The hockey system required not only TV cameras and data processing, but a puck with shock-ruggedized infrared transmitter and with an overall weight and size identical to the standard puck (and produced in bulk due to limited battery life). Developers also had to compensate for telephoto lens distortion and each playing rink had to be calibrated using holes drilled in the ice and filled with blue dye.

While the tracking system proved to be a technical success, fans were against it for many reasons and the system was abandoned after a few years. Despite the audience rejection, Honey's team realized they could adapt the technology to display the "simple" football first-down line, while also leveraging the credibility and contacts gained through the hockey project. This led to the founding of Sportvision in 1998. (In 2016, it was acquired by SMT.)

Drawing a Line on the Turf

Although drawing a straight line in the right place across a nominally green field would seem to be an easier task than finding and highlighting a fast-moving hockey puck, the football problem had new and more challenging issues. Among these: the field is not actually flat — it's crowned to encourage water drainage; the green color has many variations and brown spots, thus standard video chroma-keying was not an option; the shadings on the field change during play due to clouds, the sun's motion and artificial lighting; players’ uniforms may have some green in them; the players themselves are continually moving, yet the line needs to appear to be under them at all times; and any distortion or curving in the virtual line is immediately obvious.

The system also required calibrated image-capture zoom-lens cameras accompanied by precise readouts of their pan, zoom and tilt (PZT) angles, stadium-specific field mapping and related color keying, and much more — all to create a model of the game situation in real time.

The first system used a quartet of SGI computers, three data-acquisition systems and multiple camera encoders, among other hardware. It required a full trailer to transport and more than a dozen people per game, plus a full day to set up and initialize. While this magnitude of effort and expense for a single game may seem high, it was not a problem in this case, given the vast amount of money associated with NFL games.

(Click to enlarge.) The 1st and Ten system required six cameras feeding every captured frame image to a single preprocessor via RS-422 signaling over CAT-5 cable. Source: SportvisionThe 1st and Ten system required six cameras feeding every captured frame image to a single preprocessor via RS-422 signaling over CAT-5 cable. Each camera also sent along its PZT parameters on a video sub-channel. The output of the preprocessor was fed to another computer which built a real-time, continuously updated 3D model of the field perspective (recall that each playing field had to be laser-surveyed and plotted before the game, with critical coordinates entered before run time). The camera information, processed images and model were then used for detailed geometrical processing. Further, the captured image of the field had to be corrected for unavoidable distortion due to lens aberrations, zooming, camera angle and position. Everything had to be done at the rate of 60 image fields/sec frame-grab.

Players and Other Obstructions

The algorithm objective is simple to describe: determine which pixels in the video frame are needed to create an unobstructed first-down line, and then take into account obstructions such as players or markers. As these obstructions move over the line, the system adjusts which pixels need to be highlighted with the virtual first-down yellow line and which should show live, original video feed.

The line itself is composed of polygons, which the processor calculates as a balance polygon resolution versus color, space, time to compute and other factors. Finally, the injected line had to make allowance for any graphic overlays used by the broadcaster such as game status, player profiles, score and promotional teasers.

As processing capability has increased while computing size, cost and power-source requirements have gone down, the first-down system has expanded to also show the line of scrimmage, football trajectory and more. Versions now cover motorsports such as NASCAR races, baseball, golf, the America's Cup sailing race and even Olympic events (Stan Honey, now retired from Sportvision, holds world records in several trans-Atlantic and trans-Pacific sailing categories). Each of these new applications entails much more than just a modest reworking and extension of the original football first-down system, but instead has its own significant challenges.

From Field to Track

For example, the NASCAR implementation (introduced in 2001) shows car position and movement, speed, on/off brake status and tachometer reading. The system needed in-vehicle GPS to precisely locate the cars. This, in turn, required design of custom GPS receivers with the ability to transmit the GPS position back to a base station.

(Click to enlarge.) Tracking race cars for NASCAR proved to be a technical challenge. Source: Sportvision That seems easy enough, but the issues have proven to be complex. For example, where do you locate these telemetry boxes in the already crowded vehicle? How do you reliably receive the inherently weak GPS signal in an electrically noisy car environment where the protective metal mesh overhanging the track sometimes blocks the signal? What custom telemetry protocol do you develop because conventional TCP/IP (the most-obvious acknowledge/retry format) is too slow? Finally, can you even build a reliable under-hood system when heat from the race car engine dictates the need for ruggedized electronics, and where even toughened batteries could fail prematurely or catch fire?

Development of Sportvision's original 1st and Ten system shows that despite market failure, a technical success can still evolve to become a winning product, even if the path is less than easy. It also demonstrates that once implementation is field-proven and becomes essential to its customers, it can be leveraged into adjacent areas, although with considerable effort. By 2015, Sportvision had won 10 Emmy Awards and has been involved in more than 20,000 televised events.

Brand and product extensions do not always come easy, even for those with established products and reputations, but the opportunity is there.