How to Turn Music Into Colors With a Wi-Fi Bridge

25 August 2016

Close Encounters of the Third Kind is a 1977 American science fiction film, written and directed by Steven Spielberg. As part of the movie's theme scientists enable a computer, using light and sound patterns together, to have a musical conversation with alien guests from a UFO visiting out planet. Author Charles Lohr discusses his algorithm ColorChord, which similarly assigns colors to notes. Lohr reviews his efforts to run it first on a 168-megahertz STM32F407/417 microcontroller and then on a WiFi bridge (the ESP8266 from Espressif Systems). Lohr wants to run ColorChord on the ESP so that it could listen to sounds, determine color values, and then use them to control a string of color LED modules. Each LED accepts a serial signal that specifies 8-bit RGB (red-green-blue) values for a given color.

Article Source: IEEE Spectrum

How to Turn Music Into Colors With a Wi-Fi Bridge

The ESP8266 microchip is an unexpected powerhouse

By Charles Lohr

Photo: Charles LohrPhoto: Charles Lohr

Several years ago, I wrote an algorithm I called ColorChord, which assigned colors to notes. The neat thing about it was that every time I played a note, the same color was assigned—regardless of the octave. By using these color values to control LEDs, I had a whole new way to visualize chords and melodies. Sadly, the algorithm was a bit clunky. The Pentium-based processors available at the time had difficulty running the algorithm in real time, forcing me to use specialized graphics processors.

Fast-forward a couple of years: A friend of mine dug into the algorithm and found it could be sped up enormously. Based on his feedback, I rewrote it twice. Eventually, it could easily run on a desktop CPU, and further tweaks and changes made it possible to run on a 168-megahertz STM32F407/417 microcontroller (STM31F407/417 Datasheet). (Eventually, I wondered if I could run it on something even simpler—something as simple as a Wi-Fi bridge?)

To be fair, I did have a specific Wi-Fi bridge in mind: the remarkable ESP8266 from Espressif Systems, based in Shanghai. The ESP was designed to be little more than a serial-to-Wi-Fi device that you could plug into your favorite controller and operate using AT-style modem commands to get access to wireless networks. However, the hacker community quickly realized that the ESP’s onboard processor could be repurposed for so much more.

The ESP8266 uses a Tensilica processor core, factory-set to 80 MHz but overclockable to 160 MHz. Its features include general-purpose input/output ports (GPIOs), Integrated Interchip Sound (I2S) interfacing with direct memory access (DMA) for speedy data transfers, hardware support for pulse-width modulation, and an analog-to-digital converter (ADC). It’s possible to make Internet of Things devices and servers possessing rich Web-based-configuration graphical-user interfaces with code running on the ESP. And with WebSockets, applications running on the ESP can provide updates to and receive input from remote users at a rate of over 600 hertz.

Light Notes

The community has developed firmware tools, a compiler, demos, and integrated development environments (IDEs) that let you run Arduino and NodeMCU code on the ESP. Fortunately, Espressif has embraced this opportunity. The company has released and updated its software development kit (SDK) and released documentation, rather than holding it behind mountains of nondisclosure agreements. Consequently, the ESP has gone from a US $2 Wi-Fi chip (breakout boards cost around $10) to a fully featured 32-bit wireless microcontroller with a growing following.

I wanted to run ColorChord on the ESP so that it could listen to sounds, determine color values, and then use them to control a string of WS2812B[pdf] LEDs. WS2812Bs are individually controllable color LED modules. Each LED accepts a serial signal that specifies 8-bit RGB (red-green-blue) values for a given color. When arranged in a string, each module takes its color from the first three bytes it sees, strips those bytes off, and passes the rest to the next module.

For many projects, lots of folks will find the Arduino IDE and its Wiring language to be ideal. My go-to, however, is the C language, which I use with the official Espressif SDK to build my own firmware. Though it can be more awkward, the SDK provides all the functionality you’ll need and gives you the ability to optimize your code. You build firmware using a GCC compiler and the SDK, and upload it to the ESP over a serial port.

In writing the software for my project, I knew I would need to sample the onboard ADC to analyze incoming sound from an attached electret microphone. Though the official SDK provides a function that can read the ADC, it wasn’t precise enough to produce the continuous audio wave forms that the ColorChord algorithm needed. After doing some online digging, I found someone who had worked out how to read the ADC using precise processor interrupt requests. Using these interrupts, I found I could read the ADC at a rate of 16 kilohertz and get a suitable wave form without jumps or jitters.

The next problem was controlling the string of WS2812Bs. They require precise timing to operate, and it can take a while to shift out all the color values for a long string. I turned to the ESP’s I2S bus and DMA. The I2S bus is normally used to read or write audio data, but it can be abused as a glorified shift register. Hooking up this bus to the DMA engine allows large buffers of data to be shifted out to the string of LED modules over a single pin, all with minimal use of the CPU.

With most of the low-level input and output being handled by supporting hardware, the ESP core was freed up for the heavy lifting of running the ColorChord algorithm. I took the highly optimized ColorChord C code I used for my desktop version, dropped it on the ESP, and ran it. Much to my shock, the ESP didn’t melt down. It connected to my home network,and when I visited its Web interface, everything seemed to be running. To convince myself it really was working, I hummed into the mic—and the LEDs glowed a bright yellow. And I was running the ESP at 80 MHz, not even tapping half its potential speed! Adding additional features to the Web interface let me configure and tune the algorithm, watch what all the notes were doing, and even see an oscilloscope of the wave forms picked up by the mic. I had done it!

The entire setup was small enough to attach to a saxophone. Then I placed LEDs inside the bell so that when the sax is played by someone—such as my friend David Chapman—it emits colors as well as sounds.

I have been amazed by the ESP8266 time and time again. Espressif is still revealing more and more of what this chip can do, and the community at esp8266.com is exploring possibilities that Espressif hasn’t even considered. I am going to keep searching and programming and learning more and more.

This article appears in the August 2016 print issue as “Turning Music Into Colors.”

Learn More

Arduino ESP8266 Hands On Wi-Fi microcontrollers music visualization

Sponsored Product:

High Level of Integration

The Espressif ESP8266EX is among the most integrated Wi-Fi chips in the industry. Measuring just 5mm x 5mm, ESP8266EX requires minimal external circuitry and integrates a 32-bit Tensilica MCU, standard digital peripheral interfaces, antenna switches, RF balun, power amplifier, low noise receive amplifier, filters and power management modules - all in one small package.

Low Power Management

Engineered for mobile devices, wearable electronics and the Internet of Things (IoT) applications, ESP8266EX achieves low power consumption with a combination of several proprietary technologies. The power saving architecture features three modes of operation - active mode, sleep mode and deep sleep mode, thus allowing battery-powered designs to run longer.

32-bit Tensilica MCU

ESP8266EX integrates Tensilica L106 32-bit micro controller (MCU) which features extra low power consumption and 16-bit RSIC, reaching a maximum clock speed of 160 MHz. With the Real Time Operation System (RTOS) enabled and Wi-Fi stack functional, about 80% of the processing power is still available for user application programming and development.

Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the Engineering360
Stay up to date on:
Our flagship newsletter covers all the technologies engineers need for new product development across disciplines and industries.