Signing Out Loud

A research team led by the University at Buffalo has modified common noise-cancelling headphones to enable the electronic device to “see” and translate American Sign Language (ASL) into voiced speech when paired with a smartphone.

Dubbed SonicASL, the system proved 93.8% effective in tests performed indoors and outdoors involving 42 individual words. Under the same conditions involving 30 simple sentences, SonicASL was 90.6% effective.

According to the study’s corresponding author, Zhanpeng Jin, an associate professor in the Department of Computer Science and Engineering at UB, “[It’s] an exciting proof of concept that could eventually help greatly improve communications between deaf and hearing populations.”

Most noise-cancelling headphones rely on external microphones that pick up noise around the listener. The headphones then cancel that noise by producing anti-sound, inverted soundwaves with the same amplitude as the external noise.

SonicASL adapts that process to “listen” to sign language. Using Doppler technology, the microphones pick up the tiny fluctuations in acoustic soundwaves created by the hands of someone signing. The accompanying SonicASL smartphone app, which contains an algorithm that identifies the words and sentences, processes that information to translate the signs and speak them to the user via the earphones.

Unlike systems that put the responsibility for bridging the communication gap on the deaf, SonicASL flips the script, encouraging the hearing population to make the effort.

And it’s an incredibly flexible system, says Jin. The core algorithm can be implemented and deployed on any smartphone, and the system, “with proper training of the algorithmic model,” can be adapted for languages other than ASL. This is key, as the roughly 72 million deaf people around the world speak upward of 300 different sign languages.

There is still much work to be done before the technology will be commercially available, says Jin. The next steps will be greatly expanding SonicASL’s vocabulary and incorporating the ability to read facial expressions, another major component of ASL.