Audio Technology Helps Blind People Follow Australian Open Tennis

To hear more audio stories from publications like The New York Times, download Audm for iPhone or Android.

Rapid, echoing pops go off in Michael Marshall’s ears when he listens to an Australian Open tennis match, followed closely by high- and low-pitched clinks. Three pops on the left signal that the ball landed close to the line; a low-pitched clink means that the player returned it with a backhand stroke.

Without context, these noises might sound like arcade sound effects or some new version of Morse code — but each one is a message meant to help people who are blind or have limited vision follow the game. A new technology, called Action Audio, is being tested on a large scale for the first time at this year’s Australian Open, where every match in the Rod Laver Arena is available on a livestream with this accessibility feature.

Marshall, 35, of Melbourne, is an avid tennis fan who listens to the tournament on the radio every year. He said Action Audio had added a layer to his experience, allowing him to more clearly track the ball during a point.

“Everything is like, oh, yes, this is the final piece that I’m missing,” Marshall said. He later added, “It gives you those cues that you never really had before.”

He and other blind sports fans often listen to radio descriptions of games instead of watching them on TV. But with serves flying at over 100 miles an hour and groundstrokes routinely exceeding 80, even the most descriptive radio announcer can’t talk fast enough to capture every nuance of the action on a tennis court. Details like how close the ball landed to a line, how fast it traveled and what direction it moved are not always described, according to Machar Reid, head of innovation at Tennis Australia.

“They might do it on one shot, but it’s hard to do on every single shot on the rally,” he said of radio announcers.

Tennis Australia partnered with Monash University in Melbourne, Australia, and the digital design and communications agency AKQA to develop Action Audio in 2019. Reid said that tennis was the ideal sport to begin with because of its existing ball-tracking technology. In general, a fully equipped tennis court has 10 or 12 cameras collecting data around 50 times a second. That data is typically used to make quick calls about whether a shot landed in or out of play.

Action Audio’s technology converts the data into 3-D sound — a process that takes less than a second — allowing it to be broadcast alongside live radio commentary.

The sounds include one, two or three blips to indicate how close the ball is to the sideline or baseline. If the ball is close to the line, three blips will go off. If the ball lands farther inside the line, two blips sound. One blip means the ball was hit toward the middle of the court.

The sounds come through the left speaker if the ball was played to that side of the court, and through the right speaker if it landed on the right.

As the ball moves around, it jingles and rattles. The sound is loudest when the player strikes the ball, and it fades away gradually as the ball travels. High-pitched clinks indicate a forehand, while low-pitched clinks indicate a backhand.

In the future, Reid would love to integrate Action Audio into TV broadcasts and develop it for other sports that use optical tracking systems, such as baseball, he said.

One challenge in developing sport soundscapes is determining how many actions can be captured and translated to sound before the audio becomes overwhelming or distracting, according to Tim Devine, the executive innovation director at AKQA.

“Do you want to know where the ball is, or do you want to follow a player?” he asked.

To figure it out, AKQA and Tennis Australia reached out to fans to ask what they found most interesting about tennis and then surveyed blind people to see which sounds worked best. Developers tried to use sounds that were already familiar to users, such as the jingling rattle of the bell ball used in blind tennis, Reid said.

The developers are also collecting feedback from people who tune into this year’s Action Audio livestream, which has drawn listeners from about 70 countries.

Karl Belanger, an accessibility analyst for the National Federation of the Blind in the United States, is among those listeners. He was listening to the audio stream provided by Tennis Australia last week when he heard an advertisement for Action Audio. He was excited to tune in but became frustrated because the design of the site was not fully compatible with the screen-reader technology that blind people use to decipher web pages.

“That one layer of it just kind of mars the whole experience when you have to fight with the player to even experience it,” Belanger said.

Developers acknowledged Action Audio was a work in progress and said that AKQA and Tennis Australia would work to make the player more accessible.

To get the full benefit of Action Audio, listeners need headphones or speakers that can separate sounds for the left and right ears. Without this, Belanger said, the audio is not as helpful.

Belanger suggested other improvements — such as a sound for when the ball hits the net — but overall, he said, the audio soundscape was well designed. As a sports fan, he now wants the same concept used for every sport.

For some blind and low-vision fans, Action Audio makes it easier to enjoy the game alongside friends and family. Kala Petronijevic, 11, who is from Melbourne, is blind in her right eye and has limited vision in her left eye.

Kala has been playing blind tennis since she was 5. She is a big fan of the sport, but she didn’t always enjoy watching games. She used to have to constantly ask her father who had hit the ball and what was going on.

“It was hard following the game,” Kala said. “I wouldn’t really be interested because I wouldn’t know what was going on. But with Action Audio, I can follow the game promptly.”

Audio produced by Adrienne Hurst.