Captions are a daily part of most people’s lives even if you don’t always realize it. On social media, captions are sometimes automatically generated for videos or you can simply go into your settings to turn them on. These options are designed to be more inclusive, and people that are hard of hearing or deaf need them on videos to understand what’s being said. The fact that they’re more widely available is great news.
One developer wanted to take captions a step further, into the real world. Captions on televisions or videos playing on your device’s screen are great, but wouldn’t it be helpful to have the ability to turn them on while you’re having an actual conversation? A video posted online by Paul Mealy, product design leadership at Meta, shows a new augmented reality (AR) tool that could be a game changer for people who are deaf or hard of hearing. Augmented reality is something that people are accustomed to through video games like Halo, the first-person game that has a pop-up display showing information about the player’s surroundings.
Like the video game, the real-time live-captioning technology uses augmented reality to display captions on top of the person speaking via wearables. The captions can even follow the speaker around and work in crowded spaces, focusing only on the identified speaker’s voice. Mealy says the prototype uses directional microphones, where speech is recorded and converted to live text and attached to the person speaking via body tracking in AR. “One of the issues the deaf and hard of hearing have had to deal with during the pandemic is masks obscuring facial expressions, which often provide cues to what a person is saying,” he says. “By utilizing captioning, we can help facilitate communication regardless of if the user’s face is covered.” According to Mealy, the prototype runs on mobile platforms, but performs best on wearable computing devices such as Microsoft’s HaloLens and other hands-free wearable devices.
These augmented reality glasses help deaf people see live captions of people’s speech nnBy @paul_mealynn#techforgood #innovation #technology #AR #VR #augmentedrealitynn@TechAmazing @sebbourguignon @Nicochan33 @KirkDBorne @mvollmer1 @pierrecappellipic.twitter.com/B74kejGIug— Pascal Bornet (@Pascal Bornet)
1647922692
The post showing Mealy testing out the prototype has been shared several times across social media, with Pascal Bornet, the chief data officer of Aera Technology, sharing it with his more than 40,000 Twitter followers. Comments on the posts are positive in nature and filled with excitement about the potential benefits. Several commenters said the technology is “amazing” while some had handy tips like allowing the user to make the text stationary instead of staying with the person. Another advised that the text be placed at the side of the screen with an arrow prompting the user to look toward the text. The excitement from the commenters can be felt through the screen.
It’s not clear from the post when this technology will be available commercially, but it’s clear that when it is available it will give deaf and hard of hearing people much more accommodations to navigate hearing spaces. If people could learn ASL throughout their school years as a standard education course as well as having access to this new technology, the world would be so much more accessible to deaf and hard of hearing people. It’s wonderful that people like Mealy are working to do their part.