Using AI to allow blind people to find familiar faces

When Martin Grayson and Ed Cutrell followed a group of athletes and spectators with varying levels of vision on a trip from the United Kingdom to the 2016 for the Paralympic Games in Rio de Janeiro, Brazil, they began to pick up on clues that, even though there were a number of these athletes that could not see, an enriched understanding of social context was instilled on them, but unfortunately they could not seek the social cues necessary to perform this social interaction in the best way.

By observing how they interacted with other people, and even how they navigated through airports, attended a multitude of sporting venues, and even went sightseeing, this allowed the two collaborators to hoist key elements of social interaction, beginning with a very nuanced and elaborate sense of social understanding, which allows humans to modulate the way on how to interact with other people. Getting a sense of who is in the room, what they are doing, what is their relationship, and even if they are relevant to the person becomes an essential part of human interaction, which is stripped away from blind people.

With the help of Peter Bosher, an audio engineer who has been blind most of his life, the concept of a technology that provided information about the people around him resonated immediately, and thus project Tokyo was born with a partnership among researchers in the U.S., U.K., China, Japan, and India. By using HoloLens, these developers constructed a computer vision algorithm that provides varying levels of information about who is wherein the user’s environment. One model, for example, detects the pose of people in the environment, which provides a sense of where and how far away people are from the user, while another analyzes the stream of photos from the high-resolution camera to recognize people and determine if they have opted to make their names known to the system. All this information is relayed to the user through audio cues. There is also a LED strip fixed above the band of cameras which tracks the person closest to the user and turns green when the person has been identified to the user. Thus allowing partners or bystanders to know they’ve been seen, making it more natural to initiate a conversation.

Adapted from https://news.microsoft.com/innovation-stories/project-tokyo/
Learn more about the Project Tokyo and his developer here https://www.microsoft.com/en-us/research/people/cutrell/

DISCLAIMER: This story was written by someone who is not the author of the solution, therefore please be advised that, although it was written with the utmost respect for the innovation and the innovator, there can be some incorrect statements. If you find any errors please contact the patient Innovation team via info@patient-innovation.com or at carolina.piedade@patient-innovation.com directly.