While sitting in class one day at Columbia University, Tomer Aharoni received a voice message from a friend, and a follow-up text message indicated the call was urgent. Aharoni decided to redirect the sound output of his computer into the sound input. Then he opened a Google Doc and selected the dictation option. “I played the voice memo,” he recalls, “and thanks to Google, it was immediately translated to a text on my screen. I realized I might be onto something.”

For the Columbia University DevFest, he and four friends—Ori Aboodi, Ben Arbib, Alon Ezer, and Roy Prigat—banded together to develop an app that would allow those who were speech or hearing-impaired to speak over the phone by converting text-to-speech, and speech-to-text. All were computer science students (as well as veterans of the Israeli Defense Force). “We wanted to create something that was more than ‘cool,’ and that people in need could actually benefit from,” says Ezer.

“Throughout my life, I’ve known many people with different disabilities who have been unable to use the technologies that I rely upon every day,” says Aboodi. “I couldn’t imagine my life without the ease of communication I have. So I’d love to bring that access to those who don’t have it.”

And so Nagish—which means “accessible” in Hebrew—was born. The project has given the team a chance to collaborate on things they all value: learning new skills, seeking new challenges, and helping others.

Hacking for social good

Using Google Compute Engine’s virtual machines, the team deployed a Flask script that integrates Twilio’s voice services with Google’s Speech-to-Text and Text-to-Speech APIs to create a proxy, or middleman, between voice and text communication. They also deployed a second virtual machine running a Flask script that uses the Facebook Messenger API and the Google Meet API to send and receive written messages.

The app redirects incoming texts and calls to a new phone number linked to a chat app, converting text-to-speech and speech-to-text so that “one person can speak and hear while the other can type and read, all in real time,” says Aharoni.

Google Cloud enabled the APIs to work together so incoming and outgoing messages could be processed seamlessly. According to Aboodi, they wanted a platform that would be “easy to use, that had many solutions in one place, and good funding we could benefit from. Google Cloud is a great tool, especially for students, to build something on your own.”

With an initial grant in Google Cloud Education credits, the team built a demo in just twelve hours and demonstrated it to a live audience at the hackathon, where it was awarded Second Place overall and Best Hack for Social Good.

What’s next for Nagish

The team hopes to perfect the app before its beta release. They’re working on improving the speech recognition, adding functionality for customer support and service calls, and support for other chat applications. By integrating the Google Translate API, they could potentially offer real-time translation between speakers of different languages—thus advancing their mission to connect people across distances, abilities, and cultures.

“We study at a university with a broad diversity of people from many different backgrounds who speak different languages,” says Arbib. “We thought about how real-time translation would help if we made international calls to places that don't share a common language with us.”

The potential to help transform communication is thrilling: "I want to develop technologies that break boundaries and create a more connected world,” says Aboodi.

Aharoni says he is excited about “making such a simple technology—phone calls—accessible to whoever needs it and has never used it before. It’s about making a change for good. None of us had intensive software engineering experience when we started working on Nagish. Yet we were able to combine our skills in a way that allowed us to bring it to life.”