Inspired by their classmate’s physical therapy challenges, students at the University of California, Berkeley developed an application to guide people through their workouts. Their mobile app, EnhanceAI, helps users stay motivated, improve their form, and track their progress on key strength-training moves.

Umang Srivastav, a sophomore studying computer science (CS) at the University of California, Irvine, had been living with chronic back, neck, and shoulder pain since he was a child. His weekly PT sessions demanded regular therapeutic exercises at home, including push-ups and planks. These could be a struggle to do correctly. Even using a mirror, he wasn’t sure his repetitions maintained the proper form—essential to achieving the best results—without a physical therapist there to guide him.

These challenges provided inspiration at UC Davis’s HackDavis hackathon in February 2019. Srivastav and high school pal Maxwell Chan, a sophomore studying CS at UC Berkeley, teamed up with Chan’s classmates, Micah Yong and Aditya Ramkumar (also UC Berkeley sophomores studying CS), to brainstorm ideas at the hackathon for a fitness-related app.

“HackDavis had the prompt ‘Develop a product that encourages people to live a healthier lifestyle,’” Yong says. The four students set out to create a social media app with a messaging feature, focused on fitness. For their demo, they chose push-ups, a popular strength-building exercise in both physical therapy and fitness programs. “We thought it would be fun to gamify these exercises by allowing users to challenge each other to push-up contests on their phones,” Yong says.

Chan had attended Google’s Computer Science Summer Institute (CSSI) after high school, where he learned about artificial intelligence (AI) and machine learning (ML). “Google’s AI/ML tools looked especially intuitive, like Cloud Speech-to-Text and Cloud AutoML,” Chan says. “After CSSI, I knew I wanted to build something with Google products in the future." So the team decided to try Google Cloud’s machine learning products for their hackathon project.

They used Tensorflow to train an image classification model to correctly identify whether or not a person was doing push-ups with proper form. “Machine learning was not as hard as it sounded,” says Yong, who learned how ML works in the TensorFlow for Poets Codelab. “Because TensorFlow was so well documented, a lot of the process for training your custom ML model is handled for you as a developer.”

The team’s push-up demo attracted plenty of attention from the crowd. “But we began to see that our app lacked clarity,” Yong says. “It really wasn’t what someone like Umang needs, [which] is a personal trainer—not an app that puts his physical abilities in competition with others.”

After HackDavis, team members moved on to other projects. Yong, however, still wanted to pursue the idea of a mobile app for PT patients and fitness enthusiasts. “The perfect product usually isn’t the one that comes out instantly,” he says. “It’s the one that’s perfected with market research, intuitive design, and user testing.”

Working out the vision

Shortly after HackDavis, Yong was contacted by Arman Hezarkhani, a consultant to Google’s Cloud Team, who offered guidance on further app development. He suggested additional Google Cloud tools that Yong could use to help bring his ideas to light.

“Our idea began as a potential solution for those with musculoskeletal injuries and chronic pain who needed something to guide them through their prescribed exercises while checking their form,” Yong says. After the hackathon, Yong’s ideas “quickly evolved to an app available for anyone—from those in PT to fitness enthusiasts to students to working professionals and just about anyone who wanted additional incentive to workout while having someone or something to keep them accountable.”

Yong recruited his dormmates to hit the mat with him to create an image dataset of them doing push-ups for strength and planks for core. He found a library for real-time multi-person keypoint detection to create a custom ML model trained to recognize the human body and extract locations of joints and bones. Users could now take phone videos of themselves doing push-ups and planks and have the app compare their form to the dataset models.

To make this work, Yong again used TensorFlow, along with other open-source computer vision libraries, to train and test a custom ML model that determined whether or not the user performed an exercise correctly. Using a separate framework, he added the app’s ability to time and track jogs to help users’ build stamina. He used Firebase Realtime Database to track users’ basic profile information and exercise performance and Firebase Authentication to swiftly and securely sign in new and returning users to the app. Both of these tools/products are a part of Google’s Firebase mobile development platform."

“Not only are these tools well documented … but there are several tutorials on YouTube and Medium that cover how to use them step-by-step,” he says. “Google Cloud makes learning easy for a developer like me who wasn’t super familiar with using database tools or machine learning APIs.”

In a few months, Yong developed the EnhanceAI mobile app (see demo and GitHub). His original idea for a tool to help PT patients expanded into an app for anyone seeking ways to improve their exercise routines. Yong chose pushups and planks for core-, upper-, and lower-body strength training and jogging for stamina so the app could benefit not only PT patients but a wider, fitness-minded audience.

“It is a fitness application designed for really busy lives,” he says. “It uses computer vision and ML to guide you through your workouts, giving visual and audio feedback based on your form.” Yong says what differentiates his app from others on the market is “its ability to keep users accountable and track their own progress without the need to meet with personal trainers.” He presented EnhanceAI at UC Berkeley’s iOS Showcase and won the “Most Innovative” award.

What’s next for the fitness app

Yong launched EnhanceAI to the public in fall 2019, with nearly 450 impressions and over 140 downloads. He is currently doing research to determine the greater need within the health care and fitness communities, and to come up with a marketing plan. “I would like the app to offer additional exercises,” he says. “Reviewers say logging in is really easy, and they enjoy being able to track their progress every time they exercise.”

“Google Cloud products and tools were a huge factor in enabling me to efficiently develop this app, especially Firebase,” he says. “Firebase was huge. Implementing that functionality yourself is not practical, and of course, I don't have access to my own server and database.”

“Using Google Cloud to streamline the process of [developing] the app was a really big thing for me,” he says. To other student developers seeking to try out their ideas and build something new, he advises, “Just go for it.” With Google Cloud, “it's not as scary as you may think.”

Get started with Google Cloud’s higher education learning center at: g.co/learncloud/programs