At HackNYU 2018, four computer science graduate students—Mohammed Ilyas Habeeb, Antonio Mallia, Rakshit Sareen, and Harshit Srivastava—debuted their plan for a project called DermaScan. The team wanted to develop an app that would help users identify and scan skin moles at an early stage to determine whether to consult a dermatologist.

With more than 5 million diagnosed cases of skin cancer in the United States annually, the DermaScan team knew their app could potentially help address a major public health issue. The deadliest form of skin cancer, melanoma, is responsible for thousands of deaths each year. Catching it early significantly increases chances of survival.

Creating a new tool for self-diagnosis

In the spring of 2018, Antonio Mallia, who is a PhD candidate in the department of Computer Science at NYU, decided to explore something very different from Information Retrieval, his primary research interest, and discussed the idea of an independent early-detection tool for skin cancer with his teammates. They were immediately receptive to the notion of empowering users. “The idea is to have a tool that allows a patient to perform a basic skin mole analysis at home, with the use of artificial intelligence,” he says.

Srivastava notes that skin rashes may seem benign, “but they can turn out to be very dangerous, if not taken care of. Our plan was to not make patients have to visit a doctor every time he or she is suspicious about a skin rash or mole. A preliminary scan would be very helpful in that case, just like checking blood pressure.”

“Much of the time,” says Habeeb, “patients aren’t aware of a disease until very late in the process. By the time they visit a doctor, it might be too late. With our application, patients can figure out if they need to see a doctor. Even if it’s a false positive, there is no harm in making that appointment.”

He says that the team first heard about Google’s GPU-powered VMs from Google engineers at the hackathon. “Having that information right away helped us train our model on [a] a complete dataset within the time-frame of hackathon,” he says. (Srivastava adds: “Thanks to the technical mentors from Google, we got started on Google Cloud’s Compute Engine within minutes.”)

Mallia set about designing the app architecture and helped solve specific problems the team encountered along the way.

At HackNYU, they learned how to quickly prototype a cross-platform mobile app (Android/iOS). And they retrained the last layer of Google’s "Inception" image classification model to categorize images based on their own dataset of skin moles.

Mallia says that the technology stack is a multiplatform mobile application backed up by REST APIs, which yield prediction using a previously trained deep learning model. Using a GPU-enabled cloud instance sped up the process of training the Deep Learning model significantly. The user interface is built for Android and iOS, says Sareen, with a backend written in Python Flask.

What’s next: applying machine learning to save lives

With its potential to serve as an important early detection tool, DermaScan is an ongoing and growing project. The team is now investigating “new datasets to use to extend it to other forms of skin rashes,” Mallia says. “And, we’re studying new machine-learning models to adopt.”

“We’re also working on figuring out how to reduce false negatives, extending the project to provide a probability—rather than a binary yes or no—and researching newer machine learning models to apply,” says Habeeb.

As they continue working on their prototype, the team is passionate about DermaScan’s future. “The massive potential of our project excites me,” says Habeeb. “If this prototype is adopted by hospitals in the United States, they could provide services that go well beyond traditional diagnosis. Apps such as ours, utilizing machine learning models, could help people figure out diagnostic steps, and medications for small problems. And if there’s a bigger problem, the application could notify the person to visit a doctor, which could save lives.”

“We envision a commercial product,” says Mallia. “For example, the app might send a photo taken of the skin mole directly to a dermatologist for a remote consultation.”

He is excited about the power of Google Cloud to transform public health on a global level. “Modern tools like machine learning,” he says, “and more specifically, deep learning, are allowing researchers and developers to change the world by solving real-life problems.”