In a rush to find solutions for the Covid-19 pandemic, researchers are deploying machine learning algorithms to trawl through data that might give us more clues about the virus. Some claim to have identified potential treatments based on the data, while others are using it to screen patients or identify those at highest risk.

But, like their vaccine and drug counterparts, many of these algorithms are still unproven. With hundreds of research articles describing the use of artificial intelligence or machine learning — many of them preprints — it can be difficult to sort out which ones are most effective.

“I’ve heard a lot of hype about machine learning being applied to battling Covid-19, but I haven’t seen very many concrete examples where you could imagine in the short- or medium-term something that is going to have a substantial effect,” said John Quackenbush, chair of the Department of Biostatistics at the Harvard T.H. Chan School of Public Health, in a phone interview.

Any good model requires good data, and that can be a challenge to find in healthcare. Given that Covid-19 is a new disease, that limits the amount of information researchers have. On top of that, most clinical data is locked up in health record systems, which often have different ways of recording it.

“Everything that we’re doing gets better with a lot more well-annotated datasets,” said Dr. Eric Topol, director of the Scripps Research Translational Institute, who published a book on AI in healthcare. “In the U.S., we don’t have centralized data. Here we are at the epicenter and all of our healthcare data is fragmented.”

On the other hand, as datasets get larger, they become “noisier.” For example, a model that screens Covid-19 patients for temperature might be reasonably effective. But expanded to the general population, “it’s a terrible predictor,” Quackenbush said.

Still, both were cautiously optimistic about using AI in some settings, such as determining which patients face a higher risk from Covid-19, opening an opportunity for communication with their physician.

Searching for a treatment
In early April, drugmaker Eli Lilly announced it would launch a trial of its existing rheumatoid arthritis treatment, baricitinib, in severely ill Covid-19 patients.

The drug was identified by a British startup, BenevolentAI, which used natural language processing to skim through millions of papers and create a database of biological processes related to the novel coronavirus. From there, they identified baricitinib as a potential treatment because of two key characteristics: its anti-inflammatory properties might help temper the body’s hyperactive immune response to the virus, and it seemed like the drug might be able to prevent viral infection.

As a counterpoint, a group of rheumatologists that had treated patients in Lombardy, Italy, cautioned about potential adverse effects from the drug. Its FDA black box warning indicates patients taking the drug may face an increased risk of developing serious infections.

“In conclusion, we believe that, beyond the intriguing opportunity to directly block the penetration of SARS-CoV-2 into the cell, the use of baricitinib in susceptible patients with ongoing pneumonia associated with Covid-19 should be considered with extreme caution,” they wrote in a published article in the Lancet. 

Several companies are using machine learning to predict what drugs might work, but Quackenbush said he had seen little information on what methods they used and what data was input into the model.

“It’s easier to say I found something. It’s much harder to say it’s going to work,” he said. “Is it worth investigating? Probably. But is this going to be the magic bullet? I tend to be much more skeptical.”

AI for screening
Companies are also putting image recognition to the test to detect Covid-19 from CT scans.

At Mount Sinai, researchers were able to train a model on lung scans from more than 900 patients. The scans, received from collaborators at hospitals in China, were split between Covid-19 positive and Covid-19 negative cases.

The researchers published their results in Nature Medicine earlier this month, showing the model identified 84.3% of positive cases, while a radiologist identified 74.6% of cases. Zahi Fayad, Director of the BioMedical Engineering and Imaging Institute at Mount Sinai’s Icahn School of Medicine, said the AI model could provide a “second opinion” when a CT scan is negative or shows nonspecific findings.

Infervision, a company with offices in Beijing and Philadelphia, had originally used its machine learning solution to detect lung cancer in CT scans. In January, it began working with Wuhan Tongji Hospital and Zhongnan Hospital of Wuhan University to create a version  to detect Covid-19. The company said the model had been trained on thousands of Covid-19 cases, and exhibited great sensitivity and specificity, though it hasn’t yet published the results.

Experts have raised a few concerns with these models. First, they need to ensure that they have good datasets with enough different instruments and enough hospital settings to be useful in a real-world setting.

There’s also the question of whether or not physicians will actually use the tool. Most patients going into the hospital will receive a standard swab test; the CDC does not currently recommend using CT scans to diagnose Covid-19.

“If they send you for a CT scan, it’s probably because you have a large list of symptoms already,” Quackenbush said. “If it’s positive, what are they going to do? I would bet physicians are going to wait for a standard PCR test or antibody test to really confirm what this is.”

These concerns were reflected in a paper written by a group of researchers with the United Nations, the World Health Organization and the Mila-Quebec AI Institute taking a broader look at how AI is being used to fight Covid-19.

“While encouraging results have been achieved by many medical imagery-based AI diagnostics methods, in order for these methods to be used as clinical decision support systems, they should undergo clinical investigations and comply with regulatory and quality control requirements. In particular, their performance should be validated on a relevant and diverse set of training, validation, and test datasets, and they should demonstrate effectiveness in the clinical workflow,” they wrote. “We note that most of the papers we reviewed lacked provisions for these measures, relying on small and poorly-balanced datasets with flawed evaluation procedures and no plan for inclusion in clinical workflows.”

The Biggest Opportunity
The most promising use case for AI at this point is in helping manage patients that face an increased risk from Covid-19. Though we still don’t know why some people get so severely ill, researchers have been searching for potential indicators.

Using a retrospective analysis of blood samples from 485 patients in Wuhan, researchers identified three biomarkers that could predict mortality:  lactic dehydrogenase (LDH), lymphocyte and high-sensitivity C-reactive protein (hs-CRP). They recently published their results in Nature Machine Intelligence. 

“It turns our attention to LDH, to make sure we know who’s at high risk,” Topol said. “I thought that was a good AI contribution, one of the best to date.”

Triaging patients in a clinical setting can be a bit more complicated. The information must be validated because it could make the difference between a patient staying home or going to the emergency room.

“The problem with this condition is that some people decline very rapidly,” Topol said.

Aaron Sheedy, COO of Xealth and entrepreneur-in-residence for Providence Health & Services, worked with remote monitoring startup Twistle to implement a system to keep track of patients calling in with Covid-19 symptoms. For example, it alerts a nurse if the patient has a fever above 101 degrees. Providence St. Joseph Health and the University of Pittsburgh Medical Center both currently use the system.

“What I can tell with our current customers, their primary challenge is really dealing with their own ability to maintain relationships with patients who still need care because of patient complexity. You still need to help patients manage diabetes, COPD, or (congestive heart failure),” he said. “In that area, I think I can see the role AI might have in taking datasets across a lot of different resources — A1C readings, temperature, or reporting pain. All of those things you can weave in together to create some big datasets.”

There’s an especially big opportunity right now to help patients with chronic conditions, many of whom are foregoing urgent care out of fear of getting Covid-19 at the hospital. HealthReveal, a startup that pulls in patients’ health data to ensure they are receiving treatment consistent with the best guidelines, is now using its platform to proactively reach out to high-risk patients.

“The worry is all of these people are foregoing care because of the stay-at-home orders and social distancing,” COO and President Lori Evans Bernstein said. “Maybe they are hypertensive and we know that from the data. We see that they’re supposed to be on a medicine and they’re not on it. We would suggest a medicine for their hypertension. Or maybe they were taking it, stopped taking it, and they haven’t gotten a refill because of Covid.”

Quackenbush said these types of tools could be especially helpful if cities face a second wave of Covid-19 cases.

“If we get to the point where we have access to the right data, and can really train good models to really understand which patients need to be monitored most closely, I think we’re in a good place to respond,” he said.

 

Photo credit: from2015, Getty Images