The New(-ish) Frontier of Healthcare AI

I read an interesting article this week about FDA cleared AI tools on STAT. Their investigation was focused on potential bias in ML models relating to gender, race/ethnicity and geography. It’s an interesting topic, here’s the bottom line:

Of 161 AI products cleared by the FDA in recent years, only 73 disclosed the amount of patient data used to validate the performance of their devices in public documents. Only seven reported the racial makeup of their study populations, and just 13 provided a gender breakdown.

As a conservative and reactive regulatory authority, FDA is always behind – indeed I can’t think of a regulatory authority that isn’t. Take one example during early in the pandemic, researchers from the University of Washington wanted to repurpose a flu diagnostic for SARS-CoV-2 and sought CDC’s blessing, and later FDA. The New York Times reported:

C.D.C. officials repeatedly said it would not be possible. “If you want to use your test as a screening tool, you would have to check with F.D.A.,” Gayle Langley, an officer at the C.D.C.’s National Center for Immunization and Respiratory Disease, wrote back in an email on Feb. 16. But the F.D.A. could not offer the approval because the lab was not certified as a clinical laboratory under regulations established by the Centers for Medicare & Medicaid Services, a process that could take months.

That’s one case and I’m sure there are others. They’ll get it right but it’s slow to happen. It might be interesting to do an analysis of time-to-publication for FDA guidances. My sense it’s a slow, methodical jog to ‘final draft’.

At the moment, let’s shift gears to patients like you and me.

What’s on the Market?

Everyone loves the latest and greatest (healthcare lags–evidence is paramount first). Let’s take a quick look at the data:

## Rows: 161
## Columns: 10
## $ Product         <fct> PhysIQ Heart Rhythm and Respiratory Module, Loop Syste~
## $ Company         <fct> Phys IQ, Spry Health, CSD Labs GmbH, Stratoscientific ~
## $ Description     <fct> "detection of afib", "monitoring vital signs", "heart ~
## $ Product.Code    <fct> DPS, DQA, DQD, DQD, DQD, DQK, DQK, DQK, DQK, DRG, DSH,~
## $ Approval.Type   <fct> 510k, 510k, 510k, 510k, 510k, 510k, 510k, 510k, 510k, ~
## $ Validation.Data <fct> "None reported", "24", "120", "None reported", "None r~
## $ Gender          <fct> None reported, None reported, None reported, None repo~
## $ Race.Ethnicity  <fct> None reported, None reported, None reported, None repo~
## $ Geographic      <fct> "None reported", "None reported", "None reported", "No~
## $ Approved.Date   <fct> 7/10/2019, 3/29/2019, 4/17/2019, 7/15/2016, 10/30/2017~

##                                 Product                    Company   
##  Accipiolx                          :  1   Zebra Medical Vision:  6  
##  AccuContour                        :  1   aidoc medical       :  4  
##  Acumen Hypotension Prediction Index:  1   siemens medical     :  4  
##  ADAS 3D                            :  1   Arterys Inc.        :  3  
##  Affinity                           :  1   RapidAI             :  3  
##  Ahead 100                          :  1   Siemens Medical     :  3  
##  (Other)                            :155   (Other)             :138  
##                                  Description   Product.Code Approval.Type
##  detection of intracranial hemmorhage  :  4   LLZ    :43    510k   :144  
##  cardiac monitor                       :  3   QAS    :16    De Novo: 14  
##  detection of pneumothorax             :  3   JAK    :15    PMA    :  3  
##  radiation therapy planning            :  3   QFM    : 8                 
##  automatic labeling of brain structures:  2   DXH    : 5                 
##  detection of afib                     :  2   DQK    : 4                 
##  (Other)                               :144   (Other):70                 
##       Validation.Data            Gender           Race.Ethnicity
##  None reported:87     N/A           :  8   N/A           :  2   
##  300          : 3     None reorted  :  1   None reported :151   
##  100          : 2     None repored  :  1   None reportted:  1   
##  240          : 2     None reporrted:  1   Yes           :  7   
##  249          : 2     None reported :137                        
##  260          : 2     Partial       :  1                        
##  (Other)      :63     Yes           : 12                        
##                        Geographic     Approved.Date
##  None reported              :108   2/28/2020 :  3  
##  1 site                     :  3   1/15/2020 :  2  
##  2 sites                    :  2   10/4/2019 :  2  
##  2 US sites                 :  2   11/30/2018:  2  
##  2 US sites, 1 international:  2   11/5/2019 :  2  
##  5 sites                    :  2   12/28/2018:  2  
##  (Other)                    : 42   (Other)   :148

Some tidying to do. Also it would be useful to cross-reference FDA product codes to know more about their classifications (e.g., Product Code DQA = Oximeter). Let’s pull that data in:

## Rows: 6,709
## Columns: 3
## $ PRODUCTCODE <chr> "BRW", "BRX", "BRY", "BSE", "BSF", "BSI", "BSJ", "BSK", "B~
## $ DEVICENAME  <chr> "Protector, Dental", "Stool, Anesthesia", "Cabinet, Table ~
## $ DEVICECLASS <chr> "1", "1", "1", "2", "1", "2", "1", "2", "2", "2", "2", "1"~

Almost 7,000 product codes! Interesting.

Got those, now let’s merge with our fda_ai-data dataset:

## Rows: 161
## Columns: 5
## $ Product     <fct> PhysIQ Heart Rhythm and Respiratory Module, Loop System, e~
## $ Company     <fct> Phys IQ, Spry Health, CSD Labs GmbH, Stratoscientific Inc.~
## $ Description <fct> "detection of afib", "monitoring vital signs", "heart murm~
## $ DEVICENAME  <chr> "Electrocardiograph", "Oximeter", "Stethoscope, Electronic~
## $ DEVICECLASS <chr> "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2"~

All done!

What types of products has FDA cleared?

The first thing I think of when it comes to ML/AI technologies in healthcare is radiology – this area has been touted as the easiest on ramp for ML models. I’d expect that AI/ML is entering other areas of healthcare. Let’s see what the data reveals:

At least in this dataset, radiology is overwhelmingly dominant, followed by cardiovascular and neurology.

We know when they were approved, let’s look at trends over time:

Final Thoughts

Some interesting trends. Radiology is indeed king (so far). Market entry (i.e. FDA clearance) is really just the beginning. What’s missing here is adoption and value to patients and providers. New tech tends to be expensive, an implicit promise of radical improvement–latest and greatest. Not always so.

My personal takeaways:

• 510k, at least in this dataset, is the most common path to market. These submissions are relatively easier as they only require evidence of of “substantial equivalence” to existing products (i.e., no clinical trials like PMA).
• Radiology is a proving ground AI/ML tech, but cardiology is next.
• As of today, 2018 was a turning point (40 new products)
• The first approval was in 2008 by IB Neuro, a dynamic susceptibility contrast (DSC) method for MRI to assess cerebral blood volume in brain cancer treatments.

As for translational research, I recommend reading this 2019 review article in Nature Medicine, authored by Eric Topol.

It will be interesting to see what the future holds!