Medical Design Briefs - November 2024 - 6

Gen AI Isn't Ready
(for Medical Devices)
S
avvy
to
manufacturers looking
developers
give
their
medical
device software
and Software
as a Medical Device
(SaMD) a competitive
edge explore
ways to incorporate
technologies from
the wider tech industry - and no technology
is hotter than generative AI.1
Since the release of ChatGPT in November
2022, generative AI (Gen AI)
has become the new trend across almost
every sector of business. Organizations
are rushing to find ways to use Gen AI
in their products and operations, with
the goal of transforming their business
before their competitors.
While the medtech industry is starting
to get value out of using Gen AI, discussions
on whether live generative AI
algorithms could be put inside medical
devices have been more subdued. Unlike
other industries, where regulations are
looser (or nonexistent), medtech must
ensure that its software and devices comply
with FDA regulations and are built
under design controls using ISO 13485
and IEC 62304. In the past, we've found
safe and compliant ways to incorporate
emergent technologies into our products,
but as a technology, Gen AI presents
unique challenges to industry adoption.
It's Orthogonal's position - and
the consensus among many of our colleagues
- that barring specific use cases,
Randy Horton,
Orthogonal
Category
Medical
diagnostics
Drug
discovery
Virtual
health
assistants
Medical
research
Clinical
decision
support
6
Example Setting
AI-Rad
Companion
Insilico
Medicine
Radiology
Biotechnology
User
n Why Gen AI Isn't Ready for
Medical Devices
Like others in medtech, Orthogonal
has adopted technologies like Bluetooth
Low Energy and cloud computing into
our medical devices.2
medtech
and
Gen AI is not yet ready to be placed inside
medical devices. In this editorial, we
take a look at what makes Gen AI different
from other technologies, how it may
someday be used in our industry, and the
concerns its use raises.
With the help of
our colleagues and industry standards
organizations like AAMI, we've been
able to safely mitigate these technologies'
risks so that the benefit to patients
outweighs the uncertainties.
Generative AI is unlike any new technology
that has come before it. It's essentially
a black box with an infinite
number of inputs and outputs and designed
to try to answer any question,
making it impossible to guarantee an
outcome through repeated testing. In
medtech, if we can't test it, we can't validate
it. If we can't validate it, we can't
guarantee its safety when put into a
medical device.
At the moment, it's unclear whether
it would be practical for a device
manufacturer to incorporate Gen AI
into a device, due to the astronomical
costs required to prove out new methods
of validation. Outside of a major
health emergency where Gen AI is the
only solution (similar to how the development
of relatively untested mRNA
vaccines were accelerated to combat
COVID-19), it's unlikely that we'll see
Gen AI in the function of medical devices
in the near future.
Input Data
Radiologists Medical
images
Research
scientists
Sensely Web clinics
Anthropic
Patients
Target
proteins and
disease data
Output
Data
Text findings
Novel
molecular
structures
Personalization
Level
Individual
Semi-personalized
Workflow
Integration
Postimaging
Early-stage
research
Conversation Conversation Semi-personalized Patient
engagement
Laboratories
and academia
Glass AI
Point of care
Researchers Research
concepts and
data sets
Physicians
Patient data
Hypotheses
and
questions
Treatment
suggestions
Semi-personalized Idea
generation
Individual
Diagnosis and
treatment
n Isn't AI Already in Medical Devices?
Artificial intelligence (AI)/machine
learning (ML) algorithms, including
algorithms with generative
abilities, have increasingly become
part of medicine over the last decade.
An article in the Journal of Medical Internet
Research, " Generative AI in Medical
Practice: In-Depth Exploration of
Privacy and Security Challenges, " features
examples of current generative
AI/ML algorithms in healthcare (see
Table 1).3
AI/ML in the context of medical
devices work on wholly different principles
than ChatGPT and similar large
language models (LLMs). ChatGPT is
a general-purpose algorithm that attempts
to answer any question posed
to it, while medical device AI/ML are
focused on providing answers within
a tightly bounded set of parameters.
ChatGPT could tell you how to make
a PB&J sandwich, but an Al/ML algorithm
trained to detect wrist fractures
could not.
The FDA is the one who sets those
tight parameters on AI/ML algorithms
as a way to test, verify, and validate, and
mitigate their potential risks to patients.
Because Gen AI has no boundaries, the
current regulatory guidance for AI/ML
cannot be applied to Gen AI.
Though we're far off from an official
Gen AI policy, the FDA is opening the
floor to discussion on the topic at a public
meeting scheduled for November
2024. In the meantime, we can look to
the FDA's draft guidance on Predetermined
Change Control Plans (PCCP) as
an indication of their approach to updating
AI algorithms.4
Validation
Needed
High
High
Impact Risks
Improved
diagnostics
Faster
discoveries
Moderate
Increased
access
Low
Research
insights
High
Improved
outcomes
Reliability and
bias
Safety
and testing
requirements
Privacy and
misinformation
Human
Involvement
High
Moderate
Moderate
Misdirection Moderate
Overreliance
and bias
High
Table 1. Categories of generative artificial intelligence (AI) applications in healthcare. (Credit: Chen and Esmaeilzadeh/Journal of Medical Internet Research; jmir.org)
www.medicaldesignbriefs.com
Medical Design Briefs, November 2024
http://www.jmir.org https://www.medicaldesignbriefs.com
Medical Design Briefs - November 2024

Table of Contents for the Digital Edition of Medical Design Briefs - November 2024
