Medical Design Briefs - November 2024 - 7

n Potential Applications of Gen
AI in Medical Device Software
Development
While implementation in medical
devices is off the table for now, we
anticipate that Gen AI will soon work
its way into the production of medical
device software and the operations of
medical device manufacturers. In a
recent webinar, Orthogonal and industry
expert Clay Anselmo shared
some potential applications:
* As a software coding assistant, an
LLM could point out where developers
are missing specific items in
code, as well as potentially write first
draft code.
* As a text producer, it could generate
a first draft of a 10,000+ page
regulatory submission based on the
existing body of highly templated
regulatory documentation.
Fig. 1 - Aside from the top left, none of these are accurate.
(Credit: AI Weirdness)
* As an image generator, it could derive
additional training data from a fixed
data set.
n Why We Need to Be Careful
The potential applications of Gen AI
come with a massive caveat - the need
for robust quality control and assurance
methods from human developers
to catch problems in the output. Until
the day comes that we can peer into that
black box and understand exactly how
algorithms like ChatGPT work, medtech
needs to approach this technology
with even more caution than typical for
a new computing technology entering a
medical device.
n Bias in AI
Algorithms are not neutral. They
extrapolate outputs based solely on
what they are fed, and if their inputs
are heavily skewed or biased in some
way, the outputs will be too. For example,
if an algorithm for detecting heart
disease is trained solely on images of
cisgender men, it will be biased against
finding signs of heart disease in cisgender
women.
The issue of bias affects both AI/ML
and Gen AI, but it is heightened in the
latter. To answer any and all questions,
popular Gen AI LLMs like ChatGPT
were trained on undocumented, unstructured
data indiscriminately scooped
up from the Internet. Training an algorithm
this way may make it powerful,
but it gives the manufacturer no ability
Medical Design Briefs, November 2024
to control quality or see what biases are
being baked into the code - biases that
will inevitably be reproduced.
n Hallucinations: A Serious Risk
Gen AI is prone to making things up,
or hallucinating. The blog AI Weirdness
highlights writer Janelle Shane's
attempts to get consistent, reliable information
out of ChatGPT.5
She asked
ChatGPT to draw an illustrated graphic
of basic geometry shapes. The results
speak for themselves (see Figure 1).
Harmless hallucinations can be humorous,
but in the context of medical
device software, where decisions are
being made about patients' health,
they are inexcusable. We cannot legally
or ethically allow our products to give
incorrect information to users or simply
" wing it. " Even if Gen AI was limited
to lower-risk activities, like guiding a
user through device setup, there is still
potential to harm patients by giving
them unhelpful or misleading answers.
n Environmental Impact of Gen AI
All computing technologies need energy
to function, but Gen AI algorithms
use an incredible amount, with one estimate
stating that ChatGPT consumes the
equivalent amount of energy to 33,000
U.S. households. Computers running
Gen AI also require massive quantities of
fresh water to cool processors and generate
electricity.6
Environmental conditions are a
social determinant of health. As our
industry moves toward making our
www.medicaldesignbriefs.com
products and processes more sustainable,
it's worth reflecting on the
trade-off between the potential of
Gen AI to improve health outcomes
and the possible negative impact
on energy consumption,
natural
resources, human health, and the
environment.7
n Conclusion
Gen AI is an impressive technology
with seemingly limitless applications.
We've barely begun to scratch
the surface of what it can do. But it's
this very limitlessness that makes it a
poor fit for use inside of a medical
device. Gen AI won't be ready to be
used in the functioning of medical
device software or SaMD until medtech
manufacturers and developers
can confidently demonstrate its safety
and effectiveness. We owe it to our stakeholders,
patients, providers, and payers
to take a methodological and thoughtful
approach to addressing these barriers
so that Gen AI can be safely used to
enhance the safety and effectiveness of
medical devices.
References
1. Bernhard Kappe, " Software as a Medical Device
(SaMD): What It Is & Why It Matters, " Orthogonal,
Feb. 7, 2024, https://orthogonal.io/insights/
samd/software-as-a-medical-device-samd-basics/.
2. Bernhard Kappe, " AAMI's Groundbreaking Consensus
on Cloud Technology in Medical Devices, "
Sept. 28, 2021, https://orthogonal.io/insights/
cloud/aami-releases-consensus-on-cloud-formedical-
devices/
3. Y. Chen, et al., " Generative AI in Medical Practice:In-Depth
Exploration of Privacy and Security
Challenges, " Journal of Medical Internet Research,
2024: Vol 26; https://www.jmir.org/2024/1/
e53008/PDF.
4. " Marketing Submission Recommendations for
a Predetermined Change Control Plan for Artificial
Intelligence/Machine Learning (AI/
ML)-Enabled Device Software Functions, " Draft
Guidance, FDA, April 2023, https://www.fda.gov/
regulator y-information/search-fdaguidance-
documents/marketing-submissionrecommendations-predetermined-changecontrol-plan-artificial.
5.
Janelle Shane, " An Exercise in Frustration, AI
Weirdness, The Strange Side of Machine Learning,
https://www.aiweirdness.com.
6. Kate Crawford, " Generative AI's environmental
costs are soaring - and mostly secret, " Nature,
20 Feb 2024, https://www.nature.com/articles/
d41586-024-00478-x
7. " Social Determinants of Health (SDOH), CDC,
January 17, 2024, https://www.cdc.gov/about/
priorities/why-is-addressing-sdoh-important.html.
This editorial was written by Randy
Horton, Chief Solutions Officer, Orthogonal,
Chicago, IL. For more information,
e-mail him at rhorton@orthogonal.io or
visit www.orthogonal.io.
7

https://www.orthogonal.io/insights/ https://www.orthogonal.io/insights/ https://www.jmir.org/2024/1/ https://www.fda.gov/ https://www.aiweirdness.com https://www.nature.com/articles/ https://www.cdc.gov/about/ https://www.orthogonal.io https://www.medicaldesignbriefs.com

Medical Design Briefs - November 2024

Table of Contents for the Digital Edition of Medical Design Briefs - November 2024