APR Sept/Oct 2023 - 32

» INSIDER INSIGHT
»
uses data to produce its output. As AI development in healthcare
grows it is essential that the guidelines on its development and use
keep pace and build trust in AI solutions. Similar to how the FDA
requires that detailed ingredients be displayed on the side of cereal
boxes to help consumers make informed health decisions, the same
transparency is needed in AI technology so patients and physicians
can make educated care decisions. Rather than nutrition information,
AI guidance should help verify that a model was developed with highquality
data representative of its intended patient population, as well
as disclaimers on its use.
There is promising news that the federal government clearly recognizes
this need. In 2021 the FDA issued Good Machine Learning Practices
(GMLP) for medical device development, which outlines ten guiding
principles to inform the development of AI. While these guidelines
provide a good foundation, enforcing science-backed, ethical
practices in AI innovation will take more than a good-faith agreement
on what developers should do. We must move from recommendations
to standards governing the back-end work in algorithm development
that we are all held to, as well as requirements for transparency into
how the algorithm was created and proof that the developer adhered
to these guidelines.
Adapting Regulation to Account for
the Nature of AI
While developing a strong foundation for an algorithm is an important
first step, AI should be constantly learning and improving, and thus
always providing the most up-to-date solution to all users. So, how
does the FDA adequately regulate technology that could change
months later? Imagine if every time your iPhone needed a new iOS
update, it had to go through a regulatory review process that could
take up to three years. Similarly, the FDA realizes that the process to
review and approve new AI devices does not move quickly enough
for the way these products function, leading to the use of outdated
and potentially unsafe algorithms and stymying innovation. In order
to keep up with the evolving nature of AI, the FDA has recently issued
draft guidance allowing AI devices to be deployed using the most
updated versions of their algorithms so long as developers submit a
Predetermined Change Control Plan (PCCP) that outlines how they
expect an algorithm to change over time and as more comprehensive
training data is employed. As long as the outputs from an updated
product meet agreed upon pre-defined criteria, developers can
submit a short PCCP at the time of their product submission rather
than a full new device application each time there is an update.
This guidance would have a particular impact on algorithms that
may have initially been trained with smaller or unintentionally biased
algorithms, enabling developers to retrain AI to improve performance
across minority groups without needing to seek additional regulatory
approval. AI is only as strong as the data it is fed, and the data that
is frequently used in algorithm development may be biased against
certain minority populations if they are underrepresented in clinical
research, the source of much of the training data. It is possible to start
32 |
| September/October 2023
off with a great algorithm that works perfectly well with one population,
but if it's not trained with diverse data sets, it will inevitably end up
not performing well once applied to other populations. For instance,
if one was developing an algorithm to identify cancerous tumors, and
the algorithm was only trained using data for stage 4 tumors, it may
not be able to detect stage 1 tumors with similar accuracy. A similar
example is with skin tone: an algorithm that is only trained with data
from patients with lighter skin may struggle with data from patients of
a darker complexion.
This draft guidance takes a critical step in addressing bias in AI devices,
but additional federal regulations are needed to ensure clinical
research datasets are representative of the broader population from
the beginning, and to make certain this sensitive data can be accessed
for AI development. While the availability of diverse data such as race,
gender and age is essential for training unbiased algorithms, some
state privacy laws, like the California Privacy Rights Act, may limit the
ability to collect or retain this type of information in the first place.
It will be essential for future federal AI regulations to find a balance
between addressing bias in algorithm development and protecting
patient privacy around the capture of their sensitive demographic
data. As AI tools become increasingly integral to how we research
drugs and deliver new treatments, we should emphasize inclusivity in
the devices our patients and pharmaceutical companies use, and the
laws governing their development, to help them reach their potential
and make healthcare a more equitable industry.
Defining Privacy Parameters to
Advance Data Quality
Patient privacy is an essential part of any clinical trial's success: patients
need to be aware of and feel comfortable with the procedures in place
to keep their sensitive health information safe. As protecting patients'
privacy is immensely important, many governments across states and
countries have varying ideas on how to do it best. This has led to a
patchwork of regulations that can not only slow down the pace of
research but ultimately erode patient trust and participation in clinical
trials. The lack of consistency in privacy regulations also means that
critical data that could be used to better train AI algorithms remains
siloed and inaccessible.
Across the globe, governments, technology, and healthcare
organizations must join forces to support the development of
streamlined international privacy regulations. This way, patients can
feel more confident about sharing their vital data in clinical research,
which would empower drug companies to create therapies that
more precisely meet the patient population's needs. Once suitable
ground-rules have been set, it will be much easier for data to flow
seamlessly across organizations and borders. With global guidelines
in place outlining the proper use of de-identification of patient data,
personal health insights from prior research can be used to support
and strengthen future clinical studies and AI development. In addition,
it is also important that future regulations permit researchers to
use diverse clinical data without the risk of trial delays or worse, for
APR Sept/Oct 2023

Table of Contents for the Digital Edition of APR Sept/Oct 2023
