IEEE Technology and Society Magazine - March 2023 - 30

Feature
Without these modes of understanding, crucial
to the provision of mental health services, what is
at risk with the implementation of standalone AI
applications for mental health? Our own exchange
on January 2, 2023 with ChatGPT (https://openai.
com/blog/chatgpt/) can be found in the Supplementary
Material accompanying this article
(https://doi.org/MTS.2023.3241309) [70], asking a
series of typical mental health-related questions for
a single scenario.
Possible negative social implications
The possible negative social implications of
AI-driven standalone mental health apps are manifold.
They include the displacement of human interaction,
the shifting of responsibility for care and
management, issues of privacy, and questions of
cultural competencies. First, we must give significant
thought to the ways in which AI apps mediate and
displace human interaction. As Baldwin has written
on the ethics of technology in dementia care [47],
technological interventions can erode human intimacy
and transform a person from a recipient of care
to a user of technology. Additionally, and in contrast
to mental health apps that connect a user virtually to
a mental health professional, AI mental health apps
fundamentally change both the meaning of care and
the party responsible for the care. In traditional mental
health service provision, a person engaged in the
system anticipates receiving care. While positioning
themselves as entry points to care, AI mental health
apps almost universally advocate and encourage
self-management and individual responsibility, ultimately
leading to what is increasingly being considered
" patient work " / " information work " [48]. In
some ways, it is a type of " self-service " model.
Woebot, for example, dubs itself a " relational
agent " that forms a " therapeutic bond, " but as the
explanatory video on the website states " Woebot
doesn't do therapy, but he can be your guide to help
you figure out things on your own " [49, 0:18-0:21].
AI mental health apps, therefore, present a paradox:
the promise of 24/7 companionship and the expectation
of self-sufficiency. By passing the " management "
of nonnormative minds onto individuals themselves,
mental health conditions become increasingly individualized
and privatized, obscuring the broader
sociocultural contexts that contribute to both mental
illness and inaccessible care. Using these apps is
presented as a responsible choice among a suite of
30
options for mental health care. Yet, the accessibility
of other choices, such as formal care, is not equitably
distributed. By positioning self-management as both
desirable and accessible, the bootstraps, do-it-yourself
ethos of AI mental health apps paired with the
inaccessibility of formal care, potentially creates a
scenario in which self-management through AI mental
health applications may become an expectation
rather than a choice.
These AI applications are marketed as a way to
provide user privacy, a potentially salient feature
given the perceived or actual stigma that may be
associated with receiving mental health services in
the United States and elsewhere. Privacy may be an
important factor for many people that pursue digital
support [50]. But the privacy claim also appears
to attempt to legitimize the AI chatbot as a genuine
replacement for a human mental health worker, also
noted in direct dialog with the ChatGPT [70]. In fact,
the absence of human oversight and interaction with
these applications raises serious questions about
user impacts and safety. Users may be unaware, for
example, that the digital mental health apps that they
download onto their smartphone may be scrutinizing
their every online behavior and interaction (known
as passive monitoring), with the data being on-sold to
third parties in the name of care or being used to train
data sets to uncover even greater deep-learning patterns
and trends [51]. It does not take too much of a
stretch of the imagination to see how such functionalities
are contradictory, but also particularly intrusive
to someone who is at risk. The embodied experience
of mental illness becomes reduced to a set of eyeball
movements, the steadiness of the head on shoulders,
and a biometric photograph [52]. The extraction of
those data becomes the currency of care. There is also
nothing to stop AI chatbot companies from directing
users/mental health patients to paid advertising, such
as what happens on Psychology Today's website [53].
If the world's largest mental health and behavioral science
website engages in the creation and distribution
of marketing lists with third parties and incorporates
cookies and web beacons widely on its website [54],
as well as shares aggregate-level data with companies
outside Psychology Today, it is an obvious possibility
that AI chatbots also will be part of that value chain.
Presently, there is no clear indication that oversight
and accreditation bodies such as the American
Psychological Association (APA) or the American
Board of Professional Psychology (ABPP) intend to
IEEE Technology and Society Magazine
https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://doi.org/10.1109/MTS.2023.3241309
IEEE Technology and Society Magazine - March 2023

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2023
