IEEE Technology and Society Magazine - March 2023 - 31

engage in the design of ethical standards for the AI
chatbots, or any standards for that matter [55]. There
is a need for such organizations to become more
vocal about the methods of engagement and also
importantly, about the use of the collected data [56].
There are serious questions to be asked about data
rights and the role of developers, methods, and values
in this context, particularly as they pertain to the
use of AI for and with potentially at-risk populations.
Even though people may be accustomed to typing
their deepest secrets into that Google Search
Box, there is a crucial difference in this setting: an
emotional investment and an expectation of care.
This should also remind us that these online expressions
fuel product placement, endorsements, and
advertisements using adwords-style algorithms. If
I present as someone who requires certain mental
health therapies, the chances are that the app will
know where to get these therapies and how to on-sell
them, akin to an in-gaming purchase of armor for an
avatar and more. In the exchange with ChatGPT on
January 2, 2023, the chatbot could not provide an
online directory listing of mental health professionals,
such as a psychiatrist or psychologist, who were
" near me " in Tempe, AZ, USA, because it claimed it
was unable " to browse the internet, " but it did identify
Psychology Today and GoodTherapy as places
the patient should visit online to help with locating
a mental health professional (see Appendix A in the
supplementary material for a full transcript [70]).
The longer our written exchange with ChatGPT
went on, the more redundant the information being
presented by the chatbot became [70]. It repeated
itself over and over again, and any human in the
same situation would not likely feel that they were
being heard. There were also some contradictions in
what ChatGPT had to say that could be very confusing
to people living with anxiety or depression. For
example, someone communicating with ChatGPT
might wonder whether or not the bot remembered
a prior conversation in a " session. " Or a user might
wonder what ChatGPT considers to be confidential
in a discussion, or what the bot is open to reporting
to authorities. ChatGPT itself said it might be prone
to inaccuracies in the information it was presenting
and that it could not validate any information it was
putting forward in response to a comment or question.
And while ChatGPT reminded the human it
was merely an " AI language model... designed to
process and generate text based on the data that [it
March 2023
had] been trained on, " on numerous occasions, it
provided what seemed like medical determinations,
such as suggesting the human was having " hallucinations "
which it noted was a " symptom of psychosis. "
At the same time, it said: " I am not a licensed
mental health professional and am not qualified to
provide therapy or counseling. "
An additional criticism of ChatGPT relates to the
bot's acknowledgment that it did not possess " feelings
or emotions, " but that it then used phraseology
like " I'm glad I could help " and " Take care! " When
the human responded: " I didn't say you helped, "
ChatGPT retreated: " I apologize if my previous message
was misconstrued... " How could a self-professed
" nonemotive, " " unfeeling " thing exchange human
pleasantries? The chatbot is anything but human,
anything but caring, and anything but empathic.
What might this mean for the person seeking support
for their mental health? Does it make the human
more susceptible or vulnerable when the machine is
being driven by an algorithm that can be optimally
" tweaked " for a variety of purposes? Despite the
potential of these new apps, it was obvious, at every
turn, that ChatGPT was wiping its hands clean of any
possible legal liability.
Furthermore, as a stopgap, entry point, or
replacement for more formal care, AI mental
health apps do not appear to be particularly
reliable. A 2020 meta-analysis [58] of studies of
smartphone apps for depression found dropout
rates to be nearly 50% after accounting for bias.
A 2021 systematic review [59] of user engagement
with digital mental health interventions found
multiple barriers to use, including technological
literacy, uncertainty about the claims being made,
and costs associated with use, including monthly
subscription fees or the need for ongoing Internet
access. Ironically, the review also found that
while people experiencing more significant symptoms
were more likely to use digital interventions,
experiencing those more significant symptoms
was associated with users having more difficulty
engaging with the mental health apps.
Another potential issue that has received little
attention is the ability of these applications to provide
culturally and linguistically competent resources
and support. In the United States, nonwhite communities
have faced greater stigma and structural
barriers to securing appropriate mental health services
[60], as have other marginalized communities,
31
IEEE Technology and Society Magazine - March 2023

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2023
