IEEE Technology and Society Magazine - September 2023 - 9

Klein [5] then identifies four hallucinations about
generative AI (rather than by generative AI), which
are that this new technology can solve the climate
crisis, deliver wise governance, and liberate people
from drudgery, while its creators (BigTech) can be
trusted " not to break the world. " Arguably, none of
these " hallucinations " are true; but surely the last is
the least believable, and most problematic.
Six talking points
The problem is that it is not exactly " breaking the
world " (a phrase that carries those apocalyptic undertones
which need to be avoided), but " remaking the
world, " in particular, remaking the world such that it
is even further entrenched in asymmetric power hierarchies
based on unearned privilege [6], deep inequality,
and no common knowledge to democratize.
So, at the risk of producing " yet another article about
generative AI, " this contribution to a conversation
about remaking the world in undesirable ways has
six talking points: fraud, greed, sustainability, theft,
and cognitive deskilling (in two dimensions).
The potential for fraud has already led to the
verbification of " ChatGPT, " as in " you've been
chatgpted, " with implied meaning " you've been
misled by chatbot. " The phrase has been used in the
context of a researcher responding to a request for
a paper-the requester has almost certainly asked a
chatbot for papers by the researcher and the output
looks plausible, and even interesting: it is difficult to
criticize the " imagination " in the " hallucination " -
but it is still a complete confabulation. There are
other limitations too, including context, reasoning,
experience, and drift. Ask a stupid question, get a
stupid answer, it is said; but for chatbots, perhaps,
this should now come with a warning caveat interrogans
(asker beware): ask a sensible question, maybe
get a stupid answer.
This makes it clear that a generative AI chatbot
is not functioning in exactly the same way as a
search engine or some putative " smart " user manual.
Although the recent OpenAI technical report warns
that " Great care should be taken when using language
model outputs, particularly in high-stakes contexts "
[7, p. 10].2 This is not just about the democratization
of knowledge: it could also become the democratization
of misinformation (where in this context
2There is an earlier warning of the limitations of GPT models and the need for care
on the second page of the report; but seriously, given how few people read the Terms
and Conditions before agreeing to install software, how many people are going to
read the tenth page of a technical report calling for " great care " ?
" democratization " simply means access to the capability
to spread misinformation). The risk is that
should LLM become a widely available technology
and the costs of training (considered later) become
less prohibitive, that chatbots would be trained on
specific datasets and used for the democratization of
disinformation, that is, the spread of misinformation
(false facts) with the deliberate intention to mislead
or deceive.
This is no longer willful ignorance [8]: it is willful
bias. Two of the key features of Nowak's [9] Regulatory
Theory of Social Influence (RTSI) are, first, that
there is a symmetry in information seeking: not only
do sources seek targets to influence, but also that targets
seek sources by whom to be influenced. Second,
targets delegate information processing to sources
for reasons of cognitive efficiency. The combination
of deliberate bias in the training datasets, these
two features of RTSI, and confirmation bias [10] is
particularly disturbing: it could lead to the fragmentation
of civil society into disparate epistemic universes,
as groups choose to believe whatever " facts "
or narratives are generated by their preferred chatbot.
In this sense, generative AI could then be the
first postmodernist technology, and we need a new
critique of " obscurantism " (i.e., if obscurantism is
making knowledge inaccessible, then training LLM
on biased data is making nonknowledge accessible).
A standard internet-oriented business model
is to launch a service for free. Then, if that service
becomes so successful and dominant, given how the
network effect at the application layer of the internet
can create a de facto monopoly, the focus is no
longer service provision for the common good, but
relentless monetization for revenue maximization.
This is often achieved either by the introduction of
advertising or by subscriptions to tiered services. In
the displacement of the common good by greed,
both come with substantial risks.
In the first case, advertising, it is evident that some
chatbots do have guardrails so that the scope of questions
that are answered do have some limitations,
although, to the outsider, it is not clear where the
boundary between chatbot and guardrail is architecturally
drawn. Product placement and industrial
sponsorship are commonplace in the movies and
the arts, but there is no reason to suppose that product
references cannot be surreptitiously inserted
into a generated text or image. Given how Google
has turned knowledge search into a bunfight for
September 2023
9
IEEE Technology and Society Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2023
