IEEE Technology and Society Magazine - December 2021 - 42

Soft Law Governance of Artificial Intelligence
evaluations to demonstrate TAR's effectiveness
and judges who endorsed and sometimes mandated
its use. Once TAR was proved effective,
clients demanded that their lawyers use it to take
advantage of cost savings.
In the United States and several other jurisdictions,
litigants engage in a process of " discovery, "
which includes producing relevant information to
adversaries. With the explosion of electronically
stored information, even an ordinary business dispute
can involve millions of pages of documents.
Until about 20 years ago, lawyers would assemble
teams of junior lawyers to manually review enormous
volumes of paper documents. Over time, lawyers
began to use technology platforms to manage
the review of information and began to replace
junior lawyers with contract lawyers. Even then, the
costs were considerable, leading clients to push for
cheaper means of review and production. Before
legal professionals and courts could adopt TAR for
electronic discovery, however, they needed to determine,
" Does it work? "
In 2006, the U.S. National Institute of Standards
and Technology (NIST), through the Text REtrieval
Conference (TREC) Legal Track initiative, began
studies to evaluate the effectiveness of AI for electronic
discovery. Effectiveness was determined
based on two commonly used information retrieval
statistics: recall (the percentage of all relevant documents
retrieved) and precision (the percentage of
documents retrieved that are relevant). The IEEE's
Ethically Aligned Design (EAD) report cites the
TREC studies as a shining example of how science
can be used to advance the " truth-seeking protocols
of the law " [38]. EAD lauds TREC for creating
metrics to quantify effectiveness in a way that practitioners
could easily understand, establishing benchmarks
for applying AI to a real-world challenge, and
demonstrating the importance of multidisciplinary
collaboration by bringing together participants from
scientific and legal backgrounds.
Additional research advanced the TREC findings,
such as Grossman and Cormack's [40]
research findings, that certain TAR systems outperformed
human review. Their work illustrates
the importance of establishing the right baseline
for comparison. TAR did not need to be perfect; it
simply needed to outperform the manual review.
Today, courts in the United States, Ireland, the
United Kingdom, and Australia have approved
42
TAR for electronic discovery, and TAR is generally
accepted in these and other jurisdictions.
Nevertheless, evaluating the effectiveness of AI
for electronic discovery cannot yet be declared a
complete success. There is no consensus regarding
reasonable TAR processes and results. Despite
the proliferation of principles for effective TAR
and the availability of toolkits for evaluating
effectiveness, standard practice does not include
effectiveness evaluations [41]. In their continuing
work, Grossman and Cormack show that not all
TAR methods in use today are effective. They say
that legal professionals must " identify reasonable
tools, procedures, and validation protocols, based
on the best available evidence coupled with their
own investigations " [41]. Grossman and Cormack
[41] propose that, " [i]n a perfect world, a recognized
body would set standards for the application
of TAR technology in eDiscovery and would
certify particular tools and protocols for adherence
to those standards " .
Effectiveness evaluations of AI for other
legal-services tasks
Evaluating the effectiveness of AI tools for legal
services other than electronic discovery presents
additional challenges. Electronic discovery takes
place in a closed system. A complete corpus of
documents is available for training and testing the
AI system. Relevance is defined, the system trained
on examples from the corpus, and the outputs produced
by TAR and human review evaluated against
the corpus for effectiveness. Additionally, the threshold
required to find TAR effective is far less than
perfect, because relevance is a vague concept. With
human review, it has been shown that an inter-rater
agreement between two reviewers on documents
labeled " relevant " will not exceed 70% [41]. Therefore,
in comparison to a " gold standard " review
against which the output of an AI review is evaluated,
the positive agreement in documents labeled
relevant would not be expected to exceed 70% [41].
" Relevance is in the eye of the beholder " [41].
The use of AI for other legal-services tasks usually
involves " open systems. " In those cases, the full population
of possible training and testing examples is
unknown and unavailable. Additionally, for many
legal-services tasks, effectiveness can be measured
against an objective standard for correctness, at least
theoretically. For example, we can evaluate whether
IEEE Technology and Society Magazine
IEEE Technology and Society Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2021
