IEEE Technology and Society Magazine - December 2021 - 43

an AI contract review tool correctly identified a contract
provision as a " choice of law " clause. These factors
complicate effectiveness evaluations outside of
TAR for electronic discovery. This may explain why
there are very few public examples of effectiveness
evaluations of AI for legal-services delivery. Nevertheless,
we can learn from these examples.
In February 2018, LawGeex released a study
in which it found that its contract review tool outperformed
several lawyers it had assembled when
identifying clauses in five nondisclosure agreements
(NDAs) [42]. LawGeex reported that its AI
tool achieved a 94% " accuracy " rate, compared to
85% on average for the 20 human lawyers that they
selected. (Unfortunately, LawGeex reported " accuracy "
only. Such studies should report recall, precision,
and the F-measure, the harmonic mean of
precision and recall.) LawGeex tasked the lawyers
with labeling clauses in NDAs, which is arguably
similar to the issue-spotting a lawyer does when
reviewing a contract. The study asked the lawyer
reviewers to ignore what they knew about specific
issues and instead apply definitions of specific contract
clauses. Despite these constraints, which limit
the study's external validity, and the weaknesses in
methodology, this study provides a general framework
that could be applied to evaluate the effectiveness
of legal-services delivery tasks.
In February 2019, QuisLex published a study of
ten AI tools that extract text from contracts, without
identifying the specific tools [43]. Several tests
focused on the tools' effectiveness at identifying and
extracting specific clauses from contracts. QuisLex
reported recall, precision, F-measure, and accuracy
for each task and created a scale to report partially
correct outputs. These AI tools, with out-of-the-box
models (i.e., no customer model training), scored
from 58% to 79% on recall, from 84% to 99% on precision,
from 72% to 81% on F-measure, and from 29%
to 80% on accuracy. Four tools were tested after custom
model training, producing scores of 35%-96%
on recall, 47%-100% on precision, 41%-98% on
F-measure, and 35%-97% on accuracy. The wide
range of scores demonstrates the need for effectiveness
evaluations. QuisLex also tested an " untrained
human, " who scored 65% on recall, 99% on precision,
78% on F-measure, and 79% on accuracy. QuisLex
said that it used " entry-level resources " for the
" untrained human " reviews to create a baseline for
comparison, but it did not provide much additional
December 2021
information about how it did this. Aside from the
shortcomings of its attempt to establish a human
baseline, the QuisLex study illustrates a sound methodology
for evaluating the effectiveness of AI tools
for legal-services delivery.
Other notable contributions to testing the effectiveness
of technology for legal services include Substantive
Legal Software Quality-A Gathering Storm,
by Marc Lauritsen and Quinten Steenhuis [44] and
7 Questions Lawyers Should Ask Vendors About
Their AI Products by Grossman and Morrison [45].
Lauritsen and Steenhuis focus on how sound software
development processes, including testing, will
contribute to improving the quality and accuracy of
applications that provide legal guidance. Grossman
and Morrison [45] advise, " before you license the
tool, it is imperative to know what empirical support
there is that the software that you are about to purchase
is valid and reliable. " For example, research
by Diana Koppang and Jeremy Sullivan shows significant
variance in descriptive statistics obtained from
different legal research platforms [46]. Their findings
illustrate the need to understand data inputs for AI
tools in the process of evaluating effectiveness.
Legal Services Corporation
performance criteria requiring
effectiveness
The Legal Services Corporation (LSC), the largest
funder of civil legal aid in the United States, has
established performance criteria for ensuring that
the programs it supports " provide high-quality legal
assistance " [47]. The criteria are intended to satisfy
the requirements of " effective " and " economical "
delivery required by Section 1007(a)(3) of the
Legal Services Corporation Act [47]. LSC defines
" effectiveness " as focused on the results achieved
and " economical " as " trying to achieve a particular
result as efficiently as possible " [47]. The LSC criteria
underscore the importance of evaluating effectiveness.
At the same time, the LSC criteria could
benefit if standards for evaluating AI effectiveness
were available for incorporation into its performance
criteria. LSC could use those standards to
accelerate innovation and technology adoption by
the programs it supports.
LSC's criteria for effectiveness require that a " program
conducts its direct legal representation ... in
an effective and high-quality fashion which comports
with relevant " rules and requirements [47].
43
IEEE Technology and Society Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2021
