IEEE Technology and Society Magazine - December 2021 - 44

Soft Law Governance of Artificial Intelligence
This includes that " [t]he program utilizes systems,
approaches, and techniques sufficient to ensure that
the representation is carried out with maximum effectiveness "
[47]. LSC identifies numerous indicators for
the effectiveness of legal representation, including
up-to-date technology, adequate access to experts
and litigation support systems, systems for the ongoing
evaluation of the effectiveness of legal work,
examining both results obtained and efficiency and
quality of methods used, and the effective use of available
technology to assist in services delivery.
LSC's criteria cite the ABA Standards for the Provision
of Civil Legal Aid, specifically Standard 2.10 on
the Effective Use of Technology: " A provider should
utilize technology to support efficient operations
and the provision of high quality and responsive
services " [48]. The comments to this standard state
that " [a] provider should cultivate a commitment to
innovation and should take advantage of technology
that can increase the scope of services it offers
to its constituents " [48]. " The provider should be
aware of emerging uses of technology directly in the
conduct of representation. " This includes " stay[ing]
informed of new developments and analyz[ing]
the degree to which new strategies for serving low
income communities may be possible as a result
of technological innovations " [48]. Additionally,
" [t]echnology planning should include an assessment
of the effectiveness of the provider's current
technology and of new technological advances that
would enhance its operation and delivery of services "
[48] (emphasis added).
IEEE and ABA principles for legalservices
delivery effectiveness
Numerous soft law programs and proposals for
the governance of AI have emerged. Researchers
identified 634 soft law AI programs in existence as of
December 31, 2019 [2]. Additionally, the IEEE and
ABA have published principles for law, including for
evaluating the effectiveness of legal services. These
principles provide a useful foundation from which to
consider mechanisms for the adoption and enforcement
of standards for evaluating the effectiveness of
AI for legal-services delivery.
IEEE EAD chapter 11
The IEEE, working with several hundred AI experts
across disciplines and jurisdictions, published Ethically
Aligned Design (EAD), a nearly 300-page report
44
that aims to prioritize human well-being for autonomous
and intelligent systems [38]. Chapter 11 of
EAD focuses on " Law, " specifically the impact of AI
on the practice of law and the potential benefits and
risks of incorporating AI into legal systems.
EAD identifies " unclear efficacy " of AI systems
used in justice systems as an impediment to the
informed trust needed for the adoption of AI systems.
" To formulate policies and standards of practice
intended to foster informed trust, " EAD suggests
identifying " principles over the entire supply chain
for the delivery of [AI]-enabled decisions and guidance,
including design, development, procurement,
deployment, operation, and validation of effectiveness,
that, if adhered to, will foster trust " [38]. EAD
examines how societies can increase trust in AI systems
by advancing " publicly accessible standards
of effectiveness, competence, accountability, and
transparency " [38].
EAD acknowledges that standards in each of
these four areas will not contribute equally in each
circumstance. " For example, in many applications
of [AI], a well-established measure of effectiveness,
obtained by proven and accepted methods,
may go a considerable way to creating
conditions for trust in the given application "
[38] (emphasis added).
EAD identifies two categories of stakeholders who
will be interested in effectiveness metrics: experts
(researchers, designers, operators, and advanced
users) and nonexperts (including legal professionals,
judges, litigants, communities, victims, and system
advocates). I recommend the addition of a third
category between these two for legal-expert users.
Legal-expert users should be required to develop a
functional understanding of AI systems and expected
to contribute their legal expertise to ensure that AI
systems are properly evaluated for effectiveness.
EAD identifies several criteria for optimal effectiveness
measures.
* Meaningful metrics that provide an accurate and
readily understood gauge of effectiveness, such
as recall and precision for electronic discovery.
* Sound methods, such as proper sampling and statistical
procedures.
* Valid data, vetted for bias, that is representative
of the actual data to which the AI system would
be applied.
* Awareness and consensus that the methods are
not only technically sound, but also " widely
IEEE Technology and Society Magazine
IEEE Technology and Society Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2021
