The ATA Chronicle - November/December 2024 - 29

scalable) step in the process,
which may be supplemented
with manual analysis (i.e.,
slow, hard, and limited).
Based on personal
communication with the
developers, MATEO doesn't
store or use your data for
any purposes other than
computing the scores. But
there's no formal statement
to this effect on their websites.
So, use caution and keep in
mind that this excellent tool
was developed primarily for
educational purposes. This
said, I feel safe using it in my
translation work.
Some metrics allow
various options for the metric
parameter settings (under
" BLUE options, " " TER
options, " etc.). Don't worry
about them unless you know
what they mean. Simply use
the defaults.
If you're frustrated with
the relatively slow speed of
calculating neural metrics on
the MATEO site, you can create
an instance (i.e., a copy) of it
on HuggingFace, a repository
of open-source tools and
platforms for natural language
processing (you'll need to
create a free account there), or
simply install MATEO locally
and run it with Python.
(Obviously, you need some
programming skills for this.)
Use Cases
Despite the caveats, these
are wonderful tools! I used
MATEO in a 2023 presentation
for the Association for Machine
Translation in the Americas
that explored the translation
capabilities of large language
models (GPT-4, Claude-2,
and Bard).4 I also used it
successfully in my translation
workflow. It allowed me,
among other things, to select
the best two MT engines for
www.ata-chronicle.online
Figure 4: Levenstein and edit distances between two sentences
a massive (100K+ word) MT
post-editing project by letting
me know that one of the
engines successfully adapts
to an existing translation
memory (this is where
classical string-based metrics
such as BLEU and chrF were of
great help) with no significant
loss in general quality, while
the other engine does better
on terminology.
It's worth noting that
there is nothing sacrosanct
about machine and reference
translations. Any translation,
human or machine, can be
used as a " reference " to
evaluate another translation,
human or machine.
And getting under the hood
of all that is very empowering.
If you end up using these tools
in your work, and especially in
public presentations, please
cite their sources. The authors
worked hard to develop them,
and their efforts were
generously supported by
several European Union
agencies.5
NOTES
1
For a brief introduction to all these metrics and a reference
to the original work and code, see https://mateo.ivdnt.org/
Background.
2
MATEO is not the only such toolkit. Other, less advanced
options include Asiya Online and the Evaluate function of
MutNMT, which I reviewed in the September/October 2022
issue of The ATA Chronicle ( " MultiTraiNMT: Neural Machine
Translation for Everyone " ).
3
Vanroy, Bram, Arda Tezcan, and Macken, Lieve. " MATEO:
MAchine Translation Evaluation Online. " In the proceedings
of the 24th Annual Conference of the European Association
for Machine Translation (2023), 499-500.
4
Here are the slides from my presentation at the 2023
Association for Machine Translation in the Americas
conference, " Domain and Terminology Adaptation with
Large Language Models: A Comparative User Study in the
Americas. " The video of this presentation is also available.
5
This article is part of a project supported by the National
Science Foundation (SES-2336713).
Yuri Balashov, CT is a professor of philosophy
and a faculty fellow in the Institute for Artificial
Intelligence at the University of Georgia. He is
also an ATA-certified English>Russian translator.
He is working on a project exploring cognitive,
linguistic, and philosophical dimensions of human and
machine translation. balashov.yuri@gmail.com
American Translators Association 29

https://mateo.ivdnt.org/Background https://mateo.ivdnt.org/Background https://asiya.cs.upc.edu/demo/asiya_online.php https://ntradumatica.uab.cat/ https://www.yuribalashov.com/Papers/MultitraiNMT.pdf https://huggingface.co/docs/transformers/en/tasks/translation https://www.yuribalashov.com/Papers/MultitraiNMT.pdf https://aclanthology.org/2023.eamt-1.52 https://aclanthology.org/2023.eamt-1.52 https://aclanthology.org/2023.eamt-1.52 https://aclanthology.org/2023.eamt-1.52 https://yuribalashov.com/Papers/AMTA_2023_Yuri-Balashov.pdf https://yuribalashov.com/Papers/AMTA_2023_Yuri-Balashov.pdf https://yuribalashov.com/Papers/AMTA_2023_Yuri-Balashov.pdf https://www.youtube.com/watch?v=yDiZ59d5-kY&t=16s https://www.nsf.gov/ https://www.nsf.gov/ https://amtaweb.org/ https://amtaweb.org/ http://www.ata-chronicle.online

The ATA Chronicle - November/December 2024

Table of Contents for the Digital Edition of The ATA Chronicle - November/December 2024