IEEE Robotics & Automation Magazine - September 2015 - 145

Our proposed format was subsequently adopted for the RC
2014 finals. The divergence from statistically robust team
rankings was equivalently small:
d 1 (r a, r c) 2014 , d 1 (r p, r c) 2014 = 4.

Format
RC 2012
RC 2013
RC 2014 (Proposed)

(4)

To statistically validate that the proposed competition format is significantly more appropriate than those adopted at RC
2012 and RC 2013, 10,000 tournaments were generated for
each format by randomly sampling the game results from the
28,000-game round robin. For each tournament, the L 1 distance d 1 (r a, r b) [see (1)], was calculated to capture the discrepancy between the tournament and the true team rankings.
These results are presented in Figure 4 for the top eight teams
from RC 2012-2014. It is evident that the proposed format
yields more statistically robust rankings (i.e., smaller L 1 distance) than the formats adopted in previous years.
In addition to comparing the accuracy of team rankings
under different competition formats, it is interesting to
compare the team performance against a consistent benchmark. Before 2013, it was commonplace for the simulation
league teams to optimize their performance against the default Agent2D code [26], which is reflected in the high Spearman's rank correlation coefficient ^ t 2012 = 0.98h between
true rankings and goal difference against Agent2D (across
1,000 games per team). Since 2013, these correlations have
decreased substantially ^ t 2013 = 0.55 and t 2014 = 0.57hwith
teams opting to optimize behavior against the binaries published by top-performing teams postcompetition to gain a
competitive advantage with opponent-specific strategy. The
average goal difference against Agent2D decreased for the top
four RC teams between 2013 and 2014 accordingly. Importantly, this level of behavioral complexity (in addition to our
analysis of competition formats) would be impossible without
the support for massively parallel processing inherent to simulation leagues.
Summary and Discussion
Continual increases in data volume and computational power
have led to increased complexity in the experimental methodologies across most fields of research. Therefore, it is unsurprising that many fields (particularly in the life sciences [27])
have recently placed increased focus on enabling measurable,
replicable, and statistically robust results. Although robotics
researchers face many unique challenges due to the expense
and stochasticity inherent to physical robots, we propose that
physically realistic simulated environments (epitomized by
the RC simulation leagues) have an important and widespread role to play in the future of robotics.
The simulation leagues often serve as platforms for the initial development and the evaluation of software modules for
later integration into physical robots [10], [11], and many of
these modules have applications beyond the RC domain (e.g.,
localization and mapping [12]). They also enable the investigation of high-level emergent properties of complex robotic systems, as demonstrated in a recent study by Cliff et al. [24] that

0

2

4

6

8

10

12

14

16

18

L1 Distance
(a)
Format
RC 2012
RC 2013
RC 2014 (Proposed)

0

2

4

6

8 10 12 14 16 18 20 22 24
L1 Distance
(b)
Format
RC 2012
RC 2013
RC 2014 (Proposed)

0

2 4

6

8 10 12 14 16 18 20 22 24 26
L1 Distance
(c)

Figure 4. The discrepancy between tournament and true team
rankings, captured as an L 1 distance (1), for 10,000 randomly
generated tournaments structured according to the three considered
formats: (a) 2012, (b) 2013, and (c) 2014. It is evident that the
proposed format (red) yields more statistically robust rankings (i.e.,
a smaller L 1 distance) than the formats adopted in RC 2012 (green)
and 2013 (blue), considering the top eight teams from each RC.

presents novel information-theoretic methods for quantifying
dynamic interactions in a multiagent context.
In this article, we have provided an overview of the RoboCup simulation leagues (both 2-D and 3-D) and described
their properties as they pertain to replicable and robust robotics research. To demonstrate their utility directly, we leverage the ability to run massively parallelized experiments
to evaluate different competition formats (e.g., round robin)
September 2015

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

145
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015
