IEEE Robotics & Automation Magazine - September 2015 - 143

Enabling Replicable and Robust
Analysis
Collectively, the simulation leagues provide
ideal platforms for investigating emergent
properties of complex robotic systems. Most
team games and sports (both real and virtual) are characterized by rich and dynamic
interactions that influence the contest outcome in a nontransitive manner. As described by Vilar et al. [21] "quantitative
analysis is increasingly being used in team
sports to better understand performance in
these stylized, delineated, complex social
systems." Early examples of such quantitative analysis include sabermetrics, which at(a)
(b)
tempts to search for objective knowledge
about baseball by considering the statistics
Figure 2. The Webots-simulated NAO [3] is well-suited to global optimization across
of in-game activity [22]. A recent study by the high-dimensional parameter spaces characteristic of bipedal locomotion systems,
Fewell et al. [23] involved the analysis of as experiments may be automated, parallelized, and replicated exactly. This approach
basketball games as networks, with proper- has proved very successful in identifying near-optimal parameter sets for subsequent
local optimization on actual Aldebaran NAO robots [10], [11], highlighting the utility
ties including degree centrality, clustering, of simulation leagues for the investigation and the improvement of physical robotic
entropy, and flow centrality (calculated from systems. (a) The Webots-simulated NAO [3]. (b)The actual Aldebaran NAO (http://
measurements of ball position throughout www.aldebaran.com).
the game). This idea was extended by Vilar
et al. [21], who considered the local dynamics of collective Simulation League Case Study: Analysis
team behavior to quantify how teams occupy subareas of the of Competition Formats
field as a function of ball position. Recently, Cliff et al. [24] pre- The simulation league supports fully automated, massively parsented several information-theoretic methods of quantifying allel analysis of complex robotic systems, enabling replicable and
dynamic interactions in soccer games and used the RC 2-D robust investigation of algorithms and higher level emergent besimulation league as an experimental platform.
haviors. In the following sections, we leverage these properties to
In addition to allowing high-level analysis of robotic sys- expand upon our previous analysis of RC competition formats
tems overall, the simulation leagues provide inherent support (e.g., round robin) to determine which best approximates the
for massively parallel processing. This property has been lev- true performance rankings of competing teams [25].
eraged for the development and the analysis of algorithms
The selection of an appropriate competition format is critiwith widespread applications in robotics, e.g., optimizing bi- cal to both the success and the credibility of any competition.
pedal locomotion [10], [11], self-localization from noisy per- Unfortunately, this choice is not straightforward. The ideal forception data [12], and planning complex multiagent mat must minimize the randomness relative to the true perforstrategies without direct agent-to-agent communication [16], mance ranking of teams while keeping the number of games
[17]. Although simulation league agents have only noisy per- to a minimum, to both satisfy time constraints and retain the
ception of their environment, the soccer server itself has per- interest of participants and spectators. Furthermore, maintainfect information regarding the global state, which enables ing competition interest introduces a number of constraints to
replicable quantification of experimental performance (e.g., competition formats, e.g., multiple games between the same
walk speed/stability and localization accuracy).
two opponents (the obvious method of achieving a statistically
significant ranking) should be avoided, making the resolution
Wider Implications in Robotics Research
of nontransitive performance difficult.
Robots are generally expensive to purchase, maintain, and
transport, creating an intractably high entry barrier for insti- Robocup Competition Formats
tutes with limited access to research funding. By removing The following competition formats were adopted to deterthe requirement for physical robots, the RC simulation mine the final rank of the top eight RC 2-D simulation league
leagues allow such institutes to actively contribute to many teams from 2012 to 2014.
fields of robotics research. To validate this assertion, Figure 3 ● 2012: The top four teams played six games each [three
presents the public expenditure on education as a percentage
quarterfinals (round robin), two semifinals, and classificaof gross domestic product (GDP) public expenditure on edution matches for first versus second and third versus
cation (PEoE) at purchasing power parity per capita (GDP/
fourth] and the bottom four teams played four games each.
cap)[20] for the home country of each participating RC 2013 ● 2013: A double-elimination system was adopted, where a
team, averaged over each of the six largest RC leagues.
team is ineligible for first place upon losing two games. A
September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

143

http://http:// http://www.aldebaran.com

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015