IEEE Robotics & Automation Magazine - March 2014 - 24

ones, targeting the definition of practical protocols, like the one at RSS 2013 in
Berlin, or deep theoretical issues, like
the one at IROS 2013 in Tokyo.

Figure 3. The nodal apparatus for a manipulator dexterity evaluation. This model has
six directed inspection targets and can be used for retrieval and insertion evaluation as
well. (Image courtesy of NIST.)
●

●

Workshop on Proposals for Experimental Protocols for Robotics
Research during the RSS 2013 conference (Berlin, Germany, 27 June 2013)
Workshop on Metrics of Sensory
Motor Integration in Robots and
Animals at IROS 2012 (Vilamoura,
Algarve, Portugal, 12 October 2012)
Replicable Robotics Research, Benchmarking, and Result Exploitation:
Where We Are During the European
Robotics Forum (Lyon, France, 21
March 2013)
From Theory to Practice of Performance Comparison and Result Rep-

IEEE ROBOTICS & AUTOMATION MAGAZINE

lications in Robotics Research-
Workshop at RSS 2012 (Sydney,
Australia, 12 July 2012).
In parallel, the Performance Metrics
for Intelligent Systems Workshops focus
on performance measure challenges
coming from the application of robotics
and automation technologies to practical problems in the commercial, industrial, homeland security, and military
domains. More information can be
found at http://www.nist.gov/el/isd/permis2012.cfm. The scope of the workshops is gradually shifting from general
purpose meetings to more focused

MARCH 2014

Publications
Several publications, besides the proceedings of the workshops, have been
produced on the topics related to the
TC, and others are coming. The following are some of the most relevant:
● R. Madhavan, E. Tunstel, and E. Messina, Eds., "Quantifying the performance of intelligent systems," Int. J.
Intell. Control Syst., Special Issue, vol.
16, no. 2, pp. 37-159, June 2011.
● "Quantifying the performance of intelligent systems," Int. J. Intell. Control Syst.,
Special Issue, vol. 16, no. 2, pp. 37-39,
June 2011.
● Performance Evaluation and Benchmarking of Intelligent Systems, R. Madhavan, E. Tunstel, and E. Messina (Eds.).
Springer, ISBN: 978- 1-4419-0491-1, Sept.
2009.
● R. Madhavan, C. Scrapper, and A.
Kleiner (Eds.), Autonomous Robots, Special Issue, vol. 27, no. 4, 2009.
● R. Madhavan, A. P. del Pobil, and
E. Messina, "Performance evaluation
and benchmarking of robotic and automation systems [TC Spotlight]," IEEE
Robot. Autom. Mag., vol. 17, no. 1, pp.
120-122, 2010.
A book on the measurement of sensory-motor coordination in robots and
animals, to be published in the Cognitive Systems Monographs Springer
series, is currently in preparation.
Competitions
The TC activities include robotics competitions as competitions are a valid
complement to benchmarks for the comparison of system level intelligent behaviors, which can be difficult to quantify in
specific quantitative metrics.
Two of the cosponsored competitions of TC-PEBRAS are the Virtual
Manufacturing Automation Competition (VMAC) (http://www.vma-competition.com), which was initiated as a
demonstration event at ICRA 2008 and
has continued as a robot challenge
event, and the Mobile Microrobotics
Challenge (http://www.nist.gov/eeel/

http://www.vma-com http://www.petition.com http://www.nist.gov/el/isd/per http://www.nist.gov/eeel/

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2014