IEEE Robotics & Automation Magazine - December 2015 - 151

CR

CR

CR

2PO

EA

EA

PA
(a)

STO

LO

LO2

SO

KT

PA
(b)

MR

LO1

PA

1PO

KT

LAB

KT

TL

RA

SO2

LMR

(c)

CR

CR

EA

EA

STO

STO

LGA

HA

PA
(d)

PO

PO

SE

TL

WH

TR

TL

TR

VC

VC

(e)
Figure 1. The exemplar images for every class and edition combination. Classes are named using the keys defined in Table 3.
(a) RobotVision at ImageCLEF 2009. (b) RobotVision at ImageCLEF 2010 ICPR. (c) RobotVision at ImageCLEF 2010. (d) RobotVision
at ImageCLEF 2012. (e) RobotVision at ImageCLEF 2013.

submitted results and wrote a working note with their proposal details (see Table 4).
With respect to the obtained results, all the editions used a
score that computed the performance of the participant submission. This score was always based on positive values for
test images correctly classified and negative values for misclassified ones. We also allowed the possibility of not classifying test images, resulting in nonaltering the score. The
maximum reachable scores for the mandatory task of each
edition were 1,690, 5,102, 2,741, 2,445, and 7,030, respectively.
Regarding the optional task, the maximum scores were 1,690,
5,102, 2,741, and 4,079, respectively for the first to fourth editions. The fifth edition of the task had no optional task. All
the results are shown in Tables 5 and 6 for the mandatory and
optional tasks of each edition, respectively.
In order to graphically present the results obtained for all
the editions of the challenge, we have created two boxplots
shown in Figures 3 and 4. We can observe how the first two
editions of the competitions were quite balanced, both for the

mandatory and optional subtasks. The experience achieved in
the first year was successfully used to obtain higher results in
the second one. The third edition notoriously increased the
difficulty of the competition, with low results in the mandatory subtask. This happened because the test sequence was acquired on a different floor
of the building, which intro duced imp or t ant
Pose annotations were
changes in the objects'
distribution in the scene.
removed and monocular
The changes introduced
in the fourth edition were
images were replaced by
translated into larger variations between particistereo ones.
pant results. Finally, the
fifth edition of the competition was properly managed by participants, with five out of
six groups obtaining results higher than the 60% of the maximum score.
DECEMBER 2015

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

151



Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2015

IEEE Robotics & Automation Magazine - December 2015 - Cover1
IEEE Robotics & Automation Magazine - December 2015 - Cover2
IEEE Robotics & Automation Magazine - December 2015 - 1
IEEE Robotics & Automation Magazine - December 2015 - 2
IEEE Robotics & Automation Magazine - December 2015 - 3
IEEE Robotics & Automation Magazine - December 2015 - 4
IEEE Robotics & Automation Magazine - December 2015 - 5
IEEE Robotics & Automation Magazine - December 2015 - 6
IEEE Robotics & Automation Magazine - December 2015 - 7
IEEE Robotics & Automation Magazine - December 2015 - 8
IEEE Robotics & Automation Magazine - December 2015 - 9
IEEE Robotics & Automation Magazine - December 2015 - 10
IEEE Robotics & Automation Magazine - December 2015 - 11
IEEE Robotics & Automation Magazine - December 2015 - 12
IEEE Robotics & Automation Magazine - December 2015 - 13
IEEE Robotics & Automation Magazine - December 2015 - 14
IEEE Robotics & Automation Magazine - December 2015 - 15
IEEE Robotics & Automation Magazine - December 2015 - 16
IEEE Robotics & Automation Magazine - December 2015 - 17
IEEE Robotics & Automation Magazine - December 2015 - 18
IEEE Robotics & Automation Magazine - December 2015 - 19
IEEE Robotics & Automation Magazine - December 2015 - 20
IEEE Robotics & Automation Magazine - December 2015 - 21
IEEE Robotics & Automation Magazine - December 2015 - 22
IEEE Robotics & Automation Magazine - December 2015 - 23
IEEE Robotics & Automation Magazine - December 2015 - 24
IEEE Robotics & Automation Magazine - December 2015 - 25
IEEE Robotics & Automation Magazine - December 2015 - 26
IEEE Robotics & Automation Magazine - December 2015 - 27
IEEE Robotics & Automation Magazine - December 2015 - 28
IEEE Robotics & Automation Magazine - December 2015 - 29
IEEE Robotics & Automation Magazine - December 2015 - 30
IEEE Robotics & Automation Magazine - December 2015 - 31
IEEE Robotics & Automation Magazine - December 2015 - 32
IEEE Robotics & Automation Magazine - December 2015 - 33
IEEE Robotics & Automation Magazine - December 2015 - 34
IEEE Robotics & Automation Magazine - December 2015 - 35
IEEE Robotics & Automation Magazine - December 2015 - 36
IEEE Robotics & Automation Magazine - December 2015 - 37
IEEE Robotics & Automation Magazine - December 2015 - 38
IEEE Robotics & Automation Magazine - December 2015 - 39
IEEE Robotics & Automation Magazine - December 2015 - 40
IEEE Robotics & Automation Magazine - December 2015 - 41
IEEE Robotics & Automation Magazine - December 2015 - 42
IEEE Robotics & Automation Magazine - December 2015 - 43
IEEE Robotics & Automation Magazine - December 2015 - 44
IEEE Robotics & Automation Magazine - December 2015 - 45
IEEE Robotics & Automation Magazine - December 2015 - 46
IEEE Robotics & Automation Magazine - December 2015 - 47
IEEE Robotics & Automation Magazine - December 2015 - 48
IEEE Robotics & Automation Magazine - December 2015 - 49
IEEE Robotics & Automation Magazine - December 2015 - 50
IEEE Robotics & Automation Magazine - December 2015 - 51
IEEE Robotics & Automation Magazine - December 2015 - 52
IEEE Robotics & Automation Magazine - December 2015 - 53
IEEE Robotics & Automation Magazine - December 2015 - 54
IEEE Robotics & Automation Magazine - December 2015 - 55
IEEE Robotics & Automation Magazine - December 2015 - 56
IEEE Robotics & Automation Magazine - December 2015 - 57
IEEE Robotics & Automation Magazine - December 2015 - 58
IEEE Robotics & Automation Magazine - December 2015 - 59
IEEE Robotics & Automation Magazine - December 2015 - 60
IEEE Robotics & Automation Magazine - December 2015 - 61
IEEE Robotics & Automation Magazine - December 2015 - 62
IEEE Robotics & Automation Magazine - December 2015 - 63
IEEE Robotics & Automation Magazine - December 2015 - 64
IEEE Robotics & Automation Magazine - December 2015 - 65
IEEE Robotics & Automation Magazine - December 2015 - 66
IEEE Robotics & Automation Magazine - December 2015 - 67
IEEE Robotics & Automation Magazine - December 2015 - 68
IEEE Robotics & Automation Magazine - December 2015 - 69
IEEE Robotics & Automation Magazine - December 2015 - 70
IEEE Robotics & Automation Magazine - December 2015 - 71
IEEE Robotics & Automation Magazine - December 2015 - 72
IEEE Robotics & Automation Magazine - December 2015 - 73
IEEE Robotics & Automation Magazine - December 2015 - 74
IEEE Robotics & Automation Magazine - December 2015 - 75
IEEE Robotics & Automation Magazine - December 2015 - 76
IEEE Robotics & Automation Magazine - December 2015 - 77
IEEE Robotics & Automation Magazine - December 2015 - 78
IEEE Robotics & Automation Magazine - December 2015 - 79
IEEE Robotics & Automation Magazine - December 2015 - 80
IEEE Robotics & Automation Magazine - December 2015 - 81
IEEE Robotics & Automation Magazine - December 2015 - 82
IEEE Robotics & Automation Magazine - December 2015 - 83
IEEE Robotics & Automation Magazine - December 2015 - 84
IEEE Robotics & Automation Magazine - December 2015 - 85
IEEE Robotics & Automation Magazine - December 2015 - 86
IEEE Robotics & Automation Magazine - December 2015 - 87
IEEE Robotics & Automation Magazine - December 2015 - 88
IEEE Robotics & Automation Magazine - December 2015 - 89
IEEE Robotics & Automation Magazine - December 2015 - 90
IEEE Robotics & Automation Magazine - December 2015 - 91
IEEE Robotics & Automation Magazine - December 2015 - 92
IEEE Robotics & Automation Magazine - December 2015 - 93
IEEE Robotics & Automation Magazine - December 2015 - 94
IEEE Robotics & Automation Magazine - December 2015 - 95
IEEE Robotics & Automation Magazine - December 2015 - 96
IEEE Robotics & Automation Magazine - December 2015 - 97
IEEE Robotics & Automation Magazine - December 2015 - 98
IEEE Robotics & Automation Magazine - December 2015 - 99
IEEE Robotics & Automation Magazine - December 2015 - 100
IEEE Robotics & Automation Magazine - December 2015 - 101
IEEE Robotics & Automation Magazine - December 2015 - 102
IEEE Robotics & Automation Magazine - December 2015 - 103
IEEE Robotics & Automation Magazine - December 2015 - 104
IEEE Robotics & Automation Magazine - December 2015 - 105
IEEE Robotics & Automation Magazine - December 2015 - 106
IEEE Robotics & Automation Magazine - December 2015 - 107
IEEE Robotics & Automation Magazine - December 2015 - 108
IEEE Robotics & Automation Magazine - December 2015 - 109
IEEE Robotics & Automation Magazine - December 2015 - 110
IEEE Robotics & Automation Magazine - December 2015 - 111
IEEE Robotics & Automation Magazine - December 2015 - 112
IEEE Robotics & Automation Magazine - December 2015 - 113
IEEE Robotics & Automation Magazine - December 2015 - 114
IEEE Robotics & Automation Magazine - December 2015 - 115
IEEE Robotics & Automation Magazine - December 2015 - 116
IEEE Robotics & Automation Magazine - December 2015 - 117
IEEE Robotics & Automation Magazine - December 2015 - 118
IEEE Robotics & Automation Magazine - December 2015 - 119
IEEE Robotics & Automation Magazine - December 2015 - 120
IEEE Robotics & Automation Magazine - December 2015 - 121
IEEE Robotics & Automation Magazine - December 2015 - 122
IEEE Robotics & Automation Magazine - December 2015 - 123
IEEE Robotics & Automation Magazine - December 2015 - 124
IEEE Robotics & Automation Magazine - December 2015 - 125
IEEE Robotics & Automation Magazine - December 2015 - 126
IEEE Robotics & Automation Magazine - December 2015 - 127
IEEE Robotics & Automation Magazine - December 2015 - 128
IEEE Robotics & Automation Magazine - December 2015 - 129
IEEE Robotics & Automation Magazine - December 2015 - 130
IEEE Robotics & Automation Magazine - December 2015 - 131
IEEE Robotics & Automation Magazine - December 2015 - 132
IEEE Robotics & Automation Magazine - December 2015 - 133
IEEE Robotics & Automation Magazine - December 2015 - 134
IEEE Robotics & Automation Magazine - December 2015 - 135
IEEE Robotics & Automation Magazine - December 2015 - 136
IEEE Robotics & Automation Magazine - December 2015 - 137
IEEE Robotics & Automation Magazine - December 2015 - 138
IEEE Robotics & Automation Magazine - December 2015 - 139
IEEE Robotics & Automation Magazine - December 2015 - 140
IEEE Robotics & Automation Magazine - December 2015 - 141
IEEE Robotics & Automation Magazine - December 2015 - 142
IEEE Robotics & Automation Magazine - December 2015 - 143
IEEE Robotics & Automation Magazine - December 2015 - 144
IEEE Robotics & Automation Magazine - December 2015 - 145
IEEE Robotics & Automation Magazine - December 2015 - 146
IEEE Robotics & Automation Magazine - December 2015 - 147
IEEE Robotics & Automation Magazine - December 2015 - 148
IEEE Robotics & Automation Magazine - December 2015 - 149
IEEE Robotics & Automation Magazine - December 2015 - 150
IEEE Robotics & Automation Magazine - December 2015 - 151
IEEE Robotics & Automation Magazine - December 2015 - 152
IEEE Robotics & Automation Magazine - December 2015 - 153
IEEE Robotics & Automation Magazine - December 2015 - 154
IEEE Robotics & Automation Magazine - December 2015 - 155
IEEE Robotics & Automation Magazine - December 2015 - 156
IEEE Robotics & Automation Magazine - December 2015 - 157
IEEE Robotics & Automation Magazine - December 2015 - 158
IEEE Robotics & Automation Magazine - December 2015 - 159
IEEE Robotics & Automation Magazine - December 2015 - 160
IEEE Robotics & Automation Magazine - December 2015 - 161
IEEE Robotics & Automation Magazine - December 2015 - 162
IEEE Robotics & Automation Magazine - December 2015 - 163
IEEE Robotics & Automation Magazine - December 2015 - 164
IEEE Robotics & Automation Magazine - December 2015 - 165
IEEE Robotics & Automation Magazine - December 2015 - 166
IEEE Robotics & Automation Magazine - December 2015 - 167
IEEE Robotics & Automation Magazine - December 2015 - 168
IEEE Robotics & Automation Magazine - December 2015 - Cover3
IEEE Robotics & Automation Magazine - December 2015 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2010
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2010
https://www.nxtbookmedia.com