IEEE Robotics & Automation Magazine - June 2011 - 84

automatic recognition, categorization, and labeling of
human actions and behaviors from sensor data. Because of
the stringent requirements dictated by user acceptance,
these methods are typically robust to human variability and
hardware-dependent factors, including variability in sensor
type and placement. This makes them a potentially useful
tool for the automatic recognition and labeling of robot
behaviors and may lead to new opportunities for research
in robotics. We detail three domains in which the methods
of activity recognition can play a role in robotics.
Annotation of Large-Scale Activity Data Sets
Because of the ease of systematic data collection from
robots and their potential usefulness for data mining, a
future World Wide Web (WWW) for robots is likely to
include data sets for a large number of behavioral strategies
for different robotic platforms in different situations. Such
data sets may, for example, include sensor readings for the
walking behavior of a humanoid robot on different terrains
or trajectory information for the grasp behavior of a pickand-place robot for various target objects. Although individual robots are typically aware of their current behavior
and may partially label such data, difficulties in creating
comprehensive naming conventions and precise definitions for behaviors make such labels too vague to support
comparative performance evaluation. Current methods for
human activity recognition may allow to automatically
supplement such labels by providing systematic and
comparable categories for behaviors. In addition, they may
be used to automatically identify underlying motion primitives, further increasing their precision and potential for
data mining.
Human-Robot Interaction
Activity recognition supports natural HRI [8]. Here, we
take the viewpoint that human activities are inferred from
sensors worn by the user and broadcasted to the surrounding robots. The typical applications are in the domain of

assistive robotics. Activity recognition for assistive uses is
the typical aim of wearable computing.
Robot Self-Learning
Imagine that a human being teaches a robot by demonstration [9] (see Figure 1). Using activity recognition, the robot
can supplement its known set of behaviors with an internal
model to recognize different demonstrated activities [10],
[11] and even build a repertoire of subgoals and obtain a
hierarchical decomposition of its actions [12]. The inferred
model can provide feedback for a self-learning process
[13], e.g., using self-perception [14], reinforcement learning [15], or evolutionary techniques [16]. Such self-learning would allow the robot to develop its own realization of
the motor commands in the light of the goal to reach. In
addition, it may lead to increases in behavioral robustness.
For instance, a damaged robot such as NASA's famous
"Spirit" rover may not have been preprogrammed to cope
with all modes of failures. However, activity recognition
could provide it with insights into its actual behavioral
performance despite the damage. Using continuous selflearning, this may allow it to develop a novel, effective
motor-control program overcoming the problem.
Robot self-learning can also be applied across platforms. Rather than demonstrating all activities to all
robots, activity recognition may allow robots to learn from
each other. Given compatible perception systems, robots
can directly share and reuse activity-recognition programs
across platforms. Differences such as parameterizations
of motor control programs can then be self-learnt. The
direct sharing of an activity-recognition program is only
possible for identical perception systems. However,
efforts in activity recognition, which we review here, aim
at methods that are robust across different perception
systems [17]. In particular, a reduced set of sensors may
nevertheless allow to recognize a common subset of activities [18]. Since many of today's robots share common
sensors such as cameras or laser scanners, this potentially

Transfer of Activity-Recognition Module
Between Platforms

Activity
Recognition

Labels

Activity
Recognition

Reward
Signal

Sensor Inputs
Commands

Robot
Commands Controller

(a)

(b)

Activity
Recognition

Reward
Signal

Sensor Inputs
Robot
Commands Controller
(c)

Figure 1. (a) An activity recognition system trained by user demonstration (b) guides the adaptation of the robot motor controller.
(c) The recognition system can be transferred to another robot with a compatible perception subsystem and different actuators (c).
This allows it to self-learn the same motor skills as the previous robot.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2011

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2011