Signal Processing - November 2017 - 36

ter sample complexity, generalization, and interpretability [16].
neural network research, with the most notable success being
ALVINN, one of the earliest autonomous cars [59]. However,
In time, we also hope that our theoretical understanding of the
behavioral cloning cannot adapt to new situations, and small
properties of neural networks (particularly within DRL) will
deviations from the demonstration during the execution of
improve, as it currently lags far behind practice.
the learned policy can compound and lead to scenarios where
To conclude, it is worth revisiting the overarching goal of all
the policy is unable to recover. A more generalizable solution
of this research: the creation of general-purpose AI systems that
is to use provided trajectories to guide the learning of suitcan interact with and learn from the world around them. Interacable state-action pairs but fine-tune the agent using RL [23].
tion with the environment is simultaneously the advantage and
The goal of IRL is to estimate an unknown reward function
disadvantage of RL. While there are many challenges in seeking
from observed trajectories that characterize a desired solution
to understand our complex and ever-changing world, RL allows
[52]; IRL can be used in combination with
us to choose how we explore it. In effect, RL
RL to improve upon demonstrated behavior.
endows agents with the ability to perform
In effect, RL endows
Using the power of deep neural networks, it
experiments to better understand their suragents with the ability to
is now possible to learn complex, nonlinear
roundings, enabling them to learn even highperform experiments to
reward functions for IRL [98]. Ho and Ermon
level causal relationships. The availability
[24] showed that policies are uniquely charof high-quality visual renderers and physics
better understand their
acterized by their occupancies (visited state
engines now enables us to take steps in this
surroundings, enabling
and action distributions) allowing IRL to be
direction, with works that try to learn intuithem to learn even highreduced to the problem of measure matching.
tive models of physics in visual environments
level causal relationships.
With this insight, they were able to use gen[13]. Challenges remain before this will be
erative adversarial training [18] to facilitate
possible in the real world, but steady progress
reward-function learning in a more flexible manner, resulting in
is being made in agents that learn the fundamental principles of
the generative adversarial imitation learning algorithm.
the world through observation and action. Perhaps, then, we
are not too far away from AI systems that learn and act in more
human-like ways in increasingly complex environments.
Multiagent RL
Usually, RL considers a single learning agent in a stationary
environment. In contrast, multiagent RL (MARL) considers
Acknowledgments
multiple agents learning through RL and often the nonstationKai Arulkumaran would like to acknowledge Ph.D. funding
arity introduced by other agents changing their behaviors as
from the Department of Bioengineering at Imperial College
they learn [8]. In DRL, the focus has been on enabling (differLondon. This research has been partially funded by a Google
Faculty Research Award to Marc Deisenroth.
entiable) communication between agents, which allows them
to cooperate. Several approaches have been proposed for this
purpose, including passing messages to agents sequentially
Authors
[15], using a bidirectional channel (providing ordering with
Kai Arulkumaran (ka709@imperial.ac.uk) received a B.A.
less signal loss) [58], and an all-to-all channel [77]. The addidegree in computer science at the University of Cambridge in
2012 and an M.Sc. degree in biomedical engineering from
tion of communication channels is a natural strategy to apply
Imperial College London in 2014, where he is currently a Ph.D.
to MARL in complex scenarios and does not preclude the
degree candidate in the Department of Bioengineering. He was a
usual practice of modeling cooperative or competing agents as
research intern at Twitter's Magic Pony and Microsoft Research
applied elsewhere in the MARL literature [8].
in 2017. His research focus is deep reinforcement learning and
transfer learning for visuomotor control.
Conclusion: Beyond pattern recognition
Marc Peter Deisenroth (m.deisenroth@imperial.ac.uk)
Despite the successes of DRL, many problems need to be
received an M.Eng. degree in computer science at the University
addressed before these techniques can be applied to a wide
of Karlsruhe in 2006 and a Ph.D. degree in machine learning at
range of complex real-world problems [37 ]. Recent work
the Karlsruhe Institute of Technology in 2009. He is a lecturer of
with (nondeep) generative causal models demonstrated supestatistical machine learning in the Department of Computing at
rior generalization over standard DRL algorithms [48], [63]
Imperial College London and with PROWLER.io. He was
in some benchmarks [5], achieved by reasoning about causes
awarded an Imperial College Research Fellowship in 2014 and
and effects in the environment [29]. For example, the schema
received Best Paper Awards at the International Conference on
networks of Kanksy et al. [29] trained on the game BreakRobotics and Automation 2014 and the International Conference
out immediately adapted to a variant where a small wall was
on Control, Automation, and Systems 2016. He is a recipient of
placed in front of the target blocks, while progressive (A3C)
a Google Faculty Research Award and a Microsoft Ph.D.
networks [63] failed to match the performance of the schema
Scholarship. His research is centered around data-efficient
networks even after training on the new domain. Although
machine learning for autonomous decision making.
DRL has already been combined with AI techniques, such
Miles Brundage (miles.brundage@philosophy.ox.ac.uk)
as search [73] and planning [80], a deeper integration with
received a B.A. degree in political science at George
other traditional AI approaches promises benefits such as bet36

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

http://www.M.Sc http://www.PROWLER.io

Table of Contents for the Digital Edition of Signal Processing - November 2017