Signal Processing - November 2017 - 27

field of DRL, there have been two outstanding success stories.
The first, kickstarting the revolution in DRL, was the development of an algorithm that could learn to play a range of Atari
2600 video games at a superhuman level, directly from image
pixels [47]. Providing solutions for the instability of function
approximation techniques in RL, this work was the first to convincingly demonstrate that RL agents could be trained on raw,
high-dimensional observations, solely based on a reward signal.
The second standout success was the development of a hybrid
DRL system, AlphaGo, that defeated a human world champion in
Go [73], paralleling the historic achievement of IBM's Deep Blue
in chess two decades earlier [9]. Unlike the handcrafted rules that
have dominated chess-playing systems, AlphaGo comprised neural networks that were trained using supervised learning and RL,
in combination with a traditional heuristic search algorithm.
DRL algorithms have already been applied to a wide range
of problems, such as robotics, where control policies for robots
can now be learned directly from camera inputs in the real world
[41], [42], succeeding controllers that used to be hand-engineered
or learned from low-dimensional features of the robot's state. In
Figure 1, we showcase just some of the domains that DRL has

The advent of deep learning has had a significant impact
on many areas in machine learning, dramatically improving
the state of the art in tasks such as object detection, speech
recognition, and language translation [39]. The most important property of deep learning is that deep neural networks can
automatically find compact low-dimensional representations
(features) of high-dimensional data (e.g., images, text, and
audio). Through crafting inductive biases into neural network
architectures, particularly that of hierarchical representations,
machine-learning practitioners have made effective progress
in addressing the curse of dimensionality [7]. Deep learning
has similarly accelerated progress in RL, with the use of deeplearning algorithms within RL defining the field of DRL. The
aim of this survey is to cover both seminal and recent developments in DRL, conveying the innovative ways in which neural networks can be used to bring us closer toward developing
autonomous agents. For a more comprehensive survey of recent
efforts in DRL, we refer readers to the overview by Li [43].
Deep learning enables RL to scale to decision-making problems that were previously intractable, i.e., settings with highdimensional state and action spaces. Among recent work in the

(a)

(b)

Target

(c)

(d)
A

(e)
Standing

Giraffe

(f)

FIGURE 1. A range of visual RL domains. (a) Three classic Atari 2600 video games, Enduro, Freeway, and Seaquest, from the Arcade Learning Environment (ALE)

[5]. Due to the range of supported games that vary in genre, visuals, and difficulty, the ALE has become a standard test bed for DRL algorithms [20], [47], [48],
[55], [70], [75], [92]. The ALE is one of several benchmarks that are now being used to standardize evaluation in RL. (b) The TORCS car racing simulator, which
has been used to test DRL algorithms that can output continuous actions [33], [44], [48] (as the games from the ALE only support discrete actions). (c) Utilizing
the potentially unlimited amount of training data that can be amassed in robotic simulators, several methods aim to transfer knowledge from the simulator to the
real world [11], [64], [84]. (d) Two of the four robotic tasks designed by Levine et al. [41]: screwing on a bottle cap and placing a shaped block in the correct hole.
Levine et al. [41] were able to train visuomotor policies in an end-to-end fashion, showing that visual servoing could be learned directly from raw camera inputs by
using deep neural networks. (e) A real room, in which a wheeled robot trained to navigate the building is given a visual cue as input and must find the corresponding location [100]. (f) A natural image being captioned by a neural network that uses RL to choose where to look [99]. (b)-(f) reproduced from [41], [44], [84],
[99], and [100], respectively.

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|

27
Table of Contents for the Digital Edition of Signal Processing - November 2017
