Signal Processing - November 2017 - 32

structure underlying the environment. From early value function
methods in DRL, which took simple states as input [61], current
methods are now able to tackle visually and conceptually complex environments [47], [48], [70], [100].

Function approximation and the DQN
We begin our survey of value-function-based DRL algorithms with the DQN [47], illustrated in Figure 5, which
achieved scores across a wide range of classic Atari 2600
video games [5] that were comparable to that of a professional video games tester. The inputs to the DQN are four
gray-scale frames of the game, concatenated over time, which
are initially processed by several convolutional layers to
extract spatiotemporal features, such as the movement of the
ball in Pong or Breakout. The final feature map from the
convolutional layers is processed by several fully connected
layers, which more implicitly encode the effects of actions.
This contrasts with more traditional controllers that use fixed
preprocessing steps, which are therefore unable to adapt their
processing of the state in response to the learning signal.
A forerunner of the DQN-neural-fitted Q (NFQ) iteration-involved training a neural network to return the Q-value
given a state-action pair [61]. NFQ was later extended to train a
network to drive a slot car using raw visual inputs from a camera
over the race track, by combining a deep autoencoder to reduce
the dimensionality of the inputs with a separate branch to predict
Q-values [38]. Although the previous network could have been
trained for both reconstruction and RL tasks simultaneously, it
was both more reliable and computationally efficient to train the
two parts of the network sequentially.
The DQN [47] is closely related to the model proposed
by Lange et al. [38] but was the first RL algorithm that was
demonstrated to work directly from raw visual inputs and on
a wide variety of environments. It was designed such that
the final fully connected layer outputs Q r (s, $) for all action
values in a discrete set of actions-in this case, the various
directions of the joystick and the fire button. This not only
enables the best action, argmax a Q r (s, a), to be chosen after
a single forward pass of the network, but also allows the network to more easily encode action-independent knowledge
in the lower, convolutional layers. With merely the goal of

maximizing its score on a video game, the DQN learns to
extract salient visual features, jointly encoding objects, their
movements, and, most importantly, their interactions. Using
techniques originally developed for explaining the behavior
of CNNs in object recognition tasks, we can also inspect what
parts of its view the agent considers important (see Figure 6).
The true underlying state of the game is contained within 128
bytes of Atari 2600 random-access memory. However, the DQN
was designed to directly learn from visual inputs (210 # 160
pixel 8-bit RGB images), which it takes as the state s. It is
impractical to represent Q r (s, a) exactly as a lookup table: when
combined with 18 possible actions, we obtain a Q-table of size
S # A = 18 # 256 3 # 210 # 160. Even if it were feasible to create such a table, it would be sparsely populated, and information
gained from one state-action pair cannot be propagated to other
state-action pairs. The strength of the DQN lies in its ability to
compactly represent both high-dimensional observations and
the Q-function using deep neural networks. Without this ability,
tackling the discrete Atari domain from raw visual inputs would
be impractical.
The DQN addressed the fundamental instability problem
of using function approximation in RL [83] by the use of
two techniques: experience replay [45] and target networks.
Experience replay memory stores transitions of the form
(s t, a t, s t +1, rt +1) in a cyclic buffer, enabling the RL agent to
sample from and train on previously observed data offline.
Not only does this massively reduce the number of interactions needed with the environment, but batches of experience
can be sampled, reducing the variance of learning updates.
Furthermore, by sampling uniformly from a large memory,
the temporal correlations that can adversely affect RL algorithms are broken. Finally, from a practical perspective,
batches of data can be efficiently processed in parallel by
modern hardware, increasing throughput. While the original DQN algorithm used uniform sampling [47], later work
showed that prioritizing samples based on TD errors is more
effective for learning [67].
The second stabilizing method, introduced by Mnih et al.
[47], is the use of a target network that initially contains the
weights of the network enacting the policy but is kept frozen
for a large period of time. Rather than having to calculate the

Action
State
Reward

FIGURE 5. The DQN [47]. The network takes the state-a stack of gray-scale frames from the video game-and processes it with convolutional and fully
connected layers, with ReLU nonlinearities in between each layer. At the final layer, the network outputs a discrete action, which corresponds to one of
the possible control inputs for the game. Given the current state and chosen action, the game returns a new score. The DQN uses the reward-the difference between the new score and the previous one-to learn from its decision. More precisely, the reward is used to update its estimate of Q, and the
error between its previous estimate and its new estimate is backpropagated through the network.
32

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017