IEEE Circuits and Systems Magazine - Q4 2019 - 26

perform. The softmax operator is a commonly used output layer for classification tasks, of which the resultant
vectors present a classification probability distribution
summing to 1. For training a CNN, the goal is to minimize the objective function defined for the task. With an
appropriate optimizer, the best fitting set of parameters
(the weights and bias terms) can be solved.

V
s

U
i
Figure 7. A simple computation unit of RNNs.

ot -1

V
W

ot +1

St -1

U
it -1

St +1

it +1

Figure 8. Unfolded structures of RNNs.

s′t

A′2

A′1

A′0
s0

Figure 9. A simple bi-directional RNN.
26

IEEE CIRCUITS AND SYSTEMS MAGAZINE

3) Recurrent Neural Networks
Recurrent neural networks (RNNs) are developed to
process pattern recognition tasks of sequential data like
text, genomes, spoken words or numerical times series
data from real-world commerce [55]. Compared to the
MLP and CNNs, RNNs explore the temporal information of inputs by taking time and sequence into account.
To be specific, in the case of feedforward networks as
shown in Fig. 4, samples fed to the network are transformed straight into an output via supervised learning.
While in RNNs, the output at time step t - 1 affects the
future decision one moment later at time step t by the
feedback loops. This unique component makes RNNs
have "memory," which is indispensable for language
processing and sequence signal processing.
Fig. 7 indicates the basic schematic diagram in RNNs
which is an integration of three kinds of layers: the input
layer, the hidden layer and the output layer. The i denotes the input vector and o denotes the output vector.
The hidden neurons are denoted as the vector s while
U, V and W are the weights matrix. Note that W contains the information of the previous time steps for this
neuron. After unfolding, the diagram of this basic RNN
unit is represented as in Fig. 8. From Fig. 8, it is explicitly
shown that the value of hidden layers depend on both
the inputs and the hidden neurons of the previous time
steps. The output and hidden layers can be calculated
as below:
o t = g ^Vs t h
s t = f ^Ui t + Ws t - 1 h

(3)

For language processing, it is necessary to model
not only previous words or phrases in a sentence, but
also subsequent linguistic units at time step t + 1. Bidirectional RNNs are proposed to predict or label each
element of a sequence with its past and future contexts.
Outputs of two RNNs can be concatenated to process the sequence
from opposite directions. Fig. 9 is
ot
a graph demonstrating an example of such bi-directional RNNs.
In practical applications, it is
found that the performance of
A′t
s′0
the RNNs degrades when data
sequences are long. Exploding
st
At
gradients and vanishing gradient
problems result in an uncontrolled
propagation of the gradients in
the training procedure. Presetting
it
a threshold is an effective way to
avoid the exploding gradients issue, but the vanishing gradient is
tricky to restrain. Three methods
FOURTH QUARTER 2019

IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019