IEEE Circuits and Systems Magazine - Q4 2020 - 71

generated at time step t. Then, the derivative of L t with
respect to W can be written as follows:
Output Layer

2L t = 2L t 2y t 2s t .(4)
2W
2y t 2s t 2W

	

Considering that s t = h ^Ux t - i + Ws t - 1 + b h h and s t - 1
is also influenced by W and s t - 2 . According to the chain
rule, we have

Hidden Layer

2s t = t 2s t 2s i .	(5)
/
2W i = 1 2s i 2W

	

In addition, we also have

Input Layer

2s t =
% 2s k .(6)
2s i k = i + 1 2s k - 1

	

Therefore, 2L t 2W can be formulated as
	

t

C. Evolution of RNNs
Despite the effectiveness in sequential data processing,
RNN still suffers from gradient vanishing or explosion.
This is because that there exists a lot of partial derivatives in model training (see Eq. 7), especially when the
number of time steps is very large. If ^2s k 2s k - 1 h2 1, then
the gradient approaches infinity when the length of input time series is too large. In contrast, if ^2s k 2s k - 1 h 1 1.
then the gradient will be close to 0. To solve this problem, adopting ReLu as activation function instead of
tanh can slightly alleviate the vanishing gradient [27]. A
better method is to change the propagation structure of
RNNs to keep the partial derivatives equal to 1. In this
case, the product of partial derivatives would no longer
converge to 0 or diverge gradually. To satisfy this demand, Hochreiter and Schmidhuber [27] present a long
short-term memory (LSTM) model with several " gate "
structures. Inherently, a " gate " is a specific function.
There are generally three kinds of gates: forget gate for,
input gate in and output gate out. Then, the inference
process of LSTM can be formulated as:
	
	
	
	
	
	

for ^ x t , s t - 1 h = v ^W f Â· conc ^ x t , s t - 1 h + b f h,(9)

in ^ x t , s t - 1 h = v ^W i Â· conc ^ x t , s t - 1 h + b i h,(10)

clt = h ^W c Â· conc ^ x t , s t - 1 h + b c h,(11)

c t = for ^ x t , s t - 1 h ) c t - 1 + in ^ x t , s t - 1 h ) clt ,(12)

out ^ x t , s t - 1 h = v ^W o Â· conc ^ x t , s t - 1 h + b o h,(13)
s t = out ^ x t , s t - 1 h ) h ^c t h,(14)

FOURTH QUARTER 2020 		

W

V

t

2L t =
/ 2L t 2y t % 2s k 2s i .	(8)
2U i = 1 2y t 2s t e k = i + 1 2s k - 1 o 2U
t

yt-1

y

2L t =
/ 2L t 2y t % 2s k 2s i .	(7)
2W i = 1 2y t 2s t e k = i + 1 2s k - 1 o 2W
t

Similarly, the derivative of L t with respect to U can be
formulated as follows:
	

Context Layer
(a)

t

s
U

W

V
st-1
U

x

yt

W

V
st
U

xt-1

yt+1

W

V
st+1

W

U
xt

xt+1

(b)
Figure 4. The illustration of RNN where U, V and W are respectively weights between input layer and hidden layer,
weights between hidden layer and output layer and weights
between hidden layer and context layer. (a) The structure of
RNN [11], (b) The inference process of RNN [43].

where conc(âÂ·â) denotes the concatenation operator and
v denotes sigmoid function. These " gates " can flexibly
control the filtration of the information flow in the network. Thereby, the gradient problems of vanilla RNNs
can be solved via making the product of the partial derivatives always equal to either 0 or 1 when computing
the gradients.
Furthermore, many variants of the " gate " structures
are suggested to improve the performance. Gers and
Schmidhuber [45] enable the gates to observe the cell
state c t - 1 through some " peephole " connections. Cho
et al. [46] simplify the transmission structure of information flow via integrating the forgot gate and input
gate into one update gate. Moreover, hidden state and
cell state are also merged in [46]. Lei et al. [47] propose
a simple recurrent unit that can accelerate network
computing via parallelizing the recurrence. Recently,
another attempt to handle the gradient problem is to
replace the weight matrices of RNN with the unitary
matrix [48]. In addition, the attention mechanism is
also introduced into the structure of RNNs to improve
the flexibility and effectiveness in different application
scenarios, such as natural language processing [49] and
image captioning [50].
IEEE CIRCUITS AND SYSTEMS MAGAZINE	

71
IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020
