IEEE Computational Intelligence Magazine - May 2023 - 97

vehicle starts to flash, his prediction of
its driving route would change. Understood
from the perspective of artificial
intelligence and machine learning, the
model (equivalent to a human brain)
needs to constantly optimize its predicted
results according to the latest
information when making predictions
from video.
Guided by the above analysis, this
FIGURE 1. Architecture of DPR-LC-Net.
BeyondMSE can be used to construct
complex predictive network models. Similarly,
Lee et al. [39] designed a model that
combines the commonly used latent variational
variable model with the adversarial
training model to solve the problem of
blurry prediction results. This model can
use the latent variables for diversified predictions
and can generate realistic and natural
images. Numerous experiments have
demonstrated the high performance of
DL models in video prediction tasks.
III. Proposed Model
DPR-LC-Net has three layers, each of
which consists of several modules and
using multiple network layers. For the
sake of brevity, conventional techniques,
such as convolutional layers and dropouts,
are not explained here. The structure of
ConvLSTM is described using the following
equations:
iiiiit ¼ s WxiXt þWhiHt1 þ biðÞ; (1Þ
ffft Ct1 þ iiiiit Gt;
Ht ¼ ot tanh CtðÞ:
(5Þ
(6Þ
ffffft ¼ s WxfXt þWhfHt1 þ bf ;(2Þ
ot ¼ s WxoXt þWhoHt1 þ boðÞ; (3Þ
Gt ¼ tanh WxcXt þWhcHt1 þ bcðÞ; (4Þ
Ct ¼fff
Here, s and "  " denote the sigmoid
function and convolution operator,
respectively. W and b denote the twodimensional
convolution kernel and bias,
respectively. In time step t,the input Xt
and hidden state Ht1 are used to compute
iiiiiiit,ffffft, ot,andGt. Through the Hadamard
product "  " ,the cell state Ct and
hidden state Ht are obtained and then
passed to the next time step. In this paper,
all variables in boldface represent vectors
or matrices.
Fig. 1 illustrates the overall structure of
the proposed DPR-LC-Net model,
which includes ConvLSTM and other
network layers. In the remainder of this
section, the overall computational process
of DPR-LC-Net is first described, then
the pixel restoration module is explained,
and finally, the other special modules are
covered.
A. Frame-by-Frame Prediction and
Overall Computation Process
Imagine a situation where a man is driving
behind a vehicle with all lights offin
the daytime; he would naturally predict
that the vehicle will continue to drive
in the same direction. However, when
he notices the left turn signal of the
study designed the DPR-LC-Net to have
a frame-by-frame prediction mechanism,
whichisinlinewith the prediction characteristics
of the human brain. As shown
in the first layer of Fig. 1,after thepixel
restoration module generates the predicted
frames, both the predicted and real
frames are input to the error module to
calculate the error, which is then passed to
the second layer and the next time step,
thus helping DPR-LC-Net to adjust and
optimize itselfin time according to the latest
real frames.
This section explains the overall information
propagation and update process of
DPR-LC-Net. The subscript t and superscript
l represent the time step and layer
level, respectively.
First, as shown in Fig. 1,fromtime
step t to time step tþ1, layers 2 and 3 both
output the results ofthe error modules El
t
(indicated by the black dotted line) and
the state values of the ConvLSTM module
(i.e., Hl
t and Cl
t). The green double
line depicts the hidden state values and cell
state values. Layer 1 also outputs the
ground-truth frame Xt at time step t for
the pixel restoration module in the next
time step.
Subsequently, after obtaining the
above outputs at time step tþ1, the
ConvLSTM module oflayer 3 first performs
calculations and then transfers the
results to layer 2 and the Cov module of
layer 3. After the ConvLSTM module
oflayer 2 obtains the results, it performs
its calculations. These results are also
passed downward and toward the right.
The results of the ConvLSTM module
oflayer 1 only need to be passed toward
the right.
After the computation of the
ConvLSTMmodule has been completed
for each layer, layer 1 starts the calculation
from left to right. The results pass through
MAY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 97
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
