IEEE Computational Intelligence Magazine - May 2023 - 103

addition, DPR-LC-Net uses the optical
flow method to extract pixel values,
thereby avoiding the problems of blurring
and lack ofrealism that occur when
traditional DL models are used to generate
predicted frames directly. Therefore,
it successfully generates clear and
realistic predicted video frames.
In these three experimental stages, the
first stage initially verified the predictive
performance ofDPR-LC-Net. The second
stage confirmed the accuracy and stability
ofDPR-LC-Net with datasets with
a variety ofcharacteristics. The third stage
compared the performance ofDPR-LCNet
with that of several current state-ofthe-art
models, further demonstrating its
superiority in performance.
DPR-LC-Net still has some shortcomings.
For example, it uses MSE as
an evaluation metric for prediction performance.
MSE has been well-documented
as one of the causes of blurred
predicted frames. Our future research
will focus on designing a more appropriate
loss function for our proposed
model, making more rational and efficient
use of the per-layer error module,
and using GAN training methods to
construct an alternative model.
Acknowledgment
This work was supported in part by the
National Key R&D Program of China
under Grants 2018YFC2001600 and
2018YFC2001602; and in part by the
National Natural Science Foundation
ofChina under Grant 61473150.
References
[1] Y. B. Wang, J. J. Zhang, H. Y. Zhu, M. S. Long, J.
M. Wang, and P. S. Yu, " Memory in memory: A predictive
neural network for learning higher-order nonstationarity
from spatiotemporal dynamics, " in Proc.
IEEE Conf. Comput. Vis. Pattern Recog., Long Beach,
CA, USA, 2019, pp. 9146-9154.
[2] H. Gao, H. Z. Xu, Q. Z. Cai, R. Wang, F. Yu,
and T. Darrell, " Disentangling propagation and generation
for video prediction, " in Proc. IEEE Int. Conf.
Comput. Vis., Seoul, Korea, 2019, pp. 9005-9014.
[3] W. Lotter, G. Kreiman, and D. Cox, " Unsupervised
learning of visual structure using predictive generative
networks, " 2015, arXiv:1511.06380.
[4] Y. Lecun et al., " Backpropagation applied to
handwritten zip code recognition, " Neural Comput.,
vol. 1, no. 4, pp. 541-551, Dec. 1989.
[5] S. Hochreiter and J. Schmidhuber, " Long shortterm
memory, " Neural Comput., vol. 9, no. 8,
pp. 1735-1780, Nov. 1997.
[6] X. J. Shi, Z. R. Chen, H. Wang, D. Y.
Yeung, W. K. Wong, and W. C. Woo, " Convolutional
LSTM network: A machine learning
approach for precipitation nowcasting, " in Proc.
Int. Conf. Neural Inf. Process. Syst., Montreal, QC,
Canada, 2015, pp. 802-810.
[7] M. Majd and R. Safabakhsh, " A motion-aware
ConvLSTM network for action recognition, " Appl.
Intell., vol. 49, no. 7, pp. 2515-2521, Jul. 2019.
[8] L. Zhang, G. Zhu, L. Mei, P. Shen, S. A. A. Shah,
and M. Bennamoun, " Attention in convolutional
LSTM for gesture recognition, " in Proc. Int. Conf.
Neural Inf. Process. Syst., Montreal, QC, Canada,
2018, pp. 1957-1966.
[9] B. Lim and S. Zohren, " Time-series forecasting
with deep learning: A survey, " Philos. Trans. Roy. Soc.
A, vol. 379, no. 2194, Apr. 2021.
[10] L. Zhao, L. Shang, Y. Gao, Y. Yang, and X. Jia,
" Video behavior analysis using topic models and
rough sets, " IEEE Comput. Intell. Mag., vol. 8, no. 1,
pp. 56-67, Jan. 2013.
[11] B. K. P. Horn and B. G. Schunck, " Determining
optical flow, " Artif. Intell., vol. 17, no. 1-3,
pp. 185-203, Aug. 1981.
[12] N.Srivastava,G.Hinton,A. Krizhevsky,I.Sutskever,
and R. Salakhutdinov, " Dropout: A simple way to
prevent neural networks from overfitting, " J. Mach. Learn.
Res., vol. 15, no. 1, pp. 1929-1958,Jan. 2014.
[13] W. Lotter, G. Kreiman, and D. Cox, " Deep predictive
coding networks for video prediction and
unsupervised learning, " in Proc. Int. Conf. Learn. Representations,
Toulon, France, 2017.
[14] A. Geiger, P. Lenz, and R. Urtasun, " Areweready
for autonomous driving? The KITTI vision benchmark
suite, " in Proc.IEEEConf. Comput.Vis.Pattern Recog.,
Providence, RI, USA, 2012, pp. 3354-3361.
[15] K. Soomro, A. R. Zamir, and M. Shah,
" UCF101: A dataset of 101 human actions classes
from videos in the wild, " 2012, arXiv:1212.0402.
[16] X. J. Shi et al., " Deep learning for precipitation
nowcasting: A benchmark and a new model, " in Proc.
Int. Conf. Neural Inf. Process. Syst., Long Beach, CA,
USA, 2017, pp. 5622-5632.
[17] B. De Brabandere, X. Jia, T. Tuytelaars, and L. V.
Gool, " Dynamic filter networks, " in Proc. Int. Conf.Neural
Inf. Process. Syst., Barcelona, Spain, 2016, pp. 667-675.
[18] R. Villegas, J. Yang, S. Hong, X. Lin, and H.
Lee, " Decomposing motion and content for natural
video sequence prediction, " in Proc. Int. Conf. Learn.
Representations, Toulon, France, 2017.
[19] Y. Wang, M. Long, J. Wang, Z. Gao, and P. S.
Yu, " PredRNN: Recurrent neural networks for predictive
learning using spatiotemporal LSTM, " in Proc.
Int. Conf. Neural Inf. Process. Syst., Long Beach, CA,
USA, 2017, pp. 879-888.
[20] Y. Wang, Z. Gao, M. Long, J. Wang, and P. S.
Yu, " PredRNNþþ: Towards a resolution of the
deep-in-time dilemma in spatiotemporal predictive
learning, " in Proc. Int. Conf. Int. Conf. Mach. Learn.,
Stockholm, Sweden, 2018, pp. 5110-5119.
[21] Z. Lin, M. Li, Z. Zheng, Y. Cheng, and C. Yuan,
" Self-attention ConvLSTM for spatiotemporal prediction, "
in Proc. AAAI Conf. Artif. Intell.,New York,NY,
USA, vol. 34, no. 7, 2020, pp. 11531-11538.
[22] Q. Wu, X. T. Chen, Z. Y. Huang, and W. M.
Wang, " Generating future frames with mask-guided
prediction, " in Proc. Int. Conf. Multimedia Expo, London,
U.K., 2020, pp. 1-6.
[23] C. Schuldt, I. Laptev, and B. Caputo, " Recognizing
human actions: A local SVM approach, " in
Proc. Int. Conf. Pattern Recognit., Cambridge, U.K.,
2004, vol. 3, pp. 32-36.
[24] P. Dollar, C. Wojek, B. Schiele, and P. Perona,
" Pedestrian detection: A benchmark, " in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., Miami, FL, USA,
2009, pp. 304-311.
[25] P. Dollar, C. Wojek, B. Schiele, and P. Perona,
" Pedestrian detection: An evaluation of the state of
the art, " IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 34, no. 4, pp. 743-761, Aug. 2011.
[26] W. Yu, Y. Lu, S. Easterbrook, and S. Fidler,
" Efficient and information-preserving future frame
prediction and beyond, " in Proc. Int. Conf. Learn. Representations,
Addis Ababa, Ethiopia, 2020.
[27] Y. H. Kwon and M. G. Park, " Predicting future
frames using retrospective cycle GAN, " in Proc. IEEE
Conf. Comput. Vis. Pattern Recog., Long Beach, CA,
USA, 2019, pp. 1811-1820.
[28] Z. Straka, T. Svoboda, and M. Hoffmann, " PreCNet:
Next-frame video prediction based on predictive
coding, " IEEE Trans. Neural. Netw. Learn. Syst.,
early access, Feb. 7, 2023, doi: 10.1109/
TNNLS.2023.3240857.
[29] Y. Huang and R. Rao, " Predictive coding, "
Wiley Interdiscipl. Rev. Cogn. Sci., vol. 2, no. 5,
pp. 580-593, Mar. 2011.
[30] R. P. N. Rao and D. H. Ballard, " Predictive
coding in the visual cortex: A functional interpretation
of some extra-classical receptive-field effects, " Nature
Neurosci., vol. 2, no. 1, pp. 79-87, Jan. 1999.
[31] Y. H. Yu, C. C. Chang, and Y. C. Hu, " Hiding
secret data in images via predictive coding, " Pattern
Recogn., vol. 38, no. 5, pp. 691-705, May 2005.
[32] T. Hosoya, S. A. Baccus, and M. Meister,
" Dynamic predictive coding by the retina, " Nature,
vol. 436, no. 7047, pp. 71-77, Jul. 2005.
[33] R. Kanai, Y. Komura, S. Shipp, and K. Friston,
" Cerebral hierarchies: Predictive processing, precision
and the pulvinar, " Philos. Trans. Roy. Soc. B, vol. 370,
no. 1668, pp. 69-81, May 2015.
[34] W. Schellekens, R. Wezel, N. Petridou, N. F.
Ramsey, and M. Raemaekers, " Predictive coding for
motion stimuli in human early visual cortex, " Brain
Struct. Function,vol. 221, no. 2, pp.879-890, Mar. 2016.
[35] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert,
" High accuracy optical flow estimation based on a
theory for warping, " Lecture Notes Comput. Sci.,
vol. 3024, pp. 25-36, May 2004.
[36] Z. Belhachmi and F. Hecht, " Control of the
effects of the regularization on variational optic flow
computations, " J. Math. Imag. Vis., vol. 40, no. 1,
pp. 1-19, May 2011.
[37] J. Weickert and C. Schn€orr, " Variational optic
flow computation with a spatio-temporal smoothness
constraint, " J. Math. Imag. Vis., vol. 14, no. 3,
pp. 245-255, May 2001.
[38] M. Mathieu, C. Couprie, and Y. Lecun, " Deep
multi-scale video prediction beyond mean square
error, " in Proc. Int. Conf. Learn. Representations, San
http://dx.doi.org/10.1109/TNNLS.2023.3240857 http://dx.doi.org/10.1109/TNNLS.2023.3240857
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
