IEEE Computational Intelligence Magazine - May 2021 - 11

Publication
Spotlight

Haibo He, University of Rhode Island, USA
Jon Garibaldi, University of Nottingham, UK
Carlos A. Coello Coello, CINVESTAV-IPN, MEXICO
Julian Togelius, New York University, USA
Yaochu Jin, University of Surrey, UK
Yew Soon Ong, Nanyang Technological University, SINGAPORE
Hussein Abbass, University of New South Wales, AUSTRALIA

CIS Publication Spotlight
IEEE Transactions on Neural
Networks and Learning Systems

Digital Object Identifier: 10.1109/
TNNLS.2019.2957109
" The vast quantity of information
brought by big data as well as the evolving computer hardware encourages success stories in the machine learning
community. In the meanwhile, it poses
challenges for the Gaussian process
regression (GPR), a well-known nonparametric, and interpretable Bayesian
model, which suffers from cubic complexity to data size. To improve the
scalability while retaining desirable prediction quality, a variety of scalable GPs
have been presented. However, they
have not yet been comprehensive--
ly reviewed and analyzed to be well
understood by both academia and
industry. The review of scalable GPs in
the GP community is timely and im--
portant due to the explosion of data
size. To this end, this article is devoted
to reviewing state-of-the-art scalable
GPs involving two main categories:
global approximations that distillate the
entire data and local approximations
that divide the data for subspace learnDigital Object Identifier 10.1109/MCI.2021.3061853
Date of current version: 12 April 2021

©BUSINESS & FINANCE-©MASTERSERIES

When Gaussian Process Meets Big
Data: A Review of Scalable GPs, by
H. Liu, Y. -S. Ong, X. Shen, and
J. Cai, IEEE Transactions on Neural
Networks and Learning Systems, Vol.
31, No. 11, November 2020, pp.
4405-4423.

Learning Systems, Vol. 31, No. 10,
October 2020, pp. 4267-4278.

ing. Particularly, for global approximations, we mainly focus on sparse
approximations compr ising pr ior
approximations that modify the prior
but perform exact inference, posterior approximations that retain exact
prior but perform approximate inference, and structured sparse approximations that exploit specific structures in
kernel matrix; for local approximations,
we highlight the mixture/product of
experts that conducts model averaging
from multiple local experts to boost
predictions. To present a complete
review, recent advances for improving
the scalability and capability of scalable
GPs are reviewed. Finally, the extensions
and open issues of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future
research avenues. "
Continual Learning of Recurrent Neural
Networks by Locally Aligning Distributed Representations, by A. Ororbia, A.
Mali, C. L. Giles, and D. Kifer, IEEE
Transactions on Neural Networks and

Digital Object Identifier: 10.1109/
TNNLS.2019.2953622
" Temporal models based on recurrent neural networks have proven to
be quite powerful in a wide variety of
applications, including language modeling and speech processing. However,
training these models often relies on
backpropagation through time (BPTT),
which entails unfolding the network
over many time steps, making the process of conducting credit assignment
considerably more challenging. Furthermore, the nature of backpropagation itself does not permit the use of
nondifferentiable activation functions
and is inherently sequential, making
parallelization of the underlying training process difficult. Here, we propose
the parallel temporal neural coding
network (P-TNCN), a biologically
inspired model trained by the learning
algorithm we call local representation
alignment. It aims to resolve the difficulties and problems that plague recurrent networks trained by BPTT. The
architecture requires neither unrolling
in time nor the derivatives of its internal activation functions. We compare
our model and learning procedure with
other BPTT alternatives (which also
tend to be computationally expensive),
including real-time recurrent learning,
echo state networks, and unbiased
online recurrent optimization. We show
that it outperforms these on-sequence
modeling benchmarks such as Bouncing MNIST, a new benchmark we

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021