Signal Processing - May 2017 - 81

lecture notes
C.-C. Jay Kuo

The CNN as a Guided Multilayer RECOS Transform

here is a resurging interest in developing a neural-network-based solution
to the supervised machine-learning
problem. The convolutional neural network (CNN) will be studied in this lecture
note. We introduce a rectified-correlations
on a sphere (RECOS) [1] transform as a
basic building block of CNNs. It consists
of two main concepts: 1) data clustering
on a sphere and 2) rectification. We then
interpret a CNN as a network that implements the guided multilayer RECOS
transform with two highlights. First, we
compare the traditional single-layer and
modern multilayer signal-analysis approaches, point out key areas that enable
the multilayer approach, and provide a
full explanation to the operating principle
of CNNs. Second, we discuss how guidance is provided by labels through backpropagation (BP) in the training.

Relevance
CNNs are widely used in the computer
vision field today. They offer state-ofthe-art solutions to many challenging
vision and image processing problems
such as object detection, scene classification, room layout estimation, semantic
segmentation, image superresolution,
image restoration, and object tracking, to name a few. They are the mainstream machine-learning tool for big
visual data analytics. A great amount
of effort has been devoted to the interDigital Object Identifier 10.1109/MSP.2017.2671158
Date of publication: 26 April 2017

pretability of CNNs based on various
disciplines and tools, such as approximation theory, optimization theory, and
visualization techniques. We explain
the CNN operating principle using data
clustering, rectification, and transform,
which are familiar to researchers and
engineers in the signal processing and
pattern recognition community. As compared with other studies, this approach
appears to be more direct and insightful. It is expected to contribute to further
research advancement on CNNs.

Prerequisites
The prerequisites consist of basic calculus, probability, and linear algebra. Statistics and approximation techniques could
also be useful but are not necessary.

Problem statement
We will study the following three problems in this note.
1) Neural networks architecture evolution. We provide a survey on the
architecture evolution of neural networks, including computational neurons, multilayer perceptrons (MLPs),
and CNNs.
2) Signal analysis via multilayer RECOS
transform. We point out the differences between the single- and multilayer
signal-analysis ap proaches and
explain the working principle of the
multilayer RECOS transform.
3) Network initialization and guided
anchor vector update. We carefully
examine the CNN initialization scheme
IEEE Signal Processing Magazine

May 2017

since it can be viewed as an unsupervised clustering and the CNN selforganization property can be explained.
The supervised learning is achieved
by BP using data labels in the training
stage. It will be interpreted as guided
anchor vector update.

Solution
CNN architecture evolution
We divide the architectural evolution of
CNNs into three stages and provide a
brief survey.

Computational neuron
A computational neuron (or simply neuron) is the basic operational unit in neural networks. It was first proposed by
McCulloch and Pitts in [2] to model the
"all-or-none" character of nervous activities. It conducts two operations in cascade: an affine transform of input vector
x followed by a nonlinear activation function. Mathematically, we can express it as
y = f (b),
b = Ta (x)
N

/ an xn + a0 n

n=1

= a T x + a 0 n = al T xl,

(1)

where y, x = (x 1, f , x N ) T ! R N , and
a = (a 1, f , a N ) T ! R N are the scalar
output, the N-dimensional input and model
parameter vectors, respectively; a 0 n is
N
a bias term with n = ^1 N h/ n = 1 x n
(i.e., the mean of all input elements), f (.)
81

Table of Contents for the Digital Edition of Signal Processing - May 2017