Signal Processing - May 2017 - 85

correlation is rectified to a significantly
smaller negative value.
Based on the two requirements, one
can design different rectifiers. One
example is shown in Figure 4(d). It is
called the threshold ReLU (TReLU).
The rectification function can be
defined as TReLU (x) = 0, if x 1 z
and TReLU (x) = ^ x - z h ^1 - z h if
x $ z. When z = 0, TReLU is reduced
to ReLU. For the LeNet-5 applied to
the MNIST data set, we observe better
performance as z increases from 0 to
0.5 and then decreases. One advantage
of TReLU (z) with z 2 0 is that we
can block the influence of more anchor
vectors. When z = 0, we block the
influence of anchor vectors that have an
angle larger than 90c with respect to the
input vector. When z = 0.5, we block
the influence of anchor vectors that have
an angle larger than 60c. The design of
an optimal rectifier for target applications remains to be an open problem.

Multilayer RECOS transform
Single-layer signal analysis
via representation

The CNN approach provides a brand-new
framework for signal analysis. Instead of
finding a representation for signal analysis, it relies on a sequence of cascaded
transforms that builds a link between the
input signal space and the output decision
space. The operation at each layer is to
conduct a spherical surface's clustering of
input samples with a rectified output (i.e.,
the RECOS transform).
For MLPs, each network corresponds
to a simple cascade of multiple RECOS
transforms. Mathematically, we have
d = B L gB l gB 1 x,

(3)

wh e r e x i s a n i n p ut sig n a l , d =
(d 1, f , d c, f , d C) is an output vector in
the decision space indicating the likelihood in class c with c = 1, f , C, and B l
is the lth layer RECOS transform matrix
with l = 1, f , L. The input and output
to the lth layer RECOS transform B l
are denoted by x l -1 and x l, respectively.
Thus, we get
x l = B l x l -1, where B l = R % A l, (4)

Signal modeling and representation is
commonly used in the signal processing
field for signal analysis. Typically, we
have a linear model in form of
x = Ac,

Multilayer signal analysis via
cascaded transforms

(2)

where x ! R N denotes the signal of interest, A ! R N # M is a representation
matrix, and c ! R M is the coefficient
vector. If M = N and the column vectors
of A form a set of basis functions, (2) defines a transform from one basis to another. The task is in selecting powerful basis
functions to represent signals of interest.
Fourier and wavelet transforms are wellknown examples. Then, a subset of coefficient vector c can be used as the feature
vector. If M > N, there exist infinitely
many solutions in c. We can impose constraints on c, leading to the linear leastsquares solution, sparse coding, among
others. For the sparse representation, the
task is in finding a good dictionary, A, to
represent the underlying signal effectively. Again, a subset of coefficient vector c
can be chosen as features.

and where R is the element-wise rectification function operating on the output
of A l x l -1. Clearly, we have x 0 = x and
xL = d.
The ground truth d is that d i = 1 if
i is the target class while d j = 0 if j is
not the target class. It is called the onehot vector. The training samples have
both input x and its label d. The testing
samples have only input x, and we need
to predict its output d and convert it to its
nearest one-hot vector. The task is in finding good B l, l = 1, f , L, so as to minimize the classification error.
For CNNs, we have two types of B l
in the form of
B Cl

=P

' R % A l, s ,

filter at the next layer. The two RECOS
transforms, B Cl and B Fl , are called the
convolutional layer and the fully connected layer, respectively, in the modern
CNN literature. Clearly, B l = B Fl in (4).
It is inspiring to compare the two
signal-analysis approaches as given in
(2) and (3). The one in (2) is a single-layer
approach where no rectification is needed. The one in (3) is a multilayer approach
and rectification is essential. The singlelayer approach seeks for a better signal
representation. For example, a multiplescale signal representation was developed
using the wavelet transform. A sparse signal representation was proposed using a
trained dictionary. The objective is to find
an "optimal" representation to separate
critical components in desired signals
from others.
In contrast, the CNN approach does
not intend to decompose underlying
signals. Instead, it adopts a sequence of
RECOS transforms to cluster input data
based on their similarity layer by layer
until the output layer is reached. The
output layer predicts the likelihood of all
possible decisions (e.g., object classes).
The training samples provide a relationship between an image and its decision
label. The CNN can predict results even
without any supervision, although the
prediction accuracy would be low. The
training samples guide the CNN to form
more suitable anchor vectors (thus, better clusters) and connect clustered data
with decision labels. To summarize,
we can express the multilayer RECOS
transform as
B 1F

B FL -1

and

= R % A l,

s!X

(5)

where A l, s denotes a convolutional filter
at layer l with spatial index s, ' s is the
union of outputs from a neighborhood, Ω,
and P denotes a pooling operation. The
union of outputs from a set of parallel convolutional filters serve as the input to the
IEEE Signal Processing Magazine

|

May 2017

|

B FL

$ x L -1 $ x L = d,
C

C

B1
B2
CNN: x = x 0 $
x1 $
g
C

BF

F

Bm
m+2
m+1
$
x m B$
xm + 1 $
g
B FL - 1

B Fl

B 2F

MLP: x = x 0 $ x 1 $ g

F

BL
x L = d.
$ xL - 1 $

The output from the lth layer, x l, serves
as the input to the (l + 1) th layer. It is
called the intermediate representation at
the lth layer or the lth intermediate representation, in short.
It is important to have a deeper understanding on the compound effect of two
RECOS transforms in cascade. This was
85



Table of Contents for the Digital Edition of Signal Processing - May 2017

Signal Processing - May 2017 - Cover1
Signal Processing - May 2017 - Cover2
Signal Processing - May 2017 - 1
Signal Processing - May 2017 - 2
Signal Processing - May 2017 - 3
Signal Processing - May 2017 - 4
Signal Processing - May 2017 - 5
Signal Processing - May 2017 - 6
Signal Processing - May 2017 - 7
Signal Processing - May 2017 - 8
Signal Processing - May 2017 - 9
Signal Processing - May 2017 - 10
Signal Processing - May 2017 - 11
Signal Processing - May 2017 - 12
Signal Processing - May 2017 - 13
Signal Processing - May 2017 - 14
Signal Processing - May 2017 - 15
Signal Processing - May 2017 - 16
Signal Processing - May 2017 - 17
Signal Processing - May 2017 - 18
Signal Processing - May 2017 - 19
Signal Processing - May 2017 - 20
Signal Processing - May 2017 - 21
Signal Processing - May 2017 - 22
Signal Processing - May 2017 - 23
Signal Processing - May 2017 - 24
Signal Processing - May 2017 - 25
Signal Processing - May 2017 - 26
Signal Processing - May 2017 - 27
Signal Processing - May 2017 - 28
Signal Processing - May 2017 - 29
Signal Processing - May 2017 - 30
Signal Processing - May 2017 - 31
Signal Processing - May 2017 - 32
Signal Processing - May 2017 - 33
Signal Processing - May 2017 - 34
Signal Processing - May 2017 - 35
Signal Processing - May 2017 - 36
Signal Processing - May 2017 - 37
Signal Processing - May 2017 - 38
Signal Processing - May 2017 - 39
Signal Processing - May 2017 - 40
Signal Processing - May 2017 - 41
Signal Processing - May 2017 - 42
Signal Processing - May 2017 - 43
Signal Processing - May 2017 - 44
Signal Processing - May 2017 - 45
Signal Processing - May 2017 - 46
Signal Processing - May 2017 - 47
Signal Processing - May 2017 - 48
Signal Processing - May 2017 - 49
Signal Processing - May 2017 - 50
Signal Processing - May 2017 - 51
Signal Processing - May 2017 - 52
Signal Processing - May 2017 - 53
Signal Processing - May 2017 - 54
Signal Processing - May 2017 - 55
Signal Processing - May 2017 - 56
Signal Processing - May 2017 - 57
Signal Processing - May 2017 - 58
Signal Processing - May 2017 - 59
Signal Processing - May 2017 - 60
Signal Processing - May 2017 - 61
Signal Processing - May 2017 - 62
Signal Processing - May 2017 - 63
Signal Processing - May 2017 - 64
Signal Processing - May 2017 - 65
Signal Processing - May 2017 - 66
Signal Processing - May 2017 - 67
Signal Processing - May 2017 - 68
Signal Processing - May 2017 - 69
Signal Processing - May 2017 - 70
Signal Processing - May 2017 - 71
Signal Processing - May 2017 - 72
Signal Processing - May 2017 - 73
Signal Processing - May 2017 - 74
Signal Processing - May 2017 - 75
Signal Processing - May 2017 - 76
Signal Processing - May 2017 - 77
Signal Processing - May 2017 - 78
Signal Processing - May 2017 - 79
Signal Processing - May 2017 - 80
Signal Processing - May 2017 - 81
Signal Processing - May 2017 - 82
Signal Processing - May 2017 - 83
Signal Processing - May 2017 - 84
Signal Processing - May 2017 - 85
Signal Processing - May 2017 - 86
Signal Processing - May 2017 - 87
Signal Processing - May 2017 - 88
Signal Processing - May 2017 - 89
Signal Processing - May 2017 - 90
Signal Processing - May 2017 - 91
Signal Processing - May 2017 - 92
Signal Processing - May 2017 - 93
Signal Processing - May 2017 - 94
Signal Processing - May 2017 - 95
Signal Processing - May 2017 - 96
Signal Processing - May 2017 - 97
Signal Processing - May 2017 - 98
Signal Processing - May 2017 - 99
Signal Processing - May 2017 - 100
Signal Processing - May 2017 - 101
Signal Processing - May 2017 - 102
Signal Processing - May 2017 - 103
Signal Processing - May 2017 - 104
Signal Processing - May 2017 - 105
Signal Processing - May 2017 - 106
Signal Processing - May 2017 - 107
Signal Processing - May 2017 - 108
Signal Processing - May 2017 - 109
Signal Processing - May 2017 - 110
Signal Processing - May 2017 - 111
Signal Processing - May 2017 - 112
Signal Processing - May 2017 - Cover3
Signal Processing - May 2017 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201809
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201807
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201805
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201803
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201801
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1117
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0917
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0717
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0517
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0317
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0117
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1116
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0916
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0716
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0516
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0316
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0116
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1115
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0915
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0715
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0515
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0315
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0115
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1114
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0914
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0714
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0514
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0314
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0114
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1113
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0913
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0713
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0513
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0313
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0113
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1112
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0912
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0712
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0512
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0312
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0112
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1111
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0911
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0711
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0511
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0311
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0111
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1110
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0910
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0710
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0510
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0310
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0110
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1109
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0909
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0709
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0509
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0309
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0109
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1108
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0908
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0708
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0508
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0308
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0108
https://www.nxtbookmedia.com