Signal Processing - May 2017 - 84

in cascade for the MLP in Figure 1. One
corresponds to a filter bank. The filter
weight vector is called an anchor vector
since it serves as a reference pattern associated with a neuron unit.

a1
X
θ1

θ2

θ3

FIGURE 3. Illustrating the need of rectification [1].

Need of rectification
A neuron computes the correlation
between an input vector and its anchor
vector to measure their similarity. There
are K neurons in one RECOS unit. The
projection of x onto all anchor vectors,
a k, can be written in form of
y = Ax, A T = [a 1 ga k ga K ],

magnitude of their angle, which can be
computed by
i (x i, x j) = cos

-1

(x Ti x j) .

Since cos i is a monotonically decreasing
function for 0c # | i | # 180c, we can use
the correlation, 0 # x Ti x j = cos i # 1,
as another distance measure between
them, and cluster vectors in S accordingly. Note that, when 90c # | i | # 180c,
the correlation, x Ti x j = cos i , is a negative value.
For MLPs and CNNs, a set of neurons
is used to operate on a set of input nodes.
For example, nodes in each hidden layer
and the output layer in Figure 1 take the
weighted sum of values of nodes in the
preceding layer as their outputs. These
outputs are treated as one inseparable unit
that becomes the input to the next layer.
Each neuron has a filter weight vector denoted by a k, k = 1, f , K. In signal processing terminology, the set of neurons
forms a filter bank. The RECOS model
[1] describes the relationship between
nodes of the (l - 1) th and lth layers,
l = 1, 2, f , where the input layer is the
0th layer. There are three RECOS units

1.0

where y = (y 1, f , y k, f y K ) T ! R K , y k =
a Tk x, and A ! R K # N. For input vectors x i
and x j, their corresponding outputs are
yi and yj . If the geodesic distance of x i
and x j in S is close, we expect the distance of yi and yj in the K-dimensional
output space to be close as well.
To show the necessity of rectification, a two-dimensional (2-D) example is
illustrated in Figure 3, where x and a k
(k = 1, 2, 3) denote an input and three
anchor vectors on the unit circle, respectively, and i i is their respective angle.
Since i 1 and i 2 are lower than 90c, a T1 x
and a T2 x are positive. The angle, i 3, is
larger than 90c and correlation a T3 x is
negative. The two vectors, x and a 3,
are far apart in terms of the geodesic
distance. Since cos i is monotonically
decreasing for 0c # | i | # 180c, it can be
used to reflect the order of the geodesic
distance in one layer.
However, when two RECOS units are
in cascade, the filter weights of the second
RECOS unit can take positive or negative
values. If the response of the first RECOS
unit is negative, the product of a negative

response and a negative filter weight will
produce a positive value. On the other
hand, the product of a positive response
and a positive filter weight will also produce a positive value. If the nonlinear
activation unit did not exist, the cascaded
system would not be able to differentiate
them. For example, the geodesic distance
of x and -x should be farthest. However,
they yield the same result, and their original patches become indistinguishable
under this scenario. Similarly, a system
without rectification cannot differentiate the following two cases: 1) a positive
response at the first layer followed by a
negative filter weight at the second layer
and 2) a negative response at the first
layer followed by a positive filter weight
at the second layer.

Rectifier design
Since a nonlinear activation unit is used
to rectify correlations, it is called a rectifier here. To avoid the above-mentioned
confusion cases, we impose the following
two requirements on a rectifier.
1) The output a Tk x should be rectified to
be a nonnegative value.
2) The rectification function should be
monotonically increasing so as to
preserve the order of the geodesic
distance.
Three rectifiers are often used in MLPs
and CNNs. They are the sigmoid function, the ReLU, and the PReLU as shown
in Figure 4(a)-(c). The PReLU is also
known as the leaky ReLU. Both the sigmoid and ReLU satisfy the aforementioned two requirements. Although the
PReLU does not strictly satisfy the first
requirement, it does not have a severe
negative impact on spherical surface
clustering. This is because a negative

0.5
0.0
-6

-4

-2

0
(a)

6 -1

1 -1

(b)

1 -1

(c)

FIGURE 4. An illustration of four rectifiers: (a) the sigmoid function, (b) the ReLU (middle), (c) the leaky ReLU, and (d) the TReLU.
84

IEEE Signal Processing Magazine

May 2017

0
(d)

Table of Contents for the Digital Edition of Signal Processing - May 2017