Signal Processing - May 2017 - 87

that CNNs can provide a wide range of
learning paradigms-from the unsupervised, weakly supervised, to heavily supervised learning. This intuition
can be justified by considering a proper
anchor vector initialization scheme (for
self-organization) and providing proper
"guidance" to the proposed multilayer
RECOS transform in anchor vector
update (for supervised learning).

Random Initialization

(a)
K-Means Initialization

Network initialization

(b)

FIGURE 6. The comparison of MNIST unsupervised classification results of the LeNet-5 architecture with the (a) random and (b) k-means
initializations, where the images that are closest
to centroids of ten output nodes are shown.

Guided anchor vector update
We apply the BP for network training with a varying number of training
samples. For a fixed number of training
samples, we train the network until its

Accuracy Versus Train Sample Count

100
90
80
Accuracy (%)

The CNN conducts a sequence of representation transforms using cascaded
RECOS units. The dimension of transformed representations gradually decreases until it reaches the number of
output classes. Since labels of output
classes are provided by humans with a
semantic meaning, the whole end-to-end
process is called the guided (or supervised) transform.
Before examining the effect of label
guidance, we first compare two network
initialization schemes: 1) the random
initialization and 2) the k-means initialization. For the latter, we perform
k-means at each layer based on its corresponding input data samples (with zeromean and unit-length normalization),
and we repeat this process from the input
to the output layer after layer. Today, random initialization is commonly adopted.
Based on the previous discussion, we
expect the k-means initialization to be
a better choice. This is verified by our
experiments in the LeNet-5 applied to
the MNIST data set.
Once the network is initialized, we
can feed the test data to the network and
observe the output, which corresponds
to unsupervised learning. The comparison of unsupervised classification
results with the random and k-means
initializations is given in Figure 6, where
we show images that are closest to the
anchor vectors (or centroids) of the ten
output nodes. We see that the k-means
initialization provides ten anchor vectors pointing to ten different digits while
the random initialization cannot do the
same. Different random initialization
schemes will lead to different results,
yet the one given in Figure 6 is representative. That is, multiple anchor vectors
will point to the same digit.

performance converges and plot the correct classification rate in Figure 7. The
two points along the y-axis indicate the
correct classification rates without any
labeled training sample. The rates are
around 32 and 14% for the k-means and
random initializations, respectively. Note
that the 14% is slightly better than the random guess on the outcome, which is 10%.
Then, both performance curves increase
as the number of labeled training samples
grows. The k-means can reach a correct
classification rate of 90% when the number of labeled training samples is around
250, which is only 0.41% of the entire
MNIST training data set (i.e., 60,000 samples). This shows the power of the LeNet-5
even under extremely low supervision.
To further understand the role played
by label guidance, we examine the impact
of the BP on the orientation of anchor
vectors in various layers. We show in
Table 1 the averaged orientation changes
of anchor vectors in terms of radian (or

70
60
50
40
30
Random
k-Means

20
10

200

400
600
Number of Samples

800

1,000

FIGURE 7. The comparison of MNIST weakly supervised classification results of the LeNet-5 architecture with the random and k-means initializations, where the correct classification rate is plotted as a
function of training sample numbers.
Table 1. The averaged orientation changes of anchor vectors in terms of the radian (or degree)
for the k-means and the random initialization schemes.
In/Out Layers

k-Means

Random

Input/S2

0.155 (or 8.881°)

1.715 (or 98.262°)

S2/S4

0.169 (or 9.683°)

1.589 (or 91.043°)

S4/C5

0.204 (or 11.688°)

1.567 (or 89.783°)

C5/F6

0.099 (or 5.672°)

1.579 (or 90.470°)

F6/output

0.300 (or 17.189°)

1.591 (or 91.158°)

IEEE Signal Processing Magazine

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017