Computational Intelligence - November 2015 - 48

clusters are shown in Figure 13(a) in two
dimensions. The axes were chosen as the
first two principal components.
All 1000 vectors were trained. The network was not "told" which vector belonged
to which of the clusters. The 1000 input vectors were not labeled in any way. After convergence, the network produced 100-bit output words for each input vector.
Ten distinct 100-bit output words were observed, each corresponding to one of the clouds. For a given 100-bit output
word, all input vectors that caused that output word were given
a specific color. The colored input points are shown in Figure 13(b). The colored points associate exactly as they did in
the input clouds.
The uncolored points were trained into the network and
they were "colored by the network." The network automatically produced unique representations for each of the clouds.
This was a relatively easy problem since the number of clouds,
10, was much less than the network capacity, 100.
Figure 14 illustrates how Hebbian-LMS creates binary
outputs after the above training with the 1000 patterns.
One of the neurons in the output layer was selected and
histograms were constructed for its (SUM) before and after
training, and for its half-sigmoid output before and after
training. The histograms show that, before training, the histogram of the (SUM) was not binary and the histogram of
the half-sigmoid output appears to be almost binary but it
is not so. Observing the colors, one can see that some of
the clusters were split apart. After training, the histogram
of the half-sigmoid output shows the clusters to be intact
and all together.

The Hebbian-LMS algorithm exhibits homeostasis
about the two equilibrium points, caused by reversal
of the error signal at these equilibrium points.

2
Second Principal Component

Second Principal Component

So, if the input pattern is close to a training pattern, the
output error will be close to zero. If the input pattern is
distinct from all the training patterns, the output error will
be large. One could use this when one is not asking the
neural network to classify an input pattern, merely to indicate if the input pattern has been trained in, i.e., seen
before or not, deja vu, yes or no? This could be used as a
critical element of a cognitive memory system [12].
In yet another application, a multi-layer neural network could
be trained using both supervised and unsupervised methods. The
hidden layers could be trained with Hebbian-LMS and the output
layer could be trained with the original LMS algorithm.
An individual input cluster would produce an individual
binary "word" at the output of the final hidden layer. The output layer could be trained with a one-out-of-many code. The
output neuron with the largest (SUM) would be identified as
representing the class of the cluster of the input pattern.
A three layer purely Hebbian-LMS network was simulated with 100 neurons in each layer. The input patterns
were 50-dimensional, and the network outputs, binary after
training, were 100-bit binary numbers. A set of training patterns was generated as follows. Ten random vectors were
used as representing ten clusters. Clusters were formed as
clouds about the ten original vectors. Each cloud contained
100 randomly disposed points. The ten 50-dimensional

1
0

-1

-2

-3

-3
-4
-4

-2

-4
-4

First Principal Component
(a) Before Training.
Figure 13 50-dimensional input vectors plotted along the first two principal components.

IEEE ComputatIonal IntEllIgEnCE magazInE | novEmbEr 2015

-2

First Principal Component
(b) After Training.

Table of Contents for the Digital Edition of Computational Intelligence - November 2015