Computational Intelligence - November 2015 - 48
clusters are shown in Figure 13(a) in two
dimensions. The axes were chosen as the
first two principal components.
All 1000 vectors were trained. The network was not "told" which vector belonged
to which of the clusters. The 1000 input vectors were not labeled in any way. After convergence, the network produced 100-bit output words for each input vector.
Ten distinct 100-bit output words were observed, each corresponding to one of the clouds. For a given 100-bit output
word, all input vectors that caused that output word were given
a specific color. The colored input points are shown in Figure 13(b). The colored points associate exactly as they did in
the input clouds.
The uncolored points were trained into the network and
they were "colored by the network." The network automatically produced unique representations for each of the clouds.
This was a relatively easy problem since the number of clouds,
10, was much less than the network capacity, 100.
Figure 14 illustrates how Hebbian-LMS creates binary
outputs after the above training with the 1000 patterns.
One of the neurons in the output layer was selected and
histograms were constructed for its (SUM) before and after
training, and for its half-sigmoid output before and after
training. The histograms show that, before training, the histogram of the (SUM) was not binary and the histogram of
the half-sigmoid output appears to be almost binary but it
is not so. Observing the colors, one can see that some of
the clusters were split apart. After training, the histogram
of the half-sigmoid output shows the clusters to be intact
and all together.
The Hebbian-LMS algorithm exhibits homeostasis
about the two equilibrium points, caused by reversal
of the error signal at these equilibrium points.
3
3
2
2
Second Principal Component
Second Principal Component
So, if the input pattern is close to a training pattern, the
output error will be close to zero. If the input pattern is
distinct from all the training patterns, the output error will
be large. One could use this when one is not asking the
neural network to classify an input pattern, merely to indicate if the input pattern has been trained in, i.e., seen
before or not, deja vu, yes or no? This could be used as a
critical element of a cognitive memory system [12].
In yet another application, a multi-layer neural network could
be trained using both supervised and unsupervised methods. The
hidden layers could be trained with Hebbian-LMS and the output
layer could be trained with the original LMS algorithm.
An individual input cluster would produce an individual
binary "word" at the output of the final hidden layer. The output layer could be trained with a one-out-of-many code. The
output neuron with the largest (SUM) would be identified as
representing the class of the cluster of the input pattern.
A three layer purely Hebbian-LMS network was simulated with 100 neurons in each layer. The input patterns
were 50-dimensional, and the network outputs, binary after
training, were 100-bit binary numbers. A set of training patterns was generated as follows. Ten random vectors were
used as representing ten clusters. Clusters were formed as
clouds about the ten original vectors. Each cloud contained
100 randomly disposed points. The ten 50-dimensional
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
1
-2
0
2
4
6
-4
-4
First Principal Component
(a) Before Training.
Figure 13 50-dimensional input vectors plotted along the first two principal components.
48
IEEE ComputatIonal IntEllIgEnCE magazInE | novEmbEr 2015
-2
0
2
First Principal Component
(b) After Training.
4
6
Table of Contents for the Digital Edition of Computational Intelligence - November 2015
Computational Intelligence - November 2015 - Cover1
Computational Intelligence - November 2015 - Cover2
Computational Intelligence - November 2015 - 1
Computational Intelligence - November 2015 - 2
Computational Intelligence - November 2015 - 3
Computational Intelligence - November 2015 - 4
Computational Intelligence - November 2015 - 5
Computational Intelligence - November 2015 - 6
Computational Intelligence - November 2015 - 7
Computational Intelligence - November 2015 - 8
Computational Intelligence - November 2015 - 9
Computational Intelligence - November 2015 - 10
Computational Intelligence - November 2015 - 11
Computational Intelligence - November 2015 - 12
Computational Intelligence - November 2015 - 13
Computational Intelligence - November 2015 - 14
Computational Intelligence - November 2015 - 15
Computational Intelligence - November 2015 - 16
Computational Intelligence - November 2015 - 17
Computational Intelligence - November 2015 - 18
Computational Intelligence - November 2015 - 19
Computational Intelligence - November 2015 - 20
Computational Intelligence - November 2015 - 21
Computational Intelligence - November 2015 - 22
Computational Intelligence - November 2015 - 23
Computational Intelligence - November 2015 - 24
Computational Intelligence - November 2015 - 25
Computational Intelligence - November 2015 - 26
Computational Intelligence - November 2015 - 27
Computational Intelligence - November 2015 - 28
Computational Intelligence - November 2015 - 29
Computational Intelligence - November 2015 - 30
Computational Intelligence - November 2015 - 31
Computational Intelligence - November 2015 - 32
Computational Intelligence - November 2015 - 33
Computational Intelligence - November 2015 - 34
Computational Intelligence - November 2015 - 35
Computational Intelligence - November 2015 - 36
Computational Intelligence - November 2015 - 37
Computational Intelligence - November 2015 - 38
Computational Intelligence - November 2015 - 39
Computational Intelligence - November 2015 - 40
Computational Intelligence - November 2015 - 41
Computational Intelligence - November 2015 - 42
Computational Intelligence - November 2015 - 43
Computational Intelligence - November 2015 - 44
Computational Intelligence - November 2015 - 45
Computational Intelligence - November 2015 - 46
Computational Intelligence - November 2015 - 47
Computational Intelligence - November 2015 - 48
Computational Intelligence - November 2015 - 49
Computational Intelligence - November 2015 - 50
Computational Intelligence - November 2015 - 51
Computational Intelligence - November 2015 - 52
Computational Intelligence - November 2015 - 53
Computational Intelligence - November 2015 - 54
Computational Intelligence - November 2015 - 55
Computational Intelligence - November 2015 - 56
Computational Intelligence - November 2015 - 57
Computational Intelligence - November 2015 - 58
Computational Intelligence - November 2015 - 59
Computational Intelligence - November 2015 - 60
Computational Intelligence - November 2015 - 61
Computational Intelligence - November 2015 - 62
Computational Intelligence - November 2015 - 63
Computational Intelligence - November 2015 - 64
Computational Intelligence - November 2015 - 65
Computational Intelligence - November 2015 - 66
Computational Intelligence - November 2015 - 67
Computational Intelligence - November 2015 - 68
Computational Intelligence - November 2015 - 69
Computational Intelligence - November 2015 - 70
Computational Intelligence - November 2015 - 71
Computational Intelligence - November 2015 - 72
Computational Intelligence - November 2015 - 73
Computational Intelligence - November 2015 - 74
Computational Intelligence - November 2015 - 75
Computational Intelligence - November 2015 - 76
Computational Intelligence - November 2015 - 77
Computational Intelligence - November 2015 - 78
Computational Intelligence - November 2015 - 79
Computational Intelligence - November 2015 - 80
Computational Intelligence - November 2015 - Cover3
Computational Intelligence - November 2015 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com