Computational Intelligence - May 2015 - 76

14
12

ELM
QKLMS

Number

10
8
6
4
2
0

2
Testing MSE

Algorithm

4
-3

#10

Testing MSE
Best

Average

ELM

0.00017028

0.00092244

QKLMS

0.00014414

0.00040414

Figure 7 Histogram of the errors across realizations of QKLMS and ELM, and the best and
average results.

number of data samples. However, they
are practically different. The ESN and
ELM use random projection spaces,
while the KLMS uses data centered
functional bases. The ESN and ELM still
suffer from design choices, translated in
free parameters, which are difficult to set
optimally with the current mathematical
framework, so practically they involve
many trials and cross validation to find a
good projection space, on top of the
selection of the number of hidden PEs
and the nonlinear functions. On the
other hand, the KLMS algor ithm
answers this question easily: just map the
data nonlinearly and deterministically to
a Hilbert space and use the Hilbert

space functions as the bases selected by
the input data (the representer theorem),
and adapt online the projection. The
KAF and its data centered basis functions have the great advantage of concentrating bases on the part of the functional space where the input data exist.
Therefore, KAFs are more parsimonious
and can span the cloud of points in the
joint space with fewer bases, at least for
some applications (e.g. for prediction
where the targets and inputs come from
the same cloud). Moreover, we can
expect that the eigenspread of the basis
vector matrix is reasonable, because
there is at least one data sample per basis.
For the Gaussian kernel, the basis func-

Table 1 Comparison of CULMs.
eSN

elM

QKlMS

DynaMic (D)/Static
Mapping (S)
RanDoM baSES/cRoSS
vaLiDation
coMpLExity foR N
SaMpLES/ M baSES

yES

O ^NM + M 3h

O ^NMh

fREE paRaMEtERS

DiStRibution, Rng,
oRDER, nonLinEaRity,
SpEctRaL RaDiuS,
SpaRSEnESS

DiStRibution,
Rng, oRDER,
nonLinEaRity

KERnEL, SizE, StEpSizE, Quantization
RaDiuS

IEEE ComputatIonal IntEllIgEnCE magazInE | may 2015

tions are local, and can be controlled by
a single parameter, the kernel bandwidth
(kernel size). In the Hilbert space, the
generalization is mainly controlled by
two parameters. Smaller kernel sizes
decrease the correlation between the
basis functions, so practically, the fundamental compromise of correlated bases
is easy to control in KLMS without
affecting the computational complexity.
The second parameter is the stepsize
that controls the weight norm, where a
small stepsize also improves the generalization [33]. However, a decrease in the
learning rate achieves a better solution at
the expense of using more samples to
converge. To sparsify the bases a third
parameter is needed to control the
quantization radius. The ESN is the only
dynamic network of the three, but it
turns out that KAF has been recently
extended to recurrent structures [48].
The KAFs have been mostly applied in
adaptive filtering (regression for static
data), but recent results show that their
characteristics for classification are also
very competitive with those of ELMs.
Therefore, KAFs are also ready to be
applied by the engineering and machine
learning communities.
The current literature on ELMs fails
to acknowledge the dependency of the
results on the careful offline selection of
the basis functions. Therefore, the reported results should be interpreted as
best case. Moreover, the computation
time in finding the best basis functions,
which can be considerable, should also
be included in the ELM training time.
Theoretically, three important questions
remain in the ELM approach. How
good is the best basis set? How does one
find it reliably without cross validation?
And, is there a strategy to help select the
size of the space and the RNG parameters for the problem? Here it is proper
to recall Chen's framework [44] of incrementally adding random PEs after
successive linearized projections to the
original problem, which avoids the selection of all the weights randomly and
should decrease the correlation amongst
the basis functions. In terms of the KAF,
the randomness in KLMS appears because of the stochastic gradient and can

Table of Contents for the Digital Edition of Computational Intelligence - May 2015