Computational Intelligence - February 2016 - 50

Deep structures with multiple layers of non-linear
operations is able to learn high-level abstraction
needed to solve challenging vision, language,
and other AI-level tasks. Ensemble of deep learners
can boost the performance.
conversion method called extreme randomized discretization (ERD) to convert continuous values to discrete values.
The ERD grouped the continuous values into bins where the boundaries were
created randomly in order to create
ensembles. The authors evaluated the
proposed method with several regression
datasets and they concluded that the
ensemble method outperformed the
ensemble regression converted to classification method with equal-width discretization bins. They also found that a
greater number of bins yielded better
performance.
IV. Conclusion

This paper has reviewed the state-ofthe-art on ensemble classification and
ensemble regression. The theories relating to ensemble classification and
regression have been discussed including
bias-variance decomposition and the
diversity issue. Conventional ensemble
methods have been introduced and
recent improvements to the conventional ensemble methods have also been discussed. State-of-the-art ensemble
methods such as multi-objective optimization based ensembles, multiple kernel
learning, and fuzzy ensembles have also
been reviewed. The early stage development of ensemble incorporating deep
learning has also been surveyed. Some
divide-and-conquer based ensemble
methods specialized for time series forecasting have also been presented.
The paper has also reviewed some
methods that convert regression problems into multiple-class classification
problems and that apply ensemble classification to regression problems.
V. Future Work

Although there are numerous recent
ensemble methods for solving classification and regression problems reported

in the literature, there is still ample room
for improvement. There are several
promising research directions for ensemble classification and regression as discussed below.
"Big Data" [176] has attracted considerable attention recently. It is worthy
to investigate the benefits of ensemble
learning for solving big data problems.
We can also investigate the benefits of
ensemble methods for solving other
machine learning tasks such as clustering
[177], [178].
There are learning algorithms which
are unstable [13], [25], [179], especially
randomized learning approaches such as
random vector functional link (RVFL)
neural network. According to a recent
comprehensive evaluation [180], there is
a large gap in performance between
randomized learning algorithms and
rank 1 methods (Random Forest). On
the other hand, according to [25], highly
unstable methods are naturally suitable
for ensembles. That is, the variance of
ensemble can be reduced significantly.
Therefore, it is worthwhile to investigate
the performance of ensemble methods
with fast randomized base learners such
as RVFL.
There are few ensemble classification
and regression methods containing fuzzy
systems as base predictors. Multi-objective optimization based ensemble classification and regression methods are also
under-researched in the literature. Further research on fuzzy ensemble and
multi-objective optimization based
ensemble methods is recommended. A
possible direction is to combine fuzzy
systems and optimization algorithms
because ensemble methods with fuzzy
systems require much more parameter
tuning. Hence, employing optimization
algorithms will accelerate the process.
For multi-objective optimization based
ensemble methods, we would like to

IEEE ComputatIonal IntEllIgEnCE magazInE | FEbruary 2016

highlight the possibility of applying
optimization to different base predictors
other than feed-forward neural networks. We can also investigate the applicability of ensemble methods for evolutionary algorithms [181], [182].
In Section III-G, we reviewed some
deep learning based ensemble classification and regression methods such as
CNN ensembles and deep SVM ensembles. Current deep learning approaches
aim to generate deep representations for
the data. However, it is common sense
that deep neural networks are very difficult to train because of the vanishing gradient. Recent research remedies this by
either initializing the network with some
unsupervised training approach such as
restricted Boltzmann machine or reducing the number of parameters by 'sharing
weights' such as CNN. However, it
remains an open question how well these
approaches can regularize the network.
Ensemble methods, which can be
regarded as complementary to the current research, boost the performance of
neural networks by reducing the variance
significantly as we mentioned in Section
II. In the future, researchers should further investigate deep learning based
ensemble methods. Several questions may
be addressed: Is it necessary to pretrain
the 'base deep network' in an unsupervised manner in the ensemble? Is it possible to increase the diversity of the 'base
deep network' without losing too much
accuracy in the ensemble? How can one
develop advanced ensemble methods
which are suitable for deep learning?
Ensemble regressors for approximating fitness landscapes in the context of
evolutionary and swarm algorithms are
also under-researched [183], [184].
Moreover, deep learning ensembles are a
future research direction. Although certain deep learning ensemble classification methods can be easily transformed/
migrated to deal with regression problems, we need further development on
deep learning ensemble regression algorithms. Deep RVFL ensembles are a
promising approach due to their diversity and fast training.
Deep and complex models are much
more difficult to train than shallow and

Table of Contents for the Digital Edition of Computational Intelligence - February 2016