Computational Intelligence - November 2016 - 25

Abstract

Multivariate Analysis (MVA) comprises a family of well-known methods for feature
extraction which exploit correlations among input variables representing the data.
One important property that is enjoyed by most such methods is uncorrelation
among the extracted features. Recently, regularized versions of MVA methods have
appeared in the literature, mainly with the goal to gain interpretability of the solution. In these cases, the solutions can no longer be obtained in a closed manner, and
more complex optimization methods that rely on the iteration of two steps are frequently used. This paper recurs to an alternative approach to solve efficiently this
iterative problem. The main novelty of this approach lies in preserving several properties of the original methods, most notably the uncorrelation of the extracted features. Under this framework, we propose a novel method that takes advantage of the
, 2, 1 norm to perform variable selection during the feature extraction process.
Experimental results over different problems corroborate the advantages of the proposed formulation in comparison to state of the art formulations.

direction, while a supervised method like OPLS (right
subplot) successfully identifies the most discriminative
information. Although this toy example is based on a
classification task, the same advantages of supervised
MVA over standard PCA are encountered in regression
tasks-see [7] for a detailed theoretical and experimental review of these methods.
The simplicity of these methods, as well as the availability of highly-optimized libraries for solving the linear algebra problems they involve, justifies the extensive use of
MVA in many application fields, such as biomedical
engineering [8], [9], remote sensing [10], [11], or chemometrics [12], among many others (see also [7] for
a more detailed review of application-oriented
research in the field).
An important property of PCA, OPLS, and
CCA is that they lead to uncorrelated variables, so that the feature extraction process
provides additional advantages:
❏ The relevance of each extracted feature is
directly given by the magnitude of its associated eigenvalue, which simplifies the
selection of a reduced subset of features,
if necessary.
❏ Subsequent learning tasks are simplified, more
notably, when the covariance matrix inversion is
required. This is the case of least-square based
problems, such as Ridge Regression or lasso (least
absolute shrinkage and selection operator) [13].
Standard versions of MVA methods implement just a feature
extraction process, in the sense that all original variables are used
to build the new features. However, over the last few years there

Corresponding Author: Sergio Muñoz-Romero (e-mail: sergio.munoz@urjc.es).

have been many significant contributions to this field that have
focused on gaining interpretability of the extracted features by
incorporating sparsity-inducing norms, such as the , 1 and , 2, 1
norms [14], as a penalty term in the minimization problem.
When these regularization terms are included, the projection
vectors are favored to include zeros in some of their components, making it easier to understand the process to build the
new features and thus gaining in interpretability. In fact, the , 2, 1
rewards solutions that perform a real variable selection process,
in the sense that some of the original variables are excluded
from all projection vectors at once. In other words, only a subset
of the original variables are used to build the new features.
Some of the most significant contributions in this direction
are sparse PCA [15], sparse OPLS [8], group-lasso penalized
OPLS (also known as Sparse Reduced Rank Regression,
SRRR) [16], and , 2, 1-regularized CCA (or L21SDA) [17]. All
these approaches are based on an iterative process which combines the optimization of two coupled least-squares problems,
one of them subject to a minimization constraint. Since the
inspiring work [15], this constrained least-squares minimization
has been typically treated as an orthogonal Procrustes problem
[18], an approach that can still be considered mainstream (see,
e.g., the very recent works [19], [20]).
A first objective of this paper is to highlight and make the
computational intelligence community aware of some limitations
derived from the use of orthogonal Procrustes in the context of
regularized MVA methods. As explained in [21], these methods
1) do not converge to their associated non-regularized MVA
solutions when the penalty term is removed,
2) are highly dependent on initialization, and may even fail
to progress towards a solution,
3) do not in general obtain uncorrelated features.
As solution to these problems, [21] proposes an alternative optimization procedure avoiding the use of the Procrustes solution. In
this paper, we will briefly review the framework presented in [21]

novEmbEr 2016 | IEEE ComputatIonal IntEllIgEnCE magazInE

Table of Contents for the Digital Edition of Computational Intelligence - November 2016