IEEE Geoscience and Remote Sensing Magazine - December 2021 - 193

Having access to all these properties gives us the ability
to tackle long-standing problems in machine learning
and statistics. With accurate PDF estimates, one can model
the conditional densities of data generated from a prior
distribution, develop accurate and efficient compression
schemes, and use principled objective functions, such as
the maximum likelihood. In addition, having access to
an accurate density estimator can be useful in many hybrid
applications to deal with out-of-sample and out-ofdistribution
problems [31]. The problem is, therefore, to
estimate the density x()p
given a set of samples from X .
The simplest approach to PDF estimation assumes that the
density has a parametric functional form defined by a fixed
number of tunable parameters. The Gaussian assumption
is the most widely adopted for unimodal distributions,
which come parameterized by a mean n and a covariance
function .R If more than one mode is assumed, a mixture
of Gaussians (MoG) generally leads to better fits. However,
finding a parametric form for the distribution that fits
properly to particular data is very difficult in most cases.
The alternative technique comes from nonparametric
models, which do not assume a specific form for the
distribution and are learned from data. The simplest nonparametric
method estimates the PDF by partitioning the
data space into nonoverlapping bins, where the density is
estimated as the fraction of data points in the bin divided
by the volume of the bin. This estimator runs the risk of
overfitting or underfitting, depending upon how the bins
are selected. Thus, there are several rule-of-thumb estimators
with a wide range of guidelines for choosing the most
appropriate bin size: 1) an overall good estimator using
Sturges's Rule [32], an estimator that is better for a larger
number of samples and is more robust to outliers by using
the Freedman-Diaconis method [33], and more Bayesian
approaches [34]. However, histogram-based PDF estimation
methods are affected by the curse of dimensionality,
so they cannot be applied to a large number of features.
Alternative parametric estimates that follow probability
estimation schemes for the optimal bin width determined
by the maximum likelihood have been introduced [24].
However, they are very rigid and lead to extremely rough
density functions.
To achieve smoother PDF estimates, KDE is popular. It
places a nonlinear kernel function with a varying bandwidth
parameter to control the degree of smoothness on
top of each example. Unfortunately, a bias-variance tradeoff
will result in over/underfitting the PDF, especially in
moderate- to high-dimensional problems. In the previous
approaches, the bandwidth is typically fixed a priori following
heuristics in the literature [35], and it rarely accounts
for the concentration of points, i.e., that smaller bins
should be placed in regions with a higher concentration of
points, in the form of an adaptive bit allocation scheme.
This can be addressed by using kNNs, which have one
adaptive bandwidth per location and depend on the number
of available training points. However, all the preceding
DECEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
density estimators suffer from the curse of dimensionality:
as the dimensionality increases, the space becomes sparser,
and density estimates are unreliable.
GAUSSIANIZATION FOR PDF ESTIMATION
An alternative way to estimate a PDF from observational
data is to employ a data transformation to a convenient domain
instead of working explicitly in the high-dimensional
input domain. The question of what constitutes a convenient
domain is a long-standing one. Ideally, the domain
should have independent components so that one can work
in each dimension independently to get rid of the curse of
dimensionality. It should enable one to perform operations
and compute quantities therein, and it should be invertible
so that one can express these quantities in meaningful
units of the input domain.
The Gaussian distribution has the desirable properties
of showing independent components and being mathematically
tractable and is thus a good candidate for density
estimation. A class of Gaussianization methods [28],
[30] looks for transforms to a multivariate Gaussian domain.
These transforms are related to projection pursuit
transformations introduced in [42] and seek to transform
a multivariate distribution x(),
p
where x+!X R ,d
Gi: dd
+ p()
! 7 z !
x
+
RR
N
(, ),
0 Id
into a
standardized multivariate Gaussian distribution [27], [28]:
x
(1)
where i are the parameters learned to Gaussianize the
data x, 0 is a vector of zeros (for the means), and Id is
the identity matrix (for the covariance). By construction,
the Gaussianization transform is a parameterized
function Gi
consisting of a sequence of L iterations (or
layers), each performing an orthogonal rotation of the
data and a marginal Gaussianization transformation to
every feature.
The transformation Gi
in each iteration , is defined as
GL01:( ),
i ,, ,,xR x
+1==W
, f ,, ,,
where x0 corresponds to the original data x, W,
ginal Gaussianization of each dimension of x,
is the marfor
the iteration
, , and R, is a rotation matrix for the marginally Gaussianized
variable W,(), x . After convergence in L iterations,
the transformation contains all the needed information to
convert data coming from the original density into a multivariate
Gaussian. Here, i collectively group all parameters:
those from the rotation matrix R and the marginal
transformation
W . For example, one could use a principal
component analysis (PCA) transformation for the rotation
matrix R and a histogram transformation for the marginal
Gaussianization transformation
W . Then, the eigenvectors
obtained from PCA describing R and the parameterizations
of W would define .i See Table 1 for more details on the
decomposition of this formula and Figure 1 for a full decomposition
of a toy data set.
193
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
