IEEE Geoscience and Remote Sensing Magazine - December 2021 - 195

This facilitated simpler and faster algorithms, such as
PCA, and even randomly generated orthogonal rotation
matrices. In addition, much simpler univariate estimators,
such as histograms, were used to significantly speed
up the algorithm. Meng et al. [30] coined the term Gaussianization
flows and extended the iterative algorithm to be
fully parameterized and trainable by incorporating a mixture
of logistics as the marginal Gaussianization layer and
a sequence of Householder flows [40], [41] as the rotation
layer. They also proved that this is a universal approximator
and showed convincing results that Gaussianization
is comparable to other classes of methods specifically designed
for density estimation and sampling [30]. All transformations
and example variants can be found in Table 1.
For details about the theoretical convergence properties of
Gaussianization flows, see [27], [28], and [30].
Regardless of the chosen method, to find the parameters
i for the transformation ,Gi
cost function with respect to i:
we minimize the following
LNx
^i =h hhh
D pGi
7
KL z ^ ^^ A0ID
,,
(3)
which is the Kullback-Leibler (KL) divergence between
the estimated Gaussian distribution and the true multivariate
Gaussian distribution of mean 0 and covariance
I; in other words, this is a measure of how non-Gaussian
our distribution is after transformation. This reveals a direct
relationship with information-theoretic concepts and
measures. Chen [27], [46] showed that (3) can be decomposed
as
L xxm
() () (),TJi =+
(4)
(a)
where ()T x is the total correlation (T) (as well as multi-information
and multivariate MI) between all the marginal
distributions and ()J xm
is the KL divergence between the
marginal distributions and the standard Gaussian normal
distribution. Intuitively, this cost function is trying to
minimize the information shared among each of the marginal
distributions and ensure that they follow a normal
Gaussian distribution. We want to highlight the fact that
RBIG vastly transforms and simplifies the PDF estimation
problem, from directly estimating the density of the highdimensional
multivariate distribution in X to doing it indirectly
through a transformation to a Gaussian domain,
all by using a series of marginal transformations, which are
straightforward and fast.
An example of how RBIG works on a simple 2D toy data
set is provided in Figure 2. We transform a non-Gaussian 2D
data set into a 2D marginal and jointly Gaussian distribution
along with the inverse transformation (first row). The
second row demonstrates how we can use RBIG to synthesize
points in the data domain by using the inverse transformation.
Figure 2(f) shows evolution through iterations
of the final total correlation (as a measure of redundancy)
and the non-Gaussianity (as a measure of the distance to a
DECEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
(d)
(e)
INFORMATION
Shannon information I [47] is based on the idea that a sample,
x
,i
it is less probable. The formal definition of information is
()
Ipgxx
iix
=-lo (( )).
(5)
is more interesting (it carries more information) if
Gaussian). See the RBIG site, https://ipl-uv.github.io/rbig_
jax/, for a working implementation of the RBIG algorithm in
Python and MATLAB.
IT MEASURES USING THE RBIG TRANSFORM
RBIG was designed for density estimation but was inspired
by, and had connections to, IT [6]. The series of
transformations learned by RBIG converts data from the
original domain to a standard multivariate Gaussian
one. The features are marginally independent, which is
important for determining information-theoretic measures
using the Gaussianization scheme. This reduction
in redundancy is iteratively achieved and can be explicitly
computed by summing up all the layer redundancy reductions.
This metric is known as the total correlation, and
computing it enables us to derive information-theoretic
measures from data.
x
(b)
0.6
0.4
0.2
01020
Iterations
(f)
∆ Total Correlation
Non-Gaussianity
FIGURE 2. The density estimation of a sinusoid with heteroscedastic
noise, using RBIG. The original data distribution X is mapped
to a Gaussian domain
Z , with transform Gi
parameterized by a
set of rotations and marginal Gaussianizations collectively denoted
as
i , which has an analytic inverse transformation, xD (),z1
t
= i
-
to
recover the original data. One can sample random data from the
Gaussian in domain Z and use the inverse transformation of z to
xt
for data synthesis. We also demonstrate the losses: the equivalence
of the change in the total correlation between layers TD and
the KL divergence between transformed data and a multivariate
Gaussian (non-Gaussianity). (a)
(d) Z . (e) t
X . (b)
x Gz1
= i
-
generated with matplotlib [74].
195
zG ().x= i
t
(c)
xG ().z
1
= i
-
(). (f) TD and non-Gaussianity Plots were
t
(c)
y
Loss
https://ipl-uv.github.io/rbig_ jax/ https://ipl-uv.github.io/rbig_ jax/
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
