IEEE Geoscience and Remote Sensing Magazine - June 2019 - 167

lte

...

.
..

...

...
..
.

"Tree"

lte
rs
2

← C2 →

n
← C1 →

k
Output

← C1 →

lte
rs

...

on
v
C
...

.
..

.
..
...

←C →

←
5→

...

← 5→

Sp
pectrum
p

ol
2

ol
in
g
...

H
yp
er

on
v

lte

tru

BEST PRACTICES
During training, the algorithm of choice for optimizing
the network's weights is the backpropagation algorithm
Designing and optimizing deep networks can seem arcane,
[72]. This technique relies on the stochastic gradient deespecially considering the abundant literature on this topic
scent (SGD) or one of its variants. Many versions have been
that has been published since 2010. However, a principled approposed in the literature, using more or less hyperparamproach can drastically ease the process of experimenting with
eters. The fundamental hyperparameter is the learning rate
deep learning for novel applications. A deep-learning experiment can be divided in three broad stages: building a model,
a that controls the magnitude of the weight update after
optimizing the model, and running inferences. We are aseach iteration. A too high a will cause the loss to diverge
suming here that the model will be used for classification usor to oscillate around the local minimum without reaching a classification loss function, such as the cross entropy.
ing it, while a too small a will be very slow to converge. In
Building a model can often be summed up as choosing
practice, it is recommended to train with the highest a that
a model from the literature. Most of the time, it is better to
makes the loss not diverge at first, and then slowly decreasrely on a well-validated architecture from the state of the
ing it during the training. For example, our toolbox uses an
art than spending time designing one from scratch. On the
adaptive policy that divides a by one order of magnitude
basis of the data set and the application, choosing a model
when the validation error plateaus. An alternative is to use
can be done on the following criteria:
an SGD variant with an adaptive learning rate, such as the
popular one from [73].
◗ If the data set is an image that presents spatial correlations, a
Dealing with the backpropagation of a large number
2D approach will outperform pixel-wise classifiers in most
of weights can be complex. Optimizing deep networks incases. For hyperspectral images, 3D CNNs will be able to
volves working with many parameters. One fundamental
leverage correlations in all three dimensions of the data.
best practice that is often overlooked is the virtue of ini◗ Using bigger models means that there are more parametialization. Weights are randomly initialized at the start of
ters to optimize, requiring, in turn, more training samples.
the training, but various strategies have been developed so
◗ Large convolutional kernels tend to be slower, especially
in 3D. Most implementations are
optimized for small 2D kernels.
◗ FCNs are more efficient because
they can predict several pixels at a
time. Moreover, the absence of fully
connected layers means that they
will have a lot fewer parameters,
therefore being easier to train.
"Tree"
◗ Nonsaturating activation functions, such as rectified linear
k
units (ReLUs), alleviate vanishing
gradients and help build deeper
FC1 FC2 FC3
networks while being faster to
compute than sigmoid or tanh
FIGURE 7. The 1D CNN for spectral classification of hyperspectral data available in the toolbox.
alternatives.

FIGURE 8. The 3D CNN for classification of hyperspectral data available in the toolbox. It reproduces the architecture proposed in [71] and

alternates 3D convolutions and 3D max-pooling layers.
JUNE 2019

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

167

IEEE Geoscience and Remote Sensing Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2019