IEEE Geoscience and Remote Sensing Magazine - June 2019 - 166

COMPARATIVE STUDY OF DEEP-LEARNING
ARCHITECTURES FOR HYPERSPECTRAL DATA

classification space. This architecture is illustrated in Figure 6, with the 1D convolution in pink, the 2D convolutions in blue, and pooling in orange. It is worth noting that,
thanks to its fully convolutional architecture, this network
generates predictions for all pixels in the input patch and
not only for the central one. This means that it is more efficient at inference time when sliding over the whole image.

DEEP LEARNING FOR HYPERSPECTRAL TOOLBOX
Despite the abundant literature on data-driven hyperspectral image processing, there are only a few tools available
that can be used to compare the state-of-the-art methods in
a common framework. To this end, we provide DeepHyperX
(https://github.com/nshaud/DeepHyperX), a deep-learning
toolbox based on the PyTorch framework that can be used
to benchmark neural networks on various public hyperspectral data sets. It includes many supervised approaches,
ranging from linear SVM to state-of-the-art 3D CNNs. Indeed, many models from the literature are implemented in
the toolbox, including 1D CNNs for spectral classification
(Figure 7, with 1D convolutions in pink, pooling in orange,
and fully connected layers in blue) and the most recent 2D
and 3D CNNs as suggested in [49], [63], [68], and [71] (see
Figure 8 for a 3D convolutional network architecture example, with 3D convolutions in green, 3D pooling in orange,
and 1D fully connected layers in blue). The models can be
trained and evaluated on several data sets from the literature, e.g., Pavia Center/University, Indian Pines, Kennedy
Space Center, or DFC2018, described in the section "Reference Data Sets." Many hyperparameters can be tuned to
assess the impact of spatial features, the size of the training
set, or the optimization strategy. This toolbox allows us to
evaluate how the methods from the state of the art compare
in different use cases.
Technically, this toolbox is written in Python and consists
of an interface around the PyTorch and scikit-learn libraries.
Deep networks are implemented using PyTorch that can leverage both CPU and graphics processing unit based on the
user needs, while we rely on scikit-learn for the SVM implementations. Several public classification data sets are preconfigured for easy investigation of deep models. The modular
architecture of the toolbox allows users to easily add new remote sensing data sets and new deep architectures.

on
v
C

O
ut

×
1

1
v
on
C

pu
t

1
×

1
×
1
on
v

on
v

yp
er

THE 3D CONVOLUTIONAL NEURAL NETWORK
If spatial-spectral methods already reach particularly satisfying results, especially with respect to the spatial regularity, they
require a high level of engineering in their design that is not
fully compatible with the data-to-output motto that defines
deep learning. A promising approach [68] directly handles the
hyperspectral cube with 3D CNNs, which work simultaneously on the three dimensions using 3D convolutions. This
conceptually simpler approach slightly improves the classification performances with respect to 2D + 1D models. Many
architectures have been proposed to handle 3D CNNs for hyperspectral data, mostly to investigate well-known techniques
from deep learning for computer vision, such as multiscale
feature extraction [69] and semisupervision [70]. Instead
of producing 2D feature maps, these 3D CNNs produce 3D
feature cubes that are well-suited for pattern recognition in a
volume and seem at least theoretically more relevant for hyperspectral image classification. [71] especially showed that
3D CNNs for classification of hyperspectral images obtained
higher performances than their 2D counterparts.
Indeed, with respect to spectral or 2D + 1D CNN, 3D
CNNs combine those two pattern recognition strategies
into one filter, requiring fewer parameters and layers. They
can learn to recognize more complex 3D patterns of reflectances: colocated spectral signatures, various differences
of absorption between bands, and so on. However, all directions are not equivalent in the hyperspectral data cube,
so these patterns cannot be processed as mere volumetric
data. As for the 2D + 1D CNN, it implies extra care when
designing the network, such as using anisotropic filters.

×
5

"Tree"

LRN

Residual
Connection

FIGURE 6. The 2D + 1D CNN for classification of hyperspectral data with a residual learning mechanism as proposed in [49]. LRN: local
response normalization.

166

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

JUNE 2019

https://www.github.com/nshaud/DeepHyperX

IEEE Geoscience and Remote Sensing Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2019