IEEE Geoscience and Remote Sensing Magazine - September 2022 - 257

INTRODUCTION
Recently, machine learning algorithms have produced
promising results for land cover mapping from remote sensing
data [1]. Convolutional neural networks (CNNs) have
especially shown interesting properties on relatively simple
data sets [2], [3]. However, it appears that further work is
needed to efficiently use them on wide geospatial scenes
[4]. In particular, the diversity of soil materials, their spectral
similarities, and intraclass variability [5] are such that
the task at hand gets harder as the image gets larger. Having
a large and representative training data set is then crucial.
Labeling pixels of remote sensing images can, however,
be very time consuming and expensive as it might require
field measurements or tedious photo interpretations. One
would feel dizzy just considering the following: one small
geospatial scene of 1 km # 1 km contains as much as millions
of pixels for high spatial resolution images; spatially
closed pixels are likely redundant, making the annotation
of big polygons far from optimal; and urban environments
are full of small and spectrally heterogenous objects that do
not characterize the land cover. Therefore, building a large
and representative ground truth has become a key challenge
in the remote sensing community.
To facilitate and guide data annotation, AL techniques
have been extensively studied in machine learning literature.
The category of AL methods of interest in the present
study is referred to as pool-based AL: a query system selects
the most informative samples, and an oracle labels the given
samples. The oracle is usually an expert that can annotate
the data with confidence. Other AL methods are based on
the synthesis of training samples with generative models.
Zhu and Bento [6] use an uncertainty criterion to sample informative
samples from a generative adversarial network on
handwritten digits and natural images. Nevertheless, in our
case, it is not possible to label a spectrum without the context
given by surrounding pixels. A large neighborhood (at
least 10 # 10 pixels) is needed to distinguish many classes,
such as high vegetation from short vegetation or even tile
from bare soil. If works have been carried out in the remote
sensing community to synthesize hyperspectral spectra with
generative models [7], [8], synthetizing hyperspectral cubes,
to the best of our knowledge, has not yet been achieved with
generative models. For this reason, we kept generative-based
methods out of the scope of the present review. Moreover,
we only consider techniques for which the oracle provides
the true labels, in contrast with pseudolabeling frameworks
in which some labels are given by a classifier [9]. Pseudolabeling
approaches rely on the following hypothesis, which
does not hold true in an operational context: confident predictions
(when the classifier shows a high probability) are
always correct, and therefore can be used as annotations.
It has been shown that machine learning models can make
wrong predictions with high confidence, especially if the
training data set is not representative of the whole image.
For this reason, we argue that pseudolabeling methods are
not reliable enough to be used in an operational scenario.
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
Pool-based query systems follow different strategies, and
previous reviews in 2010 and 2011 suggested various taxonomies
[10], [11]. The recent advances have built on existing
heuristics or paved the way toward new approaches that may
lead to promising results for understanding geospatial scenes
[12]-[15]. AL on airborne hyperspectral imaging, especially,
has aroused strong interest due to the richness and complexity
of their spatial-spectral information [11]. We focus on
hyperspectral data in this study. We believe, however, that
our results could be extended to other data, such as satellite
multispectral imaging or tasks other than classification.
For instance, the authors in [16] use AL methods to enhance
change-detection algorithms and [17] apply AL algorithms
for object detection. In those cases where information lies
in rare and specific samples, we believe that AL is even more
crucial. Typically, using random sampling in the context of
change or object detection would yield very poor results.
Different works have tested AL techniques on small hyperspectral
benchmark data sets such as Pavia University
and Pavia Center, Indian Pines, or the Kennedy Space Center
[18]-[20]. To the best of our knowledge, however, none of
the most competitive methods have been experimented on
large and complex data sets. Moreover, the metrics used up
until then to evaluate AL approaches may not be representative
of their performance in a real use case. Therefore, our
goal with the present article is to present insights and guidelines
for choosing an AL method in operational conditions.
The key contributions of this article are as follows:
◗ We bring under the same framework and benchmark six
state-of-the-art AL methods, among which, the most recent
ones have not yet been tested on remote sensing data.
◗ We assess the performance of AL techniques with regard
to new metrics that provide indications on the behavior
of query systems in a real-case scenario.
◗ We introduce a preprocessing routine to fully leverage
AL methods on large and complex data.
◗ We provide an open source toolbox to benchmark and
run additional AL methods.
AL: AN OVERVIEW
The goal of AL algorithms is to build representative training
data sets at minimal cost. The flowchart of pool-based AL
methods is given in Figure 1. Given an initial labeled training
data set, data points are queried from an unlabeled data
set (termed as the pool) to maximize an acquisition function.
The queried points are labeled by an oracle and enrich
the training data set. The process is repeated over and over
for a given number of steps.
Pool-based AL methods differ in the choice of the acquisition
function (that we will interchangeably refer to as the
query system) and can be cast in three main categories: uncertainty,
representativeness, and performance-based AL.
UNCERTAINTY-BASED AL
Uncertainty heuristics rely on machine learning models
that make predictions from data, given the training data
257
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
