IEEE Geoscience and Remote Sensing Magazine - September 2021 - 178

applied similarity matching of the features based on the
2 -
| distance measure. We performed various experiments
to analyze the effectiveness of 1) learning from BigEarthNet-MM
directly (through training from scratch), instead
of using the pretrained models on ImageNet, and
2) state-of-the-art convolutional neural network (CNN)
models trained and evaluated on BigEarthNet-MM. To
use the pretrained models on ImageNet, we used the late
fusion of separately fine-tuned models on Sentinel-1 and
Sentinel-2 patches.
In the experiments, we did not use the Sentinel-2 patches
that are fully covered by seasonal snow, cloud, and
TABLE 1. CLASS-BASED F2 SCORES OBTAINED WHEN
TRANSFER LEARNING FROM IMAGENET AND DIRECT LEARNING
FROM BIGEARTHNET-MM ARE USED FOR MULTIMODAL,
MULTILABEL IMAGE CLASSIFICATION.
CLASS
Urban fabric
Industrial or commercial units
Arable land
Permanent crops
Pastures
Complex cultivation patterns
Land principally occupied by
agriculture, with significant areas of
natural vegetation
Agro-forestry areas
Broad-leaved forest
Coniferous forest
Mixed forest
Natural grassland and sparsely
vegetated areas
Moors, heathland and sclerophyllous
vegetation
Transitional woodland-shrub
Beaches, dunes, sands
Inland wetlands
Coastal wetlands
Inland waters
Marine waters
Average
TRANSFER
LEARNING
FROM
IMAGENET (%)
56.27
30.98
80.05
4.32
50.98
36.29
30.36
2.13
42.83
75.47
72.19
14.11
5.29
41.23
43.67
8.20
4.79
63.23
93.99
39.81
LEARNING
FROM
BIGEARTHNETMM
(%)
71.99
43.21
83.62
55.52
74.77
62.03
60.63
71.87
75.39
86.32
81.31
43.88
59.91
64.21
63.39
57.81
42.23
82.10
97.20
67.23
cloud shadow. After the arrangements of the new class
nomenclature, 57 pairs among the 590,326 pairs were
not associated with any LCLU labels; these pairs are not
used in the experiments. We divided the remaining data
set into 1) a training set of 269,695 pairs of patches, 2) a
validation set of 123,723 pairs of patches, and 3) a test set
of 125,866 pairs of patches.
We performed our experiments on a cluster of four
NVIDIA Tesla V100 GPUs. The results of the multimodal,
multilabel image classification were provided in terms of
four performance metrics: 1) Hamming loss (HL), 2) oneerror
(OE), 3) recall (R), and 4) F2-score (F2). For a detailed
description of the considered metrics, the reader is referred
to Sumbul and Demir [3].
THE EXPERIMENTAL RESULTS
A COMPARISON OF THE STRATEGIES OF LEARNING
DIRECTLY FROM BIGEARTHNET-MM AND TRANSFER
LEARNING FROM IMAGENET
In the first set of experiments, we compare the effectiveness
of learning directly from BigEarthNet-MM with respect to
transfer learning from ImageNet. To this end, the transfer
learning strategy is applied by using the pretrained ResNet50
model trained on ImageNet, while the direct learning strategy
is employed by using the ResNet50 trained from scratch
on BigEarthNet-MM. Table 1 shows the class-based F2 classification
scores (also known as macro-averaged F2 scores [3]).
By analyzing the table, one can see that learning directly
from BigEarthNet-MM achieves the highest score for each
class compared to the transfer learning strategy. As an example,
learning directly from BigEarthNet-MM provides
more than 12% and 25% higher scores for the classes Industrial
or commercial units and Complex cultivation patterns,
respectively, compared to the transfer learning strategy. The
difference in performance between these learning strategies
is more evident for more complex LULC classes. As an
example, learning directly from BigEarthNet-MM improves
the F2 scores by more than 54% and 69% for the classes
Moors, heathland and sclerophyllous vegetation and Agro-forestry
areas, respectively.
In the content of image retrieval, Figure 1 shows an
TABLE 2. OVERALL MULTILABEL CLASSIFICATION RESULTS
UNDER DIFFERENT METRICS AND DL MODELS FOR
BIGEARTHNET-MM.
MODEL
VGG16
VGG19
ResNet50
ResNet101
ResNet152
HL
0.078
0.080
0.074
0.074
0.073
OE (%)
7.35
8.12
5.93
6.46
6.42
R (%)
76.97
76.17
80.05
78.85
78.13
F2 (%)
76.18
75.35
78.73
77.88
77.46
example of a query pair and the retrieved pairs of images
obtained by these strategies. By assessing the figure, one
can observe that when learning is achieved directly from
BigEarthNet-MM, a semantically more similar pairs of
images is retrieved, containing the Urban fabric and Arable
land classes present in the query. Learning directly from
BigEarthNet-MM leads to retrieval of a similar pair to the
query even at the 100th retrieval order. However, using the
transfer learning strategy results in the retrieval of pairs
that do not contain Arable land class, which is present in
the query pair. One can observe this behavior even at the
5th retrieved pair.
The main reasons for the success of directly learning
from BigEarthNet-MM are that 1) transfer learning from
178
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2021
IEEE Geoscience and Remote Sensing Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2021
