IEEE Geoscience and Remote Sensing Magazine - September 2021 - 179

ImageNet limits the accurate characterization of the spectral
content of RS images; 2) fine-tuning the pretrained
model on ImageNet by using RS images cannot be sufficient
to eliminate the semantic gap since the category
labels present in ImageNet are different from the land
cover class labels present in BigEarthNet-MM; and 3) the
pretrained model was trained for a single-label image
classification scenario and thus limits the accurate characterization
of the multiple land cover classes present in
BigEarthNet-MM.
A COMPARISON OF STATE-OF-THE-ART CNN MODELS
In the second set of experiments, we compare the effectiveness
of the VGG and ResNet models in the framework of
multimodal, multilabel classification. Table 2 shows the
overall classification results under different metrics (which
are the sample-averaged scores [3]). By analyzing the table,
one can observe that the ResNet model provides the highest
scores in all metrics. As an example, ResNet50 achieves
more than 2% higher recall and F2 scores compared to the
VGG models. This improvement is due to the residual connections
of the ResNet model and their increased depth in
terms of the number of layers compared to the VGG model.
Increasing the depth of the considering models does not
significantly affect the performances; i.e., similar scores are
obtained in all of the metrics under different depth values
of the same model.
DISCUSSION AND CONCLUSION
In this article, we have presented the BigEarthNet-MM
benchmark archive containing 590,326 pairs of Sentinel-1
and Sentinel-2 image patches with a new CLC-based class
nomenclature to pay more justice to the properties of the
considered images. BigEarthNet-MM makes a significant
advancement for the use of DL in RS, opening up promising
directions to support research studies in the framework of
multimodal, multilabel RS image scene classification and
retrieval. BigEarthNet-MM is suitable to assess DL-based
methods for 1) learning from class-imbalanced multimodal
data (since the LCLU classes are not equally represented in
BigEarthNet-MM); 2) transfer learning (since BigEarthNetMM
currently contains only pairs of images from a small
number of European countries); and 3) unsupervised, selfsupervised,
and semisupervised multimodal learning for
information discovery from big data archives.
It is worth noting that BigEarthNet-MM has limitations
for the RS applications that require time-series data to accurately
describe LCLU classes, such as Nonirrigated arable
land and Permanently irrigated land. We also note that some
Sentinel-1 image patches can be contaminated by artifacts
caused by either well-known radio frequency interference
[16] or other data set-related issues, which are independent
of the preprocessing steps applied while constructing BigEarthNet-MM.
As a final remark, we would like to point
out that, because of the use of labels from the CLC map, the
BigEarthNet-MM archive can be extended to a larger scale
SEPTEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
within Europe with zero annotation cost. As a future development
of this work, we plan to enrich the BigEarthNetMM
archive by extending it to the whole of Europe.
ACKNOWLEDGMENTS
This work is funded by the European Research Council
(ERC) through the ERC-2017-STG BigEarth Project under
grant 759764 and by the German Ministry for Education
and Research as BIFOLD-the Berlin Institute for the Foundations
of Learning and Data (01IS18025A). The authors
from Direção-Geral do Território are supported through
Fundação para a Ciência e a Tecnologia-UIDB/04152/2020Centro
de Investigação em Gestão de Informação.
AUTHOR INFORMATION
Gencer Sumbul (gencer.suembuel@tu-berlin.de) is with the
Technische Universität Berlin, Berlin, 10587, Germany. He
is a Graduate Student Member of IEEE.
Arne de Wall (a.dewall@tu-berlin.de) is with the Technische
Universität Berlin, Berlin, 10587, Germany. He is a
Student Member of IEEE.
Tristan Kreuziger (tristan.kreuziger@tu-berlin.de) is
with the Technische Universität Berlin, Berlin, 10587, Germany.
He is a Student Member of IEEE.
Filipe Marcelino (fmarcelino@dgterritorio.pt) is
with Direção-Geral do Território, Lisbon, 1099-052,
Portugal.
Hugo Costa (hugoagcosta@gmail.com) is with DireçãoGeral
do Território, Lisbon, 1099-052, Portugal.
Pedro Benevides (pbenevides@dgterritorio.pt) is with
Direção-Geral do Território, Lisbon, 1099-052, Portugal.
Mário Caetano (dgtcaetano@gmail.com) is with Direção-Geral
do Território, Lisbon, 1099-052, Portugal.
Begüm Demir (demir@tu-berlin.de) is with the Technische
Universität Berlin, Berlin, 10587, Germany. He is a
Senior Member of IEEE.
Volker Markl (volker.markl@tu-berlin.de) is with the
Technische Universität Berlin, Berlin, 10587, Germany.
REFERENCES
[1] S. Roy, E. Sangineto, B. Demir, and N. Sebe, " Metric-learningbased
deep hashing network for content-based retrieval of
remote sensing images, " IEEE Geosci. Remote Sens. Lett., vol.
18, no. 2, pp. 226-230, Feb. 2021. doi: 10.1109/LGRS.2020.
2974629.
[2] Y. Li, Y. Zhang, X. Huang, H. Zhu, and J. Ma, " Large-scale remote
sensing image retrieval by deep hashing neural networks, " IEEE
Trans. Geosci. Remote Sens., vol. 56, no. 2, pp. 950-965, 2018. doi:
10.1109/TGRS.2017.2756911.
[3] G. Sumbul and B. Demir, " A deep multi-attention driven approach
for multi-label remote sensing image classification, " IEEE
Access, vol. 8, pp. 95,934-95,946, May 2020. doi: 10.1109/ACCESS.2020.2995805.
[4]
G. Cheng, X. Xie, J. Han, L. Guo, and G. Xia, " Remote sensing image
scene classification meets deep learning: Challenges, methods,
benchmarks, and opportunities, " IEEE J. Sel. Topics Appl.
179
IEEE Geoscience and Remote Sensing Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2021
