IEEE Geoscience and Remote Sensing Magazine - December 2023 - 64

for future development, with the aim of inspiring further
research in this important field.
INTRODUCTION
RS technologies have advanced rapidly in recent years,
making a large amount of Earth observation data readily
available. This has provided researchers with significant
information for approaching various geoscience applications
in novel ways. Consequently, many efforts have
been made to automatically understand the visual content
of RS scenes, with research mainly focused on object
recognition [1]; scene classification [2], [3]; and semantic
segmentation [4] methods. However, these methods typically
produce a single label or multiple labels describing
the main observation in the scene, which provide limited
information about the complex collection of objects, environments,
and semantics present in the RS scene.
To address this limitation, researchers have turned
to language, the most prevalent means of communication
consistent with cognitive behavior in humans, to
assist in understanding RS images. Combining text with
visual understanding can facilitate the interpretation
of RS scenes, enabling researchers to move beyond discrete
labels to provide more comprehensive and descriptive
information about the topology and distribution of
multiple objects and their attributes. Moreover, a key objective
in RS data processing is to develop user-friendly
interfaces that utilize models capable of perceiving the
visual information in RS scenes and communicate this
information to humans and other agents in natural language
[5]. NLP is essential in achieving this objective,
as it can boost interactivity with the RS model, make it
more accessible to a broader audience, and provide human-level
interpretation of the information contained in
RS data. By combining visual and textual understanding,
vision-language models are poised to revolutionize RS
data interpretation.
10
15
20
25
30
35
40
45
50
5
Vision-language models, which bridge the fields of
computer vision and NLP, have gained significant interest
and exploration [6]. In the field of RS, vision-language
modeling tasks encompass a wide range of tasks, including
RS image captioning, RS text-to-image retrieval, RS VQA,
and RS image generation, among others. Accomplishing
these tasks requires integrating information from both the
RS and NLP fields to produce the output. This involves
interaction, transformation, and fusion of two different
modalities: RS data represented by pixel intensities and
textual descriptions represented as a sequence of words.
The rapid development of computer vision, NLP, and
deep learning technologies has made it possible to combine
these two fields and create powerful vision-language
models that can tackle a wide range of RS applications.
The integration of NLP into RS data processing is a
fast-developing research topic that has gained increasing
attention from the community. Figure 1 provides a chronological
overview of articles published annually from 2016 to
the first half of 2023, categorized into different research
tasks. As shown in Figure 1, the field is progressing at a rapid
pace. For example, the number of publications in 2022
is approximately double that in 2021. Moreover, the interest
in combining NLP techniques with RS understanding is
evident not only in the number of publications but also in
the range of tasks explored each year. In 2020, researchers
investigated three tasks. This number increased to five in
2021 and to six in 2022. The large number of articles and
the steady increase in both publications and tasks confirm
the surge of interest in integrating NLP with the visual understanding
of RS data.
To pursue this sustainable development, this article
VQG
Visual Grounding
Change VQA
Change Captioning
Image Generation
Retrieval
VQA
Captioning
2016 2017 2018 2019 2020 2021 2022 2023
(First Half)
FIGURE 1. A chronological overview of the articles and papers published annually from 2016
to the first half of 2023, categorized into different research tasks.
64
presents a comprehensive review summarizing recent advances
in the cross-field of RS data processing and NLP.
Particularly, we review recent works related to visual-language
modeling tasks in RS, including image captioning,
text-to-image retrieval, VQA, change captioning, image
generation, and change VQA. Furthermore,
to facilitate quantitative
evaluation, the article summarizes
existing evaluation resources, such
as datasets and automatic evaluation
metrics. To collect the state of
the art for this review study, we followed
the process specified by the
Preferred Reporting Items for Systematic
Reviews and Meta-analyses
statement [7]. We first collected related
articles via Web of Science,
Google Scholar, and IEEE Xplore, using
keywords such as " remote sensing
image caption, " " remote sensing
text image retrieval, " " remote sensing
VQA, " and " remote sensing image
generation " to retrieve relevant
papers and articles. This generated a
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2023

IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023