IEEE Geoscience and Remote Sensing Magazine - December 2023 - 66

status of almost all vision-language tasks in RS, including
but not limited to image captioning, text-image retrieval,
VQA, text-image generation, and other tasks. For each task,
we provide a summary of the state-of-the-art methods, illustrations,
and numerical results for evaluating the different
proposed methodologies, followed by a discussion of current
challenges and possible future works in this field.
NATURAL LANGUAGE PROCESSING
MEETS REMOTE SENSING
In recent years, considerable efforts have been devoted to developing
methods that deploy NLP methods for RS applications.
In this section, we present a comprehensive review of
more than 100 NLP-RS works published in the RS community.
These works are categorized into different research tasks, including
image captioning, text-to-image retrieval, and VQA.
In addition to these tasks, related tasks, such as change captioning,
change VQA, and RS image generation, have also
been investigated. Despite the differences in their objectives,
all of these tasks share a common characteristic of
comprehending both RS images and text to generate informative
outputs.
IMAGE CAPTIONING
Image captioning is the first and most active research area
in RS visual-language modeling. Its objective is to provide
a concise and meaningful textual description of RS images
using natural language, which requires simulating human
vision to recognize the complex RS scene and interpreting
the recognized image content with human-readable text.
This task involves describing the objects in the image as well
as their attributes, quantity, and relationships, which can significantly
enhance the understanding
and interpretation of RS data.
The literature on image captioning
3%
3%
17%
Retrieval
114
Papers and Articles
17%
VQA
55%
Captioning
2%
(a)
IEEE Transactions on
Geoscience and Remote
Sensing: 23%
Other: 34%
114
Papers and Articles
ISPRS International
Journal of
Geo-Information: 3%
IEEE Access: 4%
IEEE Journal of Selected Topics in Remote Sensing: 9%
Applied Earth Observations and
Remote Sensing: 6%
(b)
FIGURE 3. The distribution of published articles by (a) task and (b) journal.
66
IEEE Geoscience
Remote Sensing
Letters: 10%
International
Geoscience and
Remote Sensing
Symposium
(IGARSS): 11%
in RS conveys several methodologies
to solve the problem. Generally,
these methods, which are summarized
in Table 1, can be divided
into three categories: template-based
methods [12], retrieval-based methods
[13], and encoder-decoderbased
methods [14]. Template-based
methods approach image captioning
as an object detection problem.
In this approach, a fixed number
of sentence templates are defined,
and a fully convolutional network is
used to detect objects that appear in
the scene. The recognized objects are
then used to fill in the blank slots in
the templates [12]. In retrieval-based
methods, the image representation
and text representation are mapped
into a common semantic space. At
inference time, the model searches
for the most similar image to the
query image based on a distance
metric, and the caption corresponding
to the retrieved image is assigned
as a caption to the query image [13].
Although template- and retrievalbased
approaches can produce
captions that are grammatically and
syntactically correct, they have limitations
in generating captions that are
both diverse and natural. The templatebased
approach tends to produce simplistic
and repetitive sentences
that
lack the descriptive quality of humangenerated
captions. On the other hand,
retrieval-based methods are restricted
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2023
IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023
