IEEE Geoscience and Remote Sensing Magazine - December 2023 - 75

descriptions. A text encoder is utilized to translate the input
text description into an embedding of a fixed size. Then,
generative adversarial networks (GANs) are used to synthesize
the equivalent image conditioned on the encoded text.
This model represents the text as a multilabel binary vector
that indicates the presence and absence of objects in the
description. In their subsequent work, Bejiga et al. [125] focused
on enhancing the text-encoding schema to represent
not only objects in the text but also their attributes and spatial
relationships. For this, they proposed an enhanced text
encoder based on Doc2Vec. However, the images generated
by these works are gray scale with low spatial resolution.
Chen et al. [126] proposed using GANs to synthesize satellite
images with a spatial size of 128 # 128. These images
are utilized to augment the training set for the change detection
task. In [127], a model for generating high-resolution
RS images is proposed. The model used Hopfield layers in
addition to a self-attention layer to
learn representative prototypes from
text-image embeddings. These prototypes
are then decoded to synthesize
the image. In [128], a multistage
structured GAN is proposed to improve
the authenticity and the quality
of the synthesized image.
CHANGE VISUAL QUESTION
ANSWERING
Change VQA is the task that combines
VQA and change detection.
It upgrades the change detection
interface to allow the user to ask
questions about the change of interest
in image pairs. Yuan et al.
[129], [130] explored this task and
proposed a VQA model for change
detection in multitemporal images.
The general architecture of
the change VQA model is shown in
Figure 9. It consists of two CNNs for
encoding multitemporal images, a
multitemporal fusion block to fuse
the features of the two images, a
multimodal fusion block for fusing
the vision features with the textual
question, and an answer prediction
block. The model can provide answers
in natural language depending
on the difference between two
RS images. The model is trained using
an autogenerated dataset from a
change detection dataset.
VISUAL QUESTION GENERATION
Visual question generation (VQG)
is a newly explored task in RS, as
Question:
Have the
regions of
trees
decreased?
FIGURE 9. The general architecture of the RS change VQA model.
illustrated in Figure 10. The task is closely related to VQA
but has the objective of generating questions, instead of
answers, from the visual information in the RS image. It is
also analogous to image captioning, as it takes an RS image
as input and produces text as output. This task has been
explored in [131], which proposed a model consisting of a
ViT encoder and a GPT-2 language decoder [132] to generate
a paragraph of diverse questions. The article also introduced
a VQA dataset with manually annotated questions
and answers.
VISUAL GROUNDING AND SEMANTIC LOCALIZATION
Visual grounding is a task in RS that aims to locate the
object referred to by a query in natural language within
an RS image. As depicted in Figure 11, the output of visual
grounding is a bounding box that identifies the object
within the given image, given the image and a query
Vision
Encoder
Vision
Encoder
Language
Decoder
Language
Encoder
Answer:
Yes
Vision
Encoder
Language
Decoder
Question:
How many
airplanes in the
scene?
FIGURE 10. The general architecture of the RS VQG model.
Vision
Encoder
The Airplane
on the Top
Language
Encoder
FIGURE 11. The general architecture of the RS visual grounding model.
DECEMBER 2023 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
75
Fusion
Fusion
Fusion
IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023
