IEEE Geoscience and Remote Sensing Magazine - December 2023 - 73

name questions. The authors proposed a baseline model
consisting of a pretrained ResNet-152 for encoding the image,
skip-thought vectors for encoding the question, and a
multilayer perceptron classifier for outputting the answer.
Chappuis et al. [103] evaluated the performance of two language
models for processing questions, skip-thought vectors,
and the BERT model. The results demonstrated that
the latter models outperformed the former models when
properly fine-tuned. Guo and Huang [104] proposed a
model that answers questions about the global RS scene
and the local objects within the scene. The model uses a
multiscale fusion module to consider global and local visual
features from the RS image. Zhang et al. [105] proposed
a model that encodes the question via two branches, LSTM
and Seq2Vec. The LSTM output is sent to a cross-modal fusion
module, whose output is concatenated with the question
embedding and the image features.
Yuan et al. [106] proposed a curriculum learning VQA
model that is trained first on the easiest questions before
proceeding to the more difficult questions. Faure et al. [107]
proposed a VQA dataset with questions about spatial relationships
between objects in the RS image. To build the
dataset, they used histograms of forces to model the directional
spatial relations between objects. They trained a VQA
model similar to [99] with a curriculum learning strategy
to improve the understanding of spatial relations. Chappuis
et al. [108] proposed a model that uses a multiclass classification
model to generate keywords from the RS image.
These visual keywords are used as a context, along with the
question, to guide the language model to predict the answer.
Several works have utilized multimodal transformers
to improve the representation of the image and the question
in VQA. For instance, Siebert et al. [109] utilized a
box extractor with ResNet152 to obtain image features
and a BERT tokenizer to tokenize the question. They then
employed VisualBERT to fuse the image and question
features. Bazi et al. [110] used vision and language transformers
based on CLIP [98] to extract visual and textual
representations, respectively. Two decoders were employed
to model the dependencies within and between the two
representations. Finally, two classifiers were used to predict
the answer. Silva et al. [111] enhanced a VQA model
proposed for medical images and applied it to RS. The enhanced
model used Efficient-NetV2 to extract multiscale
features, BERT for tokenizing questions, and RealFormer
as a multimodal encoder.
In [112], the image is preprocessed with an object detector
to generate objectcentric features. The question is
encoded with a language transformer model. The image
and question representations are fed to a cross-modal
transformer network that uses cross-attention between
the image and text to generate the answer. In [113], a VQA
model for open-ended questions is proposed. The authors
employed a text and a vision transformer (ViT) [116] for
encoding the question and the image, respectively. To
generate the answer, a transformer-based decoder with
DECEMBER 2023 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
a cross-attention mechanism receives the concatenated
visual and textual features and generates the answer in
an autoregressive manner. Zhang et al. [114] proposed a
VQA method that improves the visual-spatial reasoning
capability by incorporating hash-based spatial multiscale
visual representation, spatial hierarchical reasoning, and
visual-question interaction modules. The proposed method
addresses challenges posed by geospatial objects with
large-scale differences and position-sensitive properties
to enable accurate answer predictions. Yuan et al. [115]
employed a back-translation data augmentation strategy
to obtain more questions with the same meaning. They
also used contrastive learning to improve the robustness
of their VQA model.
Several studies have proposed VQA methods for postdisaster
damage assessment using aerial images acquired by
unmanned aerial vehicles (UAVs) [117], [118], [119]. These
studies have developed datasets containing questions related
to the affected areas. The proposed VQA methods have the
potential to improve the efficiency and accuracy of postdisaster
damage assessment and aid disaster response efforts.
Table 3 provides a summary of notable VQA models in RS.
CHANGE CAPTIONING
The objective of change detection is to automatically detect
temporal changes in a specific geographical area by
analyzing two or more coregistered images [120]. Change
detection systems typically output a change map that
highlights the spatial distribution of the identified temporal
changes. However, utilizing such maps is not always
straightforward due to the scene's diversity and the sparsity
of the changes.
Inspired by the application of natural language in many
RS tasks, researchers have begun to explore the integration
of natural language into more RS-related applications.
Change captioning, which incorporates natural language
to interpret the detected changes, aims to provide humanlike
language description about the changes in bitemporal
images acquired over the same geographical location. The
change captioning task can be viewed as a combination
of image captioning and change detection. In contrast to
change detection, which only localizes changes at the pixel
level in the form of a change map, change captioning requires
understanding changes at the semantic level. Change
captioning also requires describing the visual changes of
interest in image pairs, as opposed to image captioning,
which only describes the visual content of a single image.
Chouaf et al. [121] explored the change captioning task
in RS and developed a model with an architecture similar
to Figure 7. The model consists of a Siamese-CNN to extract
high-level features from bitemporal images, a fusion block
that merges the images' features, and a language decoder
that translates the identified changes into a description. In
subsequent work, Hoxha et al. [122] proposed a model composed
of two CNNs to extract features from the bitemporal
images, a fusion module, and a decoder based on SVMs or
73
IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023
