IEEE Geoscience and Remote Sensing Magazine - December 2023 - 68

datasets in RS. In [16], a CNN-LSTM model is presented, but
instead of using the high-level image features, the word embedding
vectors of the detected objects are used to initialize
the hidden state of an LSTM at the initial time step. Lu et al.
[17] explored several deep learning encoder-decoder models
and compared them with handcrafted visual features.
They also extended the encoder-decoder model by adding a
visual attention layer. The results indicated that deep learning
models with an attention mechanism outperformed
other methods.
Many studies in RS image captioning have utilized attention
mechanisms to improve the encoder-decoder architecture.
This mechanism is placed between the encoder
and decoder to emphasize the relevant regions in the image
when generating the corresponding words. Different
attention techniques have been implemented for this
purpose in the field of RS. For instance, Zhang et al. [18]
proposed a model that integrates an attribute attention
component into the encoder-decoder model to consider
not only the objects but also their attributes when generating
the caption. In [19], a model with three levels of
attention was presented: attention to different areas of the
image, different words, and vision and semantics. Zhang
et al. [20] used a CNN to predict the label of the input image,
which is then utilized to guide the calculation of the
attention mask. Additionally, Chen et al. [21] proposed a
geospatial relation captioning model using an attentionbased
LSTM to describe the relationships between objects
in RS images.
In [22], a two-branch encoder model was proposed. The
first branch uses a CNN with spatial and scalewise attention
components to extract multilevel features, while the
second branch employs a graph convolutional network to
explore attribute words and their relationships. The extracted
features from both branches are then fed into the
decoder to generate the caption. Other studies have utilized
attention-based global-local feature representation
to model the relationships between global and local image
regions as well as between image regions and their corresponding
text [23], [24]. Additionally, some works have
employed semantic segmentation to generate the attention
map, such as that by Cui et al. [25], who proposed an attention-based
semantic segmentation and spatial relationship
recognition model based on a CNN, a U-Net, and an LSTM
network. In [26], an attention mechanism is proposed that
utilizes pixel-level segmentation masks to generate more
accurate descriptions.
Recent studies have proposed various enhancements
to the attention mechanism in RS image captioning. For
Vision
Encoder
Language
Decoder
instance, Wu et al. [27] proposed fusing the current hidden
state of the LSTM and the global visual features of
the RS image to improve the attention mechanism. Meng
et al. [28] presented a multiview attention model composed
of two attention modules that integrate features
from several views. The model is trained with a weight
discrepancy term to force the two modules to focus on
different information. Furthermore, Cheng et al. [29]
proposed an architecture with multilevel and contextual
attention that adaptively aggregates different levels of
features and introduced the NWPU-captions dataset, a
large-scale dataset for image captioning with 31,500 images
and 157,500 captions.
Zhang et al. [30] improved the attention mechanism
by training the model on a new loss function called visual
aligning loss, which filters out nonvisual words based on the
similarity between the attended image region and the corresponding
visual word. In addition, Zhang et al. [31] proposed
a model with two attention modules: the encoder's
global visual feature-guided attention, which combines
global and local visual features, and the decoder's linguistic
state-guided attention, which provides textual features in
each decoding time step. Wang et al. [32] introduced the
intensive positioning layer designed to extract the coordinates
of important regions and region features from the RS
image. Kumar et al.
[33] proposed another region-based
method that multiplies the extracted image features by the
image classification features to increase the probability of
finding important regions in the image.
Given that scale variation among objects is an inherent
Caption:
Aircrafts are parked
at a terminal.
FIGURE 4. The general architecture of the RS image captioning model.
68
characteristic of RS images, researchers have explored various
methods to address this issue. For example, Wang et al.
[34] proposed a model that extracts features from multiple
levels: the object level, patch level, and global level. These
features are then fused together and used to generate the
attention mask. In [35], two CNNs are used for encoding
the image: one for extracting large-scale features and the
other for extracting small-scale features. Then, a multiscale
LSTM is used for decoding. Huang et al. [36] extracted multiscale
features from different layers of the CNN and fused
them with the denoising operation. Ma et al. [37] proposed
two multiscale methods: one extracts features from different
layers and uses the multihead attention mechanism
to obtain the context feature, and the other combines the
target-level features and the scene-level features by using
the target-detection task as an auxiliary task. Wang et al.
[38] collected multiscale features from different stages of
the CNN and enhanced these features by using a multi-interaction
feature representation module. In [39], the image
is represented by a global feature derived
from multiscale features. The
decoder consists of recurrent attention,
attention LSTM, and language
LSTM, which together transform
visual features into a caption. Recently,
Zhang et al. [40] proposed a
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2023
IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023
