IEEE Geoscience and Remote Sensing Magazine - June 2022 - 282

incorporated topic information with image features during
the forward process by creating a novel retrieval-topic recurrent
memory unit based on an LSTM network. In [259],
the authors found that environmental sound can be useful
to indicate the attention or topic of an image, then added
an extra branch to encode audio information in the previous
architecture. More recently, Wang et al. proposed a
general word-sentence network with a word extractor and
sentence generator for RS image captioning with clear explainability
[260].
RS IMAGE QUESTION ANSWERING
Although captioning methods have allowed for automatic
descriptions of RS images, these approaches output general
results, which summarize image information. However, at
times, people focus only on the information of interest, and
the general descriptions become uninformative. As a result,
RS image question answering has been developed [250],
[261], [262]. Question-answering tasks query sentences as
input and produce corresponding answers on the basis of
the information for RS images. Accordingly, a model can
accurately output results that the user cares most for and
help filter information. Given that the methods and data
sets must be well defined due to the high complexity of
visual question-answering (VQA) tasks, its application in
RS data has been neglected for a long time. Recently, Lobry
et al. recommended describing RS images in the form of
question answering because they considered it a direct way
to retrieve desired information from RS images. A method
is created to generate two RS image VQA data sets and techniques
to solve VQA questions and support model development
[250]. Zheng et al. also shared a mutual attention
inception network and an RSIVQA data set with more than
37,000 images with questions and corresponding answers
for RS image VQA [251].
RS IMAGE GENERATION WITH SEQUENTIAL DATA
The inversion of captioning tasks is text-to-image generation,
which considers description sentences as inputs to
reproduce the visual appearance of target contents and has
been shown to be useful in the computer vision community
[263]. In the field of RS, several studies have explored
RS image-generation methods. Bejiga et al. [252] first used
a conditional image generator model to reconstruct RS
data from ancient text documents: a recurrent model is
used to encode description sentences into a latent space
and utilized a GAN to synthesize an image on the basis
of the latent code. Zhao and Shi developed a multistage,
structured GAN to structurally synthesize RS images, given
text descriptions [264]. Another task with a similar pipeline
is audio-to-image generation. Specifically, models generate
the corresponding images with audio data as inputs.
To generate RS images, the authors in [253] suggested first
translating audio data into a sequential representation (e.g.,
sentences) and then producing images using text-to-imagegeneration
models.
282
RS IMAGE GENERATION WITH AUDIO SEQUENCE
In addition to text documents, audio data are also an important
source of extra information that can be used to
assist the applications of RS data. In comparison with text
data that describe the high-level information already present
in RS images, audio data provide current or additional
information. Such data add further details to the scene presented
in RS images, and consequently, its use is also explored
to help understand RS images. At times, RS images
are paired with audio data (i.e., sound documents recording
the environment presented in RS images). However,
image and sound data are usually separately stored due
to format limitations. Accordingly, an effective retrieval
mechanism is desired. To create such a system, Guo et al.
offered a multimodal system that maps audio and image
data into the same feature space and retrieves them on the
basis of similarities in embedding features [255]. Chen
et al. suggested using NNs to map paired image-audio data
to the same feature space, and allocated hash-like codes to
achieve improved efficiency and minimal quantization errors
during the calculation [254]. More recently, Ning et al.
proposed semantics-consistent representation learning
with the consideration of pairwise, intramodality, and nonpaired
intermodality relationships for RS image-voice data
sets cross-modal retrieval [265]. Aside from information retrieval,
audio information is also used to help improve the
models for RS scene recognition. Hu et al. developed ADVANCE,
a multimodal data set that provides paired aerial
image and audio data, which can benefit the recognition of
aerial images in several approaches [256].
AI SECURITY
Security and reliability are important factors when addressing
RS tasks. Although AI techniques have remarkably
improved the interpretation performance of RS data
in the past few decades, its potential risks have been observed.
For example, the deployed AI system might be attacked
by specific deception algorithms [266], known as
adversarial attacks, which can generate subtle perturbations
imperceptible to a human observer but may greatly mislead
state-of-the-art AI systems to make wrong predictions.
Correspondingly, algorithms that can help resist such attacks
are named adversarial defenses. In this section, we first
provide a brief review of the development of adversarial
attack algorithms. Then, representative adversarial defense
methods are introduced. Finally, related research in the RS
field is summarized.
ADVERSARIAL ATTACKS
Although DL models have achieved remarkable results
in many challenging computer vision tasks, Szegedy et al.
[267] first revealed that these state-of-the-art machine
learning algorithms are vulnerable to adversarial attacks in
image classification tasks. Adversarial examples can be generated
by simply adding subtle perturbations to original input
images. Although these adversarial examples may look
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2022
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
