IEEE Geoscience and Remote Sensing Magazine - June 2021 - 90

knowledge. Here, the transparency of the model is achieved
by construction because the product is a self-explanatory
causal graph. All three fields (explainability, hybrid modeling,
and causal inference) also have in common the need
of an active and tight interaction among domain experts
and computer scientists to make a decisive, nonincremental
leap in Earth sciences.
With this position article, we present six research directions
that we think hold particular promise for the future of
Earth observation data analysis. In the following, we argue
their potential and relevance and provide some pointers to
pertinent resources. Our goal is to trigger curiosity and foster
successful research to truly advance the field of Earth
sciences with AI.
DIRECTION 1: REASONING AND
HUMAN-MACHINE DIALOGS
Current research on the interface between ML and remote
sensing focuses largely on the direct recognition of materials
and objects or on estimating geophysical parameters.
Reasoning goes beyond the concept of recognition and
aims at mimicking how people think and learn. It is centered
around tasks such as induction, deduction, spatial
and temporal reasoning, and structural inference [24].
To date, only a few pioneering studies have been published
on reasoning for remote sensing tasks. In computer
vision, reasoning is mostly interpreted as the capability to
link meaningful transformations of entities over space or
time. This is a fundamental property of intelligent species
and is also the way people understand visual data. Recently,
papers implementing reasoning in convolutional neural
networks (CNNs) have started to appear. Santoro et al. proposed
a relational reasoning network as a simple plug-andplay
module to solve problems requiring the understanding
of arbitrary relationships between objects (ordering or
comparisons of relative positions/sizes) and applied it to
the problem of visual question answering (VQA) [25] (for
more information on VQA, see direction 3 in Table 1). A
second pioneering work concerns temporal relationships
in video sequences. When it comes to understanding what
takes place between two sampled video frames, humans
can easily infer the temporal relationships and transformations
between observations, unlike neural networks.
In [26], the authors proposed a temporal relationship network,
which learns intuitive and interpretable commonsense
knowledge in videos.
WHY SHOULD RELATIONAL REASONING
MATTER IN REMOTE SENSING?
Earth observation images carry strong spatial and temporal
information because each pixel is precisely referenced and
connected to neighbors in space and time. When considering
land processes (and, in particular, the geophysical ones),
relevant relationships can be learned by using models. In
[6], the authors explicitly modeled long-range relationships
for semantic segmentation in aerial scenes. With the goal
90
of increasing the representation capacity of a fully convolutional
network, two tailored relationship modules were
used: one describing relationships between observations
in convolved images and another producing relationshipaugmented
feature representations. Given that convolutions
operate by blending spatial and cross-channel information
together, they captured relationships in both the spatial and
channel domains.
PERSPECTIVES
The work mentioned in the previous section showcases how
spatial-relational reasoning helps with improving semantic
understanding of remote sensing images, and many other
problems may also benefit from visual reasoning. One exciting
example is temporal reasoning for the analysis of multitemporal
data/aerial videos, e.g., for event recognition. This
is a new and exciting field where one is concerned with understanding
complex events being imaged or filmed, such
as cultural events, manifestations, or locating people in distress.
Using reasoning enables knowing whether a person
on a roof during a flood is in need of actual help or whether
the video of a crowd of people is related to a peaceful or
violent demonstration. This could be of interest to various
stakeholders, including local authorities. To foster this research
direction, Mou et al. [7] have introduced the data set
Event Recognition in Aerial videos (ERA), which consists of
3,000 unmanned aerial vehicle videos manually annotated
into dozens of types of events (Figure 1).
In addition to videos, other interesting examples include
VQA (see direction 3 in Table 1), captioning [27], and
audiovisual reasoning, i.e., linking remote sensing images
to in situ audio signals [28]. In the long run, we hope that
reasoning Earth observation systems will be capable of deducing
clues and making structural inferences to explain
processes (see direction 5 in Table 1) and understand causal
structures in the Earth system (see direction 6 in Table 1).
DIRECTION 2: EXTREMELY MULTIMODAL
REMOTE SENSING
Remote sensing is no longer restricted to observation with
airborne or satellite sensors. Today, we can monitor our
planet's health and status with social media data and socioeconomic
indicators as well as all kinds of imagery, audio,
and text, in addition to satellite imagery [29]. This direction
raises several questions related to the importance of
the different sources and their adequacy for specific tasks.
For this research direction, we discuss some of these aspects
at the crossroads between (sometimes) extremely different
data sources.
THE TRADITIONAL APPROACH
The first step in creating a traditional remote sensing system
is to identify a property of interest y (e.g., land use or snow
depth). We typically restrict the properties to a set of locations
l, often described as a grid of points within a polygon, and
times t. We can formalize this as modeling a conditional
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2021
IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021
