IEEE Geoscience and Remote Sensing Magazine - June 2021 - 93

and automatic geolocalization, using these additional data
could unlock new applications, such as modeling soundscapes
[45], landscape scenicness [46], or place-perception
analysis [47]-[49]. Further, one could study phenomena
closer to the consumer providing the data; for example, one
could think of a system finding the most visually appealing
driving route.
This will inevitably raise the question of data set biases
because social media data are personalized views of space:
photographers tend to take pictures from places that are
easy to reach; they are biased in the types of subjects they
prefer and usually take more pictures in the daytime and
in good weather conditions (see, for instance, the oversampling
of particular photographic forms and scenes in
the Instagram account insta_repeat (https://www.instagram
.com/insta\_repeat/). This problem was recently considered
when observations collected by citizen scientists were
used for species distribution mapping [50]. In general,
biases in learning models is a growing topic of study in
both the ML (see, for example, [51] and [52]) and social
media (see the reviews in [53] and [54]) communities.
There is ample room for such studies in remote sensing
and biases issued by fusion in multimodal settings
or in hallucinations when using generative adversarial
networks.
Using social media also implies the development
of models that are robust to differences in the appearance
of classes, which becomes critical when predicting
in new geographies or in time moments. As acquiring
new labeled data is not always an option, one could
envision using these alternative sources as a form of
weekly supervised training data or even as an unsupervised
supervisory signal for knowledge discovery,
as Law and Neira have proposed [55] to explore the urban
latent space of London's streetscapes. Considering
that the data acquired by autonomous vehicles and Internet
of Things devices will push this need even further,
it
will also unlock the potential of mapping on
demand with extremely multimodal remote sensing.
Finally, further integration could make it possible to
use social media as an early detection system for events,
such as natural disasters [56]. For instance, social media
imagery could be used to detect damaged structures
or people in distress [57]. Such a system could even be
used to cue satellite image acquisition over areas of interest
based on image content, location-data densities,
or tweets.
DIRECTION 3: INTERACTIVE AND SEMANTIC ML
With their massive increase in availability, remote sensing
images are now used beyond scientific research. First, images
are available worldwide and with a high update rate.
But they are also much more accepted by the general public:
no one is surprised anymore when they are shown a satellite
view from Google Maps; consumer-level drones can
be used by virtually anyone for all kinds of tasks, such as
JUNE 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
farmers monitoring crops, ecologists surveying animals, or
architects keeping track of construction sites.
But, despite the massive potential for image acquisition
and updating, the use of images remains static in the sense
that images are mostly used for visualization or, at best, to
compute standard indices such as the normalized difference
vegetation index, which is then assumed to represent
vegetation status. Moreover, models answering the specific
needs of users are scarce and more often limited to classical
processing tasks (e.g., car detection or land cover mapping)
and cannot cover the variety of tasks in which different users
could be interested. Another limitation is that end users
rarely have the technical skills necessary to design and run
ML models and would like to receive an answer to a specific
question of interest asked in a natural language (e.g.,
in English).
Fortunately, many of these questions boil down to the
presence of objects, to counting, or to some kind of relational
attribute (e.g., whether there was an increase of
forest area or whether there are buildings in risk zones):
a model capable of pursuing some kind of reasoning
about the image content (see direction 1 in Table 1) but
taking into account a specific question (in English) by
a user could open the door to a new type of interaction
with remote sensing. Similar to what search engines do
on the Internet, a remote sensing VQA (RSVQA) [12] engine
could allow anyone, from scientists to laymen and
journalists, to retrieve relevant information contained in
the images.
Research into VQA is a vivid topic in computer vision
[11], where it has had considerable impact on creating systems
that support vision-impaired people with everyday
tasks [58]. A traditional VQA system, in this context, can indeed
be used to help people when buying groceries, crossing
the street, and so on.
DIALOGS AMONG USERS AND EARTH OBSERVATION
IMAGES REQUIRE BOTH REMOTE SENSING AND
NATURAL LANGUAGE PROCESSING
In remote sensing, the first VQA system was proposed by
Lobry et al. [12] and is summarized in Figure 3. To become
truly general purpose, such a model needs to be trained using
a large quantity of data from several areas and different
thematic objectives: in [12], two models were designed, one
for Sentinel-2 data and another for subdecimeter-resolution
aerial images. The models were trained with large sets of
image/answer pairs spanning tasks of classification, relative
position reasoning, and object counting. As a large quantity
of labels was necessary, OpenStreetMap (OSM) vector data
were used to automatically generate labels: following the
Compositional Language and Elementary Visual Reasoning
(CLEVR) protocol [59], 100 questions per image involving
objects occurring in the image (as informed by OSM)
were generated. For each image/question pair, the answer
(i.e., the label) was automatically obtained by querying
OSM directly. The data and models are openly available
93
https://www.instagram.com/insta/_repeat/ https://www.instagram.com/insta/_repeat/
IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021
