IEEE Geoscience and Remote Sensing Magazine - June 2021 - 91

distribution, (| ,),Py lt over property y for a given set of
locations and times. Using the traditional approach, one
would resort to a pixel- or object-based classifier learning
input-output relationships from a labeled set of image pixels.
The goal is to provide an accurate estimate of the uncertainty
over property y and to have this be generalized to new
locations and times.
The traditional approach is limited in several ways:
1) It requires a person able to label the training satellite imagery,
either in the field or through manual image interpretation.
This can be expensive and limiting, especially
when the annotator is asked label difficult, fine-grained
label spaces.
2) It is able to make distinctions only among phenomena
that are easily visible from an overhead perspective. It
cannot, for example, see inside buildings.
3) It is tightly coupled to the geographic region, task, and
source of data. This approach has led to a profusion of
remote sensing papers that use minor variations of the
same computational methods.
MULTIMODAL APPROACHES WITH SOCIAL MEDIA
Social media data can be used to address our fundamental
task, the estimation of (| ,).Py lt We begin by considering
how social media can be incorporated as an input, by using
l and t to query for nearby media content, and then how it
can be used to expand the types of properties, y, that can
be estimated.
SOCIAL MEDIA AS AN ALTERNATIVE-INPUT MODALITY
Traditional remote sensing largely ignores social media and
uses only l and t to index the image; however, today countless
photographic, audio, and textual data from cellphones
are being collected. These data are often associated with
the location and time of capture, making each a potentially
useful source of information about the state of the world
(see Figure 2, where a system based on both remote sensing
and ground-level images uses both modalities in synergy
to provide likelihoods about the presence of objects).
People use these data to make decisions on a daily basis
(e.g., when reading reviews and looking at photographs
while trying to make travel plans). With rapid advances in
the automatic interpretation of imagery, audio, and text, we
can start thinking of these data as potential inputs for remote
sensing. Recent work [30]-[32] has begun to explore
the use of social media imagery, especially for fine-grained
land-use classification. A new methodological framework
of information fusion with ML is actually emerging [33].
These methods tend to use black-box models to extract vector-valued
features from ground images and combine them
with remote sensing features. They also tend to ignore the
rich geometric information the images contain.
SOCIAL MEDIA AS A SOURCE OF SUPERVISION
An untrained person can easily interpret a wide array of
properties from a single social media object. The interpretation
will be fine grained and include subjective properties
FIGURE 1. An overview of the ERA data set, a benchmark for event recognition from aerial videos [7]. For each class, the first (top left) and
last (bottom right) frames of an example video are shown.
JUNE 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
91

IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021