IEEE Geoscience and Remote Sensing Magazine - December 2022 - 159

Putting extracted information into the spatial domain
is done mostly by aggregating it into raster-based formats.
This eliminates smaller errors from GPS sensors and sets
the information into a larger spatial context. On the other
hand, this approach might lead to vanishing information if
signals from different images are highly diverse.
Relating objects in images to spatial objects requires
more than a geotag. If a geotag and a compass direction
are present, a line of sight can be calculated and related to
the spatial objects that it intersects [85]. More sophisticated
approaches can make use of the camera's angle of view to
define the area depicted in an image. However, as the standard
for metadata does not take into account information
about roll and pitch, such algorithms have to assume that
the camera's position was nearly orthogonal to the ground.
METHODS
Georeferenced social media images have been used for land
cover and land use classification on various spatial levels,
from the parcel down to the building instance level [83],
[86], [87], [88], [89], [91], [92], [93]. Additionally, they can
be used to automatically identify landmarks and POIs due
to spatial clusters resulting from multiple users taking photos
in one area with the same motif [94]. Zhu et al. [93] use
Flickr images to predict one of 45 predefined land use classes
on a parcel level. They train their model on global Google
and Flickr weakly labeled images, testing with Flickr images
in San Francisco. They jointly fine-tune two CNN networks.
The first network, called the object stream, is an ResNet101
architecture pretrained on ImageNet, in which the last layer
is replaced by a 45-class classification head. The second network,
called the scene stream, is also an ResNet101 architecture
pretrained on the Places365 dataset, in which the
last layer is also replaced by a 45-class classification head.
The final classification decision is a late-average fusion of
the outputs of the two subnetworks. Hoffmann et al. [83]
address the same problem as Zhu et al. [93] but at the more
fine-grained level of the individual building. Hoffmann
et al. [83] use a classification scheme with five classes and
train and test their model using Flickr images in Los Angeles.
They simply assign each geotagged Flickr image to the
closest OSM building in Los Angeles, assigning the label
of the building to the image. Then, to classify an image,
they first transform the image to a vector of features using
VGG-16, a state-of-the-art vision neural network pretrained on
ImageNet. Next, they pass feature vectors to a five-classes
logistic regression classifier. Finally, the building-level predicted
class is the majority vote of the predicted classes of
the images that are assigned to that building.
Social media images can provide critical hints during
natural disasters. There are two different datasets supporting
this application: one focusing on image retrieval, i.e.,
finding all images related to a disaster in the stream of all
incoming data [95], and the other aiming at image classification
so that the correct situation, like flooding or fire, can
be reported to emergency teams [96]. In the case of flooding,
studies have shown that social media images can be
used to estimate water level [97], [98]. For example, Chaudhary
et al. [98] use multitask learning on a dataset that is
manually compiled from Flickr, Google, and National Geographic.
As accurate water levels in meters are cumbersome
to collect, their model learns to jointly predict water levels
as a regression task and relative levels as a ranking task. The
assessment of water levels is based on five common objects
in urban environments: persons, cars, buses, bicycles, and
buildings/houses, which allows the ground truth to be estimated
by relative comparison. However, this method does
not use any geotags for spatial interpretation.
Another application is environmental monitoring, either
by mapping animal and plant species [99], or by assessing
landscape preferences [100] and aesthetics [81],
[101]. Urban areas can especially benefit from this data
Operations
Data
Acquisition
Data Storage
Information
Extraction in
Feature
Domain
ML Driven
Get Restricted
Access
Accept
Uncontrollable Data
Quality
Manage Images
and Metadata at
the Same Time
Enable Efficient
Random Data
Access
Analyze Image
Content to Extract
Information
Handle Sparse and
Uneven Spatial Data
Distribution
Estimate Spatial
Influence of
Features
FIGURE 5. The data processing chain for extracting geospatial knowledge from geotagged social media images.
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
159
Deal With Unclear
Target Variables
Tackle Class
Imbalance
Mapping to
Targets in
Spatial
Domain
Evaluation

IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022