IEEE Geoscience and Remote Sensing Magazine - June 2021 - 97

and its components to understand, for example, why
the model came to a certain decision. Therefore, explanations
become application dependent, and identical
interpretations can lead to different explanations when
linked to different domain knowledge.
HOW EXPLAINABLE AI MAY HELP REMOTE SENSING
Recently, many tools have been proposed for increasing
interpretability and explainability when combined with
domain knowledge [18]. Two major groups have emerged.
1) Posthoc interpretability: In this group, the outcomes and
decisions of the model are interpreted and explained
by looking at the input. The most common visualizations
for interpretations are heatmaps and prototypes.
Heatmaps highlight the parts of the input data that are
prominent, important, or occlusion sensitive; for example,
they are created using the gradient flows in the
neural network. Prototypes are optimized input data
that, given a model, maximize the targeted output. Both
of these approaches help to explain what a model bases
its decisions on, what influences the output, or what is a
typical input for the learned input-output relationships.
In all cases, attention must be paid to the confirmation
bias, which is defined as the tendency to try to explain
interpretations that are consistent with our existing
knowledge even if the explanation does not apply to the
given case (see [74] for an example of the overinterpretation
of saliency maps).
2) Interpretability by design: In this case, the model is inherently
designed so that it can be interpreted. Interpretability
is achieved by representing model components
or obtained latent variables such that they can be explained
with knowledge from a certain application
domain. For instance, the units in hidden layers can
be designed in such a way that the underlying factors
of variation, such as the driving forces in Earth system
data, become disentangled and are captured in separate
units. This could be seen, for example, by the fact that
simple correlations exist between variations of the input
and the activation of the neurons. Interesting applications
of this idea are proposed in [75], where the authors
disentangle the physical forces applied between objects
in videos, or in [76], for explaining human perceptions
of beauty in landscapes.
To ensure the scientific value of the output, interpretation
tools can be used to check its reliability. Besides the
inherently existing output score of the neural network,
for example, visualizations of the processes within the
Explainability
Scientific
Discoveries
and Insights
Interpretability
Domain Knowledge
(Reference Labels, Expert
Knowledge, and so on)
Transparency
ML Model
Earth Data
Model Output
FIGURE 7. Explainable ML can be used to gain scientific discoveries and insights by explaining a learned model and/or results (shown in
the light gray box). The prerequisites are interpretability and potential transparency, which lead to scientific explanations when combined
with domain knowledge. A feedback loop allows for extending and improving the known domain knowledge. One potential application
is the derivation of improved definitions, for example, for certain land-use classes, which are currently only vaguely, incompletely, or not
uniformly described.
JUNE 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
97

IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021