IEEE Geoscience and Remote Sensing Magazine - September 2022 - 224

Multiattention
Autoencoder Network
Encoder
Decoder
HRMS
Image
LRHS
Image
Loss Function
MSFG
Convolution
Layer + BN
+ LReLU
Convolution Layer
(S = 2) + BN +
LReLU
Upsample
+ Convolution
Layer + BN +
LReLU
Upsampling
Guide
Block
Nonlocal
Block
Spatial Cross
Attention
Block
Laplacian
Guide
Block
Random
Code
FIGURE 21. An outline of UMAG-Net comprising an encoder and a decoder with spatial cross attention mechanism. (Source: [143];
used with permission.) BN: batch normalization; LG: Laplacian guide; NL: nonlocal block; S: stride; SCA: spatial cross attention;
UG: upsampling guide.
an overview of all STF methods. Note that we refer to
the HSLT images on time t as Ft
and the LSHT images as
C ,t respectively.
CONTEXT- AND TARGET-ASSISTED METHODS
Several researchers argue that the spatial resolution gap
between certain sensors, such as those carried by MODIS
and Landsat, is quite large and that data coming from both
sources undergo different atmospheric and geometric corrections.
Therefore, they design models that produce intermediate
images enhanced by a smaller scaling factor to
facilitate the downscaling process. For example, Song et al.
[154] (STFDCNN) (Figure 23) propose a two-stage model
that takes as the input an arbitrary pair of Landsat-5/7
(25-m) and MODIS (500-m) images and learns to predict
an intermediate enhanced image of 250-m spatial resolution.
The intermediate image is computed in a preupsampling
fashion, while the final 25-m image is computed via
a postupsampling SRCNN structure. During the inference,
features are extracted from MODIS images at times ,,
(where t2
tt
and t3
12
is the prediction date), which are linearly
combined with the corresponding Landsat images on t1
and t3
to produce the final HR result. Building on this,
Zheng et al. [158] (VDCNSTF) propose deeper network architectures
and redesign the SRCNN stage as a multiscale
model producing images at 125 m and 25 m.
A slightly different approach is followed by Liu et al. [147]
(StfNet), who argue that the temporal changes expressed by
a time difference image are highly correlated with the contents
of the original images. Therefore, they design a model
224
that takes as the input an LSHT MODIS image (250-300 m)
at prediction date ,t2
a date before ()t1
and a date after ()t3
the prediction date, and a corresponding HSLT Landsat image
at dates t1
and ;t3 produces time difference images; and
then reconstructs the HR image on date t2 by transferring
information from these temporal relations.
More specifically, they propose two CNNs that take as
the input a concatenation of the MODIS time difference
image and the Landsat image and produce a time difference
Landsat. They employ these networks to learn the
following mappings: 1) (, )CF F
and 2) (, )CF F " and (, ).CF F " Mapping 1 can be
supervised by the label F ,13 which is available in the train12
112
13 113 " and (, )CF F
23 323
13 313
"
ing data, forming the time difference reconstruction term
of the loss function. The results of mapping 2 are summed
FF FF
CC CC C
tprev
ttarget
(a)
tnext
tprev
ttarget
(b)
tnext
FIGURE 22. The data used for (a) CT-A and (b) C-A STF during
training. F refers to the HSLT image, C refers to the LSHT image, and
tprev and tnext are one or multiple dates before and after the target
date ttarget, respectively.
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
HRMS Image
(3,3), S = 2
(3,3)
(3,3)
Radom
Code
(1,1)
LG Block
NL Block
LG Block
UG Block
(3,3)
(3,3)
SCA Block
(3,3)
(3,3)
HRMS Image
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
