IEEE Geoscience and Remote Sensing Magazine - June 2023 - 47

vectors, and then extracting the canonical correlative feature
as a fused feature for the joint dimensionality reduction
of a high-dimensional feature space, which is a statistical
theory for the correlation analysis of two random
vectors. This kind of algorithm includes neural network
CCA [155], kernel CCA [156], local preserving CCA [157],
sparse CCA [158], discriminant CCA [159], and 2D-CCA
[160]. The partial least squares (PLS) integrates the advantages
of multiple linear regression, principal component
analysis, and CCA, and have been applied for feature fusion
[161]. Next, researchers have proposed a series of improved
methods [162], [163], such as conjugate orthogonal PLS
analysis, kernel PLS analysis, 2D-PLS, and sparse PLS analysis.
Another type of feature-level fusion method obtains a
fused feature by projecting multiple original feature spaces
onto the same space with common attributes. For example,
Long et al. [164] proposed a feature fusion algorithm based
on multiview spectral clustering.
Previous feature fusion methods were based on handcrafted
features. Feature learning based on deep learning
methods can essentially be considered as stage-by-stage
multiple feature fusion, such as convolution and full connection
operations of a convolutional neural network
(CNN) [165]. The different layer outputs of the deep learning
network correspond to different visual features and
semantics; for example, the lower layer corresponds to
brightness, edge, and texture; the middle layer corresponds
to shape and direction; and the upper layer corresponds
to category. Feature fusion can then be realized in the different
layers. Currently, most deep learning algorithms
are designed for single-modality data. The coupled correlation
can be mined, and the redundancy can be reduced
as much as possible for different dimensional features by
combining deep learning and information fusion. Therefore,
researchers have gradually focused on information
fusion-based deep learning. For example, some CNN-based
feature fusion structures and strategies have been proposed
[166], [167], [168], [169], [170], [171], [172], [173], [174],
[175], [176], [177], [178], [179]. Some methods have been
proposed for fusing deep-learning features based on CCA,
topic models, joint dictionaries, and bag-of-words [169].
The key step in feature fusion based on deep learning is the
selection of a feature fusion layer and architecture.
According to the fusion layer in the deep learning network,
feature fusion methods based on deep learning can
be classified into early-, middle-, and late-stage fusion.
Early- and middle-stage fusion use the convolution layer,
and late-stage fusion uses the output of the convolution
layer or the output of the full connection layer. Another
method for feature fusion based on deep learning is multimodality
deep learning, which first learns the features of
single-modality data individually and then learns the fused
feature. Ngiam et al. [170] proposed a cross-modality deep
autoencoder model that can better learn shared representations
between modalities. Srivastava and Salakhutdinov
[171] proposed a multimodal deep Boltzmann machine
JUNE 2023 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
for multimodality learning. The model works by learning
the probability density over the space of multimodal data,
and it uses the states of the latent variables as representations
of different modality data. Multimodality data can be
organized and represented with a tensor; therefore, tensorbased
deep learning methods can be used for multimodality
deep learning and can learn deep correlations between
data of different modalities. Yu et al. [172] proposed a deep
tensor neural network (DTNN) model. The DTNN extends
the deep neural network (DNN) by replacing one or more
of its layers with a double-projection layer, in which each
input vector is projected into two nonlinear subspaces and
a tensor layer, in which two
subspace projections interact
and jointly predict the next
layer in the deep architecture.
Hutchinson et al. [173]
proposed a tensor-deep stacking
network, which consists
of multiple stacked blocks,
where each block contains a
bilinear mapping from two
hidden layers to the output
layer using a weight tensor
to incorporate higher-order
statistics of the hidden binary
features. Zhang et al. [174] proposed a deep computation
model to fully learn the data distribution, which uses a
tensor to model the complex correlations of heterogeneous
data. The model uses a tensor autoencoder as the basic
module of the tensor deep learning model, which adopts
tensor distance as the average sum-of-squares error term of
the reconstruction error in the output layer. A high-order
back-propagation algorithm is designed for training.
Information fusion methods for space-based observation
data are primarily designed for multisource remote
sensing image fusion and target recognition. Target-recognition
algorithms based on information fusion use fuzzy
mathematics and the evidential reasoning theory for decision-level
fusion.
PROBLEMS AND CHALLENGES
The onboard data-processing capability of Earth observation
satellites remains weak, and the main existing problems
are as follows:
◗ Most remote sensing satellites are capable of onboard
data compression and preprocessing. Some have realized
onboard detection, classification, and recognition
of targets of interest, such as ships and airplanes. Currently,
remote sensing satellites carry only one type of
sensor and either work independently or unite with
each other. Intersatellite high-speed communication
and data interchange are still in the testing stage. Observation
planning mainly involves scheduling on the
ground, and onboard autonomous intelligent planning
for multisatellite cooperation is not yet practical.
47
INFORMATION FUSION
PRIMARILY INVOLVES
MULTISENSOR DETECTION,
TRACKING AND STATE
ESTIMATION, TARGET
RECOGNITION, SITUATION
AWARENESS, AND THREAT
ASSESSMENT.
IEEE Geoscience and Remote Sensing Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2023
