IEEE - Aerospace and Electronic Systems - July 2021 - 87

Blasch et al.
MULTI-IMAGERY FUSION USING AI/ML
Henry Leung presented the contemporary opportunities
ofAI/ML as a powerful method for big data analytics. For
example, AI/ML methods can detect an anomaly in an
image, such as a moving object in a static scene, through
the use of different sensors. For image fusion, substantial
improvements in image processing using AI/ML methods
have outperformed standard pixel-level and feature-level
statistical histogram processing. The use of different sensors
such as electro-optical and infrared (EO/IR) imagery
affords robustness as detection of features from variations
in the scene.
EXAMPLE: ANOMALY DETECTION
Anomalies and noise in the images appear different
from the different sensors and hence it is highly
unlikely that noise or anomalies in one image are the
same as in the other image. With the labeled data,
learning supports feature extraction and object detection.
One variation is to label and train one modality
and then when deployed, use the known imagery to
help label the second imagery. For example, with video
imagery, 3-D convolution operations are used to form
the feature map (extract appearance/motion from
frames) corresponding to the location in one input converted
to a 1-D set of features. Examples have been
shown for the faster region-based convolutional neural
networks (RCNN) image fusion with convolution features
for anomaly detection (AD) [19]. For example, in
Figure 8, the accuracy of AI/ML with SDF improves
from 82% with optical imagery to 87% with infrared
imagery.
CHALLENGES AND OPPORTUNITIES
One of the challenges with the emergence of DL methods
is that the features have to be handcrafted. Typically,
developers have to label features in all modalities. With
the labeled data, the image fusion system can extract spatiotemporal
features and perform automatic AD. AI/ML
for feature extraction challenges include hand-crafting traditional
features (e.g., edge, gradient) specific to each sensor
(e.g., EO, IR, radar, etc.), tailored motion features
(e.g., optical flow) dependencies of scale and preprocessing;
and modality-specific features (e.g., color features
does not exist in IR).
ML advances have demonstrated other options such as to
increase input resolution, improve signals for better detection,
and enhance detection through superresolution (SR)
[20]. Instead ofsearching for the nearest patch similarity to a
database, ML for SR can learn the mapping ofa low-resolution
patch to a high-resolution patch, and combining to create
JULY 2021
Figure 8.
(a) Multimodal AI/ML data fusion and (b) performance enhancement
through fusion (green line) over no fusion (blue line).
a full image. Recent results include a GenerativeAdversarial
Network (GAN) with a generator for the features and discriminator
as an analysis ofthe simulated features. An example
is for ship detection in synthetic aperture radar in which
there is an SRGAN for resolution and RCNN for object classification
[21]. Other examples of GANs for SDF include
multiple-perspective fusion for radar ATR [22] and multimodal
image fusion [23].
Future issues for the coordination of AI/ML and data
fusion include:
Efficient labeling in multiple modality imagery
sources;
Effectiveness in making machines more autonomous;
Robustness of replacing handcrafted features with
learned features; and
Scaling for natural detection in complex scenes.
MULTI-SOURCE AI/ML FUSION AT THE EDGE
Tien Pham highlights that AI/ML designs require further
testing to move from the laboratory towards a complex
multi-domain environment, such as large-scale, cluttered,
and contested urban environments. In the future, MDO
will require the capacity, capability, and endurance to
IEEE A&E SYSTEMS MAGAZINE
87
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
