IEEE - Aerospace and Electronic Systems - July 2021 - 48

A Survey of Multimodal Sensor Fusion for Passive RF and EO Information Integration
information from its peers. Decentralized sensor fusion is
accomplished autonomously, occurring at no single point.
In contrast, distributed nodes interchange data at a given
communication rate. The benefits of a decentralized sensor
fusion scheme can be seen in [32] in which the use of
sparse approximation (Joint Sparse Representation) is
capable of achieving the same estimation result as a centralized
algorithm while significantly reducing the communication
cost.
Hierarchical methods are a hybrid of the other techniques
[33], instead of performing sensor fusion at different
levels within the architecture's hierarchy, some using
feedback while others are simply feedforward. The nature
of the system still centralizes its output to some degree,
but with the added benefit of reducing the calculations
needed by designating separate nodes to achieve a form of
decision or feature-in decision-out fusion. There are many
variations on this sort of sensor fusion architecture, Kahler
et al. [34], for example, used a two-step probabilistic cue
integration for the purposes of achieving object tracking
in three dimensions while Song et al. [35] used a hierarchical
architecture to reduce the computational complexity
of a decentralized data fusion algorithm by being
placed between the clusters and fusion center.
Another notable sensor fusion categorization discriminates
between upstream and downstream data fusion.
Upstream data fusion is the processing, exploitation, and
fusion ofsensor data as closely to the raw sensor data feed as
possible. In contrast to downstream (post-decision) fusion,
upstream data fusion processes the input information and
minimizes the data loss that can result from conventional
data reduction methods. Upstream fusion improves the performance
by accessing the data at an appropriate point in the
processing chain near the data source, strategically chosen
to acquire the desired data within the earliest point in the
fusion architecture [36]. Upstream fusion was used by Garagic
et al. [12] to integrate FMV and passive RF data, using
multimodal emitter tracking and localization architecture.
The algorithm combines deep learning and feature manifold
representations to achieve upstream sensor fusion.
METHODOLOGIES IN EO AND PASSIVE RF SENSOR
FUSION
When dealing with modalities that involve radio frequency
(RF) and electro-optical (EO) sensors, the focus
for fusion has traditionally been on active RF sensors.
Doppler radar and imaging radar (e.g., side-looking airborne
radar), as well as other similar active RF sensors,
are well suited for tracking a moving target when used in
concert with a form of EO modality. However, the combined
exploitation of the two sensor modalities can still be
improved [37]. That being said, RF modalities excel in
providing range, angular, and spectral resolution of
48
information from RF modalities and the benefits of combining
RF data with higher spatial resolution of EO-based
sensors is extremely desirable for detection and tracking.
There are a number ofRF-basedmodalities that are used
in applications such as tracking, proximity, localization, and
detection. While many EO modalities are intuitively easier
for humans to understand and to implement for similar applications,
unlike RF modalities; RF approaches to such problems
are less susceptible to problems such as ocular
interference. RF-based sensors are not limited by factors like
visual interference from natural phenomenon such as fog,
clouds, snow, or any other form ofweather that would otherwise
interfere in the collection of EO data. In addition, RFbased
sensors can provide repetitive coverage over a wide
geographical area, and in doing so, can determine the precise
distance and velocity ofa target. Asmentioned earlier in this
section, many RF modalities in detection and tracking applications
utilize active RF sensing.
One example ofactive RF/EO fusion was given by Seo
[38], where SAR data are used in combination with multispectral
(MS) images using random forest regression. This
approach fuses together the SAR image containing the surface
roughness characteristics while retaining the spectral
characteristics of the MS images, before being fed into a
modified random forest regression algorithm, outperforming
the comparison algorithms with the same KOMPSAT5
and Landsat-8 OLI datasets. The use ofSAR imaging to
facilitate fusion between RF and EO sensor data is a commonly
used approach for automatic ground target recognition.
While Orynbaikyzy et al. [39] used a more traditional
algorithm, Kim et al. [40] utilized a double weighted NN
fusion scheme that uses sum-based linear fusion to generate
features and a NN- based fusion at the decision level.
Another example of active EO/RF fusion is Bui et al.
[41] in which SAR and multisource satellite imagery were
fused together at the data, feature, and decision levels. As
mentioned earlier [41], 2-D radar, EO, and IR sensor data
were used with an extended Kalman Filter and State Vector
Fusion to track a target in 3-D Cartesian coordinates in
a Monte Carlo simulation. In Zhang et al. [13], the use of
Rao-Blackwellized particle filtering was implemented in
order to fuse the asymmetrical fields of view for the radar
and EO modalities. The application of adaptive waveform
design and control incorporated dynamic agility selection
in order to improve the performance of the system's ability
to track an unknown number of targets using the two
modalities. Other notable algorithms include the use of
sparse representation in order to combine medium wavelength
IR (MWIR) cameras and RF Doppler sensors for
vehicle tracking [42], which uses a joint sparse approximation
approach for multimodality images.
There are a number of advantages for the implementation
of passive RF modalities such as passive radar or
RFID. Passive RF modalities are difficult to detect, require
lower power and have lower costs than the ones associated
IEEE A&E SYSTEMS MAGAZINE
JULY 2021
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
