IEEE - Aerospace and Electronic Systems - July 2021 - 58

A Survey of Multimodal Sensor Fusion for Passive RF and EO Information Integration
modalities used. Just explaining how some of these sensor
fusion schemes are classified or described is a difficult
task as there currently is no singular all-encompassing
organization or classification ofmethods for this field.
Besides surveying the state-of-the-art literature based
Figure 9.
Comparison of accuracy for different NNs implemented for decision
level fusion.
from feature-level fusion over decision-level fusion. Likewise,
the results indicate the preference ofupstream fusion
to downstream fusion.
Within ensemble learning fusion, soft voting, which uses
the individual outputted probabilities over simple majority
voting; still performed as well if not better. Based on the
approach and results, the association of the RF features with
the corresponding EO frame produced more accurate results
compared to the direct estimations of the output class in
question. From the ensemble learning experiments and the
standalone EO and RF network results, it can also be concluded
that the association of changes in the RF histogram
features was relied on more than the EO, which was less
accurate on its own compared to the ground truth.
When compared to the decision-level fusion models
that were the SVM and LFNN, the results showed that the
ensemble decision fusion schemes significantly underperformed.
The LFNN can achieve 90.7% accuracy with the
same training set that the SVM decision-level fusion
model was able to achieve 88% accuracy with. In comparison,
the soft and hard voting decision-level fusion with
traditional classifiers however, both failed to achieve even
80% accuracy. Compared to all the decision-level fusion
methods tested, FERNN, the proposed feature-level fusion
performed with the highest accuracy, achieving a 95% F1
score versus the LFNN's 90.7% F1 score.
CONCLUSION
Multimodal sensor fusion, especially in the context of EO
and passive RF fusion, is an active research field that is
growing with many different innovative applications and
approaches. The sheer volume and variety of methods
makes it often difficult to pick and choose for a particular
situation or dataset, a problem that is made worse by the
complex sensor sources. It goes without saying that there
is no singular or general solution for determining the optimal
approach for information fusion, as the answer is
always dependent on the situation, the dataset, and
58
on the contributions to multimodal sensor fusion, the focus
was on EO and passive RF fusion. While this research was
primarily focused on the application of deep learning in
information fusion, there exist many suitable classification
schemes for exploiting the advantages of EO and passive
RF modalities. For the purposes of EO and passive RF
fusion, the use of raw data (i.e., upstream fusion) and features
provides more robust and reliable results when compared
to decision or postclassification fusion schemes.
Related literatures have also found similar results
when using passive RF and any form of EO modality,
such as previously mentioned [12] and [43]. For the purposes
of using passive RF in applications that do not use
forms of RF, such as Doppler or SAR imaging, the value
that lower level fusion provides is greater than the metalearning
of higher level (i.e., situation) or decision-level
(i.e., product) fusion for the purposes of fusing these two
modalities. While it is a fundamental issue for any multimodal
sensor fusion application, appropriate synchronization
of different modalities is still a subject of interest.
Determining when and how much data need to be processed
from different modalities in order to optimize correlation
and best help extract relevant features is an issue
that has not been explored exhaustively. The use of spiking
deep belief networks, such as [55] for example, could
have potential unsupervised use in probabilistically reconstructing
the passive RF data to perform classification.
While there are many approaches to machine learning
and classification, some other possible directions using
passive RF in sensor fusion could be addressed by including
machine vision. For EO and passive RF, a few
machine vision approaches have been researched, such as
[56] and [57]. But the question of how to properly integrate
context in the fusion process in order to improve a
classification algorithm's ability to find relevant features
and better discriminate between different classes is important
[58]. Using the machine vision approach for passive
RF creates a need to formalize the concept of context and
also to explore how the changing context could influence
the fusion process, as well as determining what model
would be best suited to handle such a change.
While feature- and raw-level fusion have shown promising
results, the question ofwhat value correlation at the decision
level could have for classification has not been explored
thoroughly for passive RF/EO fusion. While it may be difficult
to apply for the passive RF modality there could be an
intrinsic value in using a dynamic classifier selection
approach, similar to [22] which uses a dynamic settings hidden
Markov model (HMM) classification algorithm for
object detection with passive RFID tags. Even if decisionIEEE
A&E SYSTEMS MAGAZINE
JULY 2021
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
