IEEE - Aerospace and Electronic Systems - July 2021 - 58
A Survey of Multimodal Sensor Fusion for Passive RF and EO Information Integration
modalities used. Just explaining how some of these sensor
fusion schemes are classified or described is a difficult
task as there currently is no singular all-encompassing
organization or classification ofmethods for this field.
Besides surveying the state-of-the-art literature based
Figure 9.
Comparison of accuracy for different NNs implemented for decision
level fusion.
from feature-level fusion over decision-level fusion. Likewise,
the results indicate the preference ofupstream fusion
to downstream fusion.
Within ensemble learning fusion, soft voting, which uses
the individual outputted probabilities over simple majority
voting; still performed as well if not better. Based on the
approach and results, the association of the RF features with
the corresponding EO frame produced more accurate results
compared to the direct estimations of the output class in
question. From the ensemble learning experiments and the
standalone EO and RF network results, it can also be concluded
that the association of changes in the RF histogram
features was relied on more than the EO, which was less
accurate on its own compared to the ground truth.
When compared to the decision-level fusion models
that were the SVM and LFNN, the results showed that the
ensemble decision fusion schemes significantly underperformed.
The LFNN can achieve 90.7% accuracy with the
same training set that the SVM decision-level fusion
model was able to achieve 88% accuracy with. In comparison,
the soft and hard voting decision-level fusion with
traditional classifiers however, both failed to achieve even
80% accuracy. Compared to all the decision-level fusion
methods tested, FERNN, the proposed feature-level fusion
performed with the highest accuracy, achieving a 95% F1
score versus the LFNN's 90.7% F1 score.
CONCLUSION
Multimodal sensor fusion, especially in the context of EO
and passive RF fusion, is an active research field that is
growing with many different innovative applications and
approaches. The sheer volume and variety of methods
makes it often difficult to pick and choose for a particular
situation or dataset, a problem that is made worse by the
complex sensor sources. It goes without saying that there
is no singular or general solution for determining the optimal
approach for information fusion, as the answer is
always dependent on the situation, the dataset, and
58
on the contributions to multimodal sensor fusion, the focus
was on EO and passive RF fusion. While this research was
primarily focused on the application of deep learning in
information fusion, there exist many suitable classification
schemes for exploiting the advantages of EO and passive
RF modalities. For the purposes of EO and passive RF
fusion, the use of raw data (i.e., upstream fusion) and features
provides more robust and reliable results when compared
to decision or postclassification fusion schemes.
Related literatures have also found similar results
when using passive RF and any form of EO modality,
such as previously mentioned [12] and [43]. For the purposes
of using passive RF in applications that do not use
forms of RF, such as Doppler or SAR imaging, the value
that lower level fusion provides is greater than the metalearning
of higher level (i.e., situation) or decision-level
(i.e., product) fusion for the purposes of fusing these two
modalities. While it is a fundamental issue for any multimodal
sensor fusion application, appropriate synchronization
of different modalities is still a subject of interest.
Determining when and how much data need to be processed
from different modalities in order to optimize correlation
and best help extract relevant features is an issue
that has not been explored exhaustively. The use of spiking
deep belief networks, such as [55] for example, could
have potential unsupervised use in probabilistically reconstructing
the passive RF data to perform classification.
While there are many approaches to machine learning
and classification, some other possible directions using
passive RF in sensor fusion could be addressed by including
machine vision. For EO and passive RF, a few
machine vision approaches have been researched, such as
[56] and [57]. But the question of how to properly integrate
context in the fusion process in order to improve a
classification algorithm's ability to find relevant features
and better discriminate between different classes is important
[58]. Using the machine vision approach for passive
RF creates a need to formalize the concept of context and
also to explore how the changing context could influence
the fusion process, as well as determining what model
would be best suited to handle such a change.
While feature- and raw-level fusion have shown promising
results, the question ofwhat value correlation at the decision
level could have for classification has not been explored
thoroughly for passive RF/EO fusion. While it may be difficult
to apply for the passive RF modality there could be an
intrinsic value in using a dynamic classifier selection
approach, similar to [22] which uses a dynamic settings hidden
Markov model (HMM) classification algorithm for
object detection with passive RFID tags. Even if decisionIEEE
A&E SYSTEMS MAGAZINE
JULY 2021
IEEE - Aerospace and Electronic Systems - July 2021
Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
Contents
IEEE - Aerospace and Electronic Systems - July 2021 - Cover1
IEEE - Aerospace and Electronic Systems - July 2021 - Cover2
IEEE - Aerospace and Electronic Systems - July 2021 - Contents
IEEE - Aerospace and Electronic Systems - July 2021 - 2
IEEE - Aerospace and Electronic Systems - July 2021 - 3
IEEE - Aerospace and Electronic Systems - July 2021 - 4
IEEE - Aerospace and Electronic Systems - July 2021 - 5
IEEE - Aerospace and Electronic Systems - July 2021 - 6
IEEE - Aerospace and Electronic Systems - July 2021 - 7
IEEE - Aerospace and Electronic Systems - July 2021 - 8
IEEE - Aerospace and Electronic Systems - July 2021 - 9
IEEE - Aerospace and Electronic Systems - July 2021 - 10
IEEE - Aerospace and Electronic Systems - July 2021 - 11
IEEE - Aerospace and Electronic Systems - July 2021 - 12
IEEE - Aerospace and Electronic Systems - July 2021 - 13
IEEE - Aerospace and Electronic Systems - July 2021 - 14
IEEE - Aerospace and Electronic Systems - July 2021 - 15
IEEE - Aerospace and Electronic Systems - July 2021 - 16
IEEE - Aerospace and Electronic Systems - July 2021 - 17
IEEE - Aerospace and Electronic Systems - July 2021 - 18
IEEE - Aerospace and Electronic Systems - July 2021 - 19
IEEE - Aerospace and Electronic Systems - July 2021 - 20
IEEE - Aerospace and Electronic Systems - July 2021 - 21
IEEE - Aerospace and Electronic Systems - July 2021 - 22
IEEE - Aerospace and Electronic Systems - July 2021 - 23
IEEE - Aerospace and Electronic Systems - July 2021 - 24
IEEE - Aerospace and Electronic Systems - July 2021 - 25
IEEE - Aerospace and Electronic Systems - July 2021 - 26
IEEE - Aerospace and Electronic Systems - July 2021 - 27
IEEE - Aerospace and Electronic Systems - July 2021 - 28
IEEE - Aerospace and Electronic Systems - July 2021 - 29
IEEE - Aerospace and Electronic Systems - July 2021 - 30
IEEE - Aerospace and Electronic Systems - July 2021 - 31
IEEE - Aerospace and Electronic Systems - July 2021 - 32
IEEE - Aerospace and Electronic Systems - July 2021 - 33
IEEE - Aerospace and Electronic Systems - July 2021 - 34
IEEE - Aerospace and Electronic Systems - July 2021 - 35
IEEE - Aerospace and Electronic Systems - July 2021 - 36
IEEE - Aerospace and Electronic Systems - July 2021 - 37
IEEE - Aerospace and Electronic Systems - July 2021 - 38
IEEE - Aerospace and Electronic Systems - July 2021 - 39
IEEE - Aerospace and Electronic Systems - July 2021 - 40
IEEE - Aerospace and Electronic Systems - July 2021 - 41
IEEE - Aerospace and Electronic Systems - July 2021 - 42
IEEE - Aerospace and Electronic Systems - July 2021 - 43
IEEE - Aerospace and Electronic Systems - July 2021 - 44
IEEE - Aerospace and Electronic Systems - July 2021 - 45
IEEE - Aerospace and Electronic Systems - July 2021 - 46
IEEE - Aerospace and Electronic Systems - July 2021 - 47
IEEE - Aerospace and Electronic Systems - July 2021 - 48
IEEE - Aerospace and Electronic Systems - July 2021 - 49
IEEE - Aerospace and Electronic Systems - July 2021 - 50
IEEE - Aerospace and Electronic Systems - July 2021 - 51
IEEE - Aerospace and Electronic Systems - July 2021 - 52
IEEE - Aerospace and Electronic Systems - July 2021 - 53
IEEE - Aerospace and Electronic Systems - July 2021 - 54
IEEE - Aerospace and Electronic Systems - July 2021 - 55
IEEE - Aerospace and Electronic Systems - July 2021 - 56
IEEE - Aerospace and Electronic Systems - July 2021 - 57
IEEE - Aerospace and Electronic Systems - July 2021 - 58
IEEE - Aerospace and Electronic Systems - July 2021 - 59
IEEE - Aerospace and Electronic Systems - July 2021 - 60
IEEE - Aerospace and Electronic Systems - July 2021 - 61
IEEE - Aerospace and Electronic Systems - July 2021 - 62
IEEE - Aerospace and Electronic Systems - July 2021 - 63
IEEE - Aerospace and Electronic Systems - July 2021 - 64
IEEE - Aerospace and Electronic Systems - July 2021 - 65
IEEE - Aerospace and Electronic Systems - July 2021 - 66
IEEE - Aerospace and Electronic Systems - July 2021 - 67
IEEE - Aerospace and Electronic Systems - July 2021 - 68
IEEE - Aerospace and Electronic Systems - July 2021 - 69
IEEE - Aerospace and Electronic Systems - July 2021 - 70
IEEE - Aerospace and Electronic Systems - July 2021 - 71
IEEE - Aerospace and Electronic Systems - July 2021 - 72
IEEE - Aerospace and Electronic Systems - July 2021 - 73
IEEE - Aerospace and Electronic Systems - July 2021 - 74
IEEE - Aerospace and Electronic Systems - July 2021 - 75
IEEE - Aerospace and Electronic Systems - July 2021 - 76
IEEE - Aerospace and Electronic Systems - July 2021 - 77
IEEE - Aerospace and Electronic Systems - July 2021 - 78
IEEE - Aerospace and Electronic Systems - July 2021 - 79
IEEE - Aerospace and Electronic Systems - July 2021 - 80
IEEE - Aerospace and Electronic Systems - July 2021 - 81
IEEE - Aerospace and Electronic Systems - July 2021 - 82
IEEE - Aerospace and Electronic Systems - July 2021 - 83
IEEE - Aerospace and Electronic Systems - July 2021 - 84
IEEE - Aerospace and Electronic Systems - July 2021 - 85
IEEE - Aerospace and Electronic Systems - July 2021 - 86
IEEE - Aerospace and Electronic Systems - July 2021 - 87
IEEE - Aerospace and Electronic Systems - July 2021 - 88
IEEE - Aerospace and Electronic Systems - July 2021 - 89
IEEE - Aerospace and Electronic Systems - July 2021 - 90
IEEE - Aerospace and Electronic Systems - July 2021 - 91
IEEE - Aerospace and Electronic Systems - July 2021 - 92
IEEE - Aerospace and Electronic Systems - July 2021 - 93
IEEE - Aerospace and Electronic Systems - July 2021 - 94
IEEE - Aerospace and Electronic Systems - July 2021 - 95
IEEE - Aerospace and Electronic Systems - July 2021 - 96
IEEE - Aerospace and Electronic Systems - July 2021 - 97
IEEE - Aerospace and Electronic Systems - July 2021 - 98
IEEE - Aerospace and Electronic Systems - July 2021 - 99
IEEE - Aerospace and Electronic Systems - July 2021 - 100
IEEE - Aerospace and Electronic Systems - July 2021 - 101
IEEE - Aerospace and Electronic Systems - July 2021 - 102
IEEE - Aerospace and Electronic Systems - July 2021 - 103
IEEE - Aerospace and Electronic Systems - July 2021 - 104
IEEE - Aerospace and Electronic Systems - July 2021 - 105
IEEE - Aerospace and Electronic Systems - July 2021 - 106
IEEE - Aerospace and Electronic Systems - July 2021 - 107
IEEE - Aerospace and Electronic Systems - July 2021 - 108
IEEE - Aerospace and Electronic Systems - July 2021 - 109
IEEE - Aerospace and Electronic Systems - July 2021 - 110
IEEE - Aerospace and Electronic Systems - July 2021 - 111
IEEE - Aerospace and Electronic Systems - July 2021 - 112
IEEE - Aerospace and Electronic Systems - July 2021 - 113
IEEE - Aerospace and Electronic Systems - July 2021 - 114
IEEE - Aerospace and Electronic Systems - July 2021 - 115
IEEE - Aerospace and Electronic Systems - July 2021 - 116
IEEE - Aerospace and Electronic Systems - July 2021 - 117
IEEE - Aerospace and Electronic Systems - July 2021 - 118
IEEE - Aerospace and Electronic Systems - July 2021 - 119
IEEE - Aerospace and Electronic Systems - July 2021 - 120
IEEE - Aerospace and Electronic Systems - July 2021 - Cover3
IEEE - Aerospace and Electronic Systems - July 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_july2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_june2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_april2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_february2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_january2023
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_july2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_june2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2022_tutorial
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_april2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_february2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_january2022
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_july2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_june2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_april2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2021_tutorials
https://www.nxtbook.com/nxtbooks/ieee/aerospace_february2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_january2021
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_july2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_june2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_april2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_february2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_january2020
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2019partII
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_july2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_june2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_april2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_may2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_march2019
https://www.nxtbook.com/nxtbooks/ieee/aerospace_december2018
https://www.nxtbook.com/nxtbooks/ieee/aerospace_august2018
https://www.nxtbook.com/nxtbooks/ieee/aerospace_october2018
https://www.nxtbook.com/nxtbooks/ieee/aerospace_september2018
https://www.nxtbook.com/nxtbooks/ieee/aerospace_november2018
https://www.nxtbookmedia.com