IEEE - Aerospace and Electronic Systems - July 2021 - 45

Credit: Image licensed by Ingram Publishing
organized as follows. Common definitions, conceptualizations,
and related literature in EO/RF fusion are discussed.
Details are provided for the experimental design and
architecture of fusion of EO and RF neural network
(FERNN), and the results of those experiments are discussed.
Finally, we present concluding remarks and discusses
future research directions.
REVIEW OF STATE-OF-ART TECHNOLOGIES
SENSOR FUSION ARCHITECTURE AND TERMINOLOGY
Sensor fusion is a common method for the analysis and
utilization of information and is an essential part of many
applications such as data mining and machine learning.
As there exist many applications and unique datasets, different
approaches have been developed for each of the
corresponding applications. The nature of sensor modalities
and sources of information determine the approach to
process and correlate with different information sources.
Describing and classifying these sensor fusion methods
is therefore important to distinguish the fundamental
principles taken to implement sensor fusion. An important
aspect of classifying fusion between sensors is the nature
of the sensors themselves. The source of data can generally
be categorized as heterogeneous versus homogeneous.
Homogeneous data are typically simpler to fuse together,
due to the data already being in a compatible form. If the
sensor data are heterogeneous, most fusion methods
require some level of preprocessing and a compatible
means of feature extraction.
These considerations can vary depending on the specific
application, the form of the input information, and how the
system interprets the input information. For example, synthetic
aperture radar (SAR) images are a type ofRF modality,
but its fusion with EO images does not necessarily
JULY 2021
require any complicated preprocessing beyond registration
as both modalities are in the same 2-D matrix format.
While there is no common model ofsensor fusion or any
singular comprehensive system of classifying sensor fusion
methods, the majority of existing models propose partitioning
of the information fusion architecture based on how the
source input information undergoes preprocessing, feature
extraction, pattern processing, situation assessment, and
decision making [2]. Classifications can be based on the
communication scheme between sensors prior to fusion,
how the information is processed between classification
algorithms after preprocessing and feature extraction by the
levels where fusion takes place in the fusion architecture.
Level-based classifications generally categorize fusions as
low-level, mid-level, and high-level sensor fusion, normally
corresponding to the terms data-level, feature-level, and
decision-level fusion, respectively [3].
Among the earliest widely adopted sensor fusion classification
criteria is from Dasarathy [4]. Dasarathy divides
data fusion into five categories, data in-data out, data
in-feature out, feature in-feature out, feature in-decision
out, and decision in-decision out. The Dasarathy model
classifies sensor fusion techniques into five categories by
their respective input and output in order to avoid ambiguity
between data selection, feature extraction, feature
fusion, pattern recognition, and decision analysis.
In the context of EO/RF sensor fusion, data input for
EO can vary from still images to video, while data input
for RF can receive signal strength (RSS) values. The EO
features that could be extracted include decluttered
images, corner detection, segmented regions of interest,
etc. For RF features, time of arrival (TOA), time difference
of arrival (TDOA), angle of arrival (AOA), and
Doppler are often calculated. The decisions used as an
input could be detection or motion estimation results
based on individual sensor modality, while the output of
fusion provides low-level object assessment.
IEEE A&E SYSTEMS MAGAZINE
45
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
