IEEE - Aerospace and Electronic Systems - July 2021 - 53

Vakil et al.
Figure 3.
The extracted EO frame and the corresponding P-RF histogram
(Simulation 9, NADIR RF reciever).
specific state. The first state, i.e., s ¼½ 10 00,is
defined as having no cars in the simulation, which is
included for the purposes of determining the accuracy of
the EO modality as the ground truth versus what the EO
modality can detect. Within the DIRSIG simulations, the
ground truth is that there is always at least one vehicle in
all the simulations. The other three states are for the presence
of a single car, two cars, and three cars, respectively,
within the simulation.
For data from EO sensors, each simulated video input
was first processed into image frames, resized, and converted
into grayscale for simulation video that was not
originally grayscale. For the purposes of reducing training
time and conserving processing power, the image frames
were resized to match the P-RF histograms.
Prior to being fed into FERNN, the raw P-RF data
are processed to generate histograms of the I/Q data.
The histogram depicts the estimation of the probability
distribution of the P-RF data. The histograms are then
fed into the NN in conjunction with the corresponding
EO frames. Figure 3 shows an extracted EO frame and
the corresponding histogram of P-RF sensor that have
been aligned in time. In order to achieve heterogeneous
feature-level fusion, a deep neural network is trained
over the pairs of 2-D matrices from the two modalities.
The fusion NN itself is a sequential model that compares
the predicted states with the labels and then modifies
its weights accordingly, as shown in Figure 6 and
described in part C. For the purpose of decision-level
fusion, the NNs are trained for standalone modalities,
i.e., an EO NN and a P-RF NN.
EO NEURAL NETWORK
A separate CNN is trained for the detection and estimation
of the number of vehicles based on EO data. This imagebased
NN can achieve 91% accuracy upon using the modified
labels that are exclusively just for an image classification.
For these modified labels, if a frame is generated for
a period in which the vehicle is temporarily obscured by
foliage, the frame is labeled as having no vehicle detected,
which is inaccurate when compared to the ground truth of
the simulation. Likewise, simulation frames in which the
JULY 2021
Figure 4.
Comparison of I/Q histograms collected by SIGINT in simulation
1 (left) and simulation 9 (right).
vehicle has not entered the scope of the EO sensor are also
labeled as not detecting a vehicle.
In order to accomplish classification, both FERNN and
the standalone EO network begin preprocessing, resizing,
and labeling the frames. After that the networks categorize
the images by the number of vehicles detected. Unlike the
ground truth for the overall scenario simulation, the standalone
EO NN will output that no vehicle is detected when
vehicle is not visible to the EO sensor. For the purposes of
comparison testing, the standalone EO NN retains the
original training it received to classify an image by the
number of vehicles it detects. When being tested against
the ground truth for each simulation, in terms of the number
of vehicle(s) traveling in the area, the accuracy of the
EO network decreased to an accuracy of only 72%. This
result is expected, as the simulation has a number of
optically obscured examples inside of the simulation set.
RF FEATURE EXTRACTION AND NN
The most basic signal that can be collected for the RF
sensing is known as in-phase and quadrature components.
These I/Q components are the basis of complex RF signal
modulation and demodulation, and the backbone of modern
communication systems. During previous experiments,
our group had successfully trained a CNN to detect
the human occupancy of an enclosed indoor space using
the raw I/Q data of passive RF signals. For the DIRSIG
dataset, however; the I/Q data in its raw format were ineffective
for the purposes of vehicle detection. The I/Q data
were processed to generate a 2-D histogram, which is an
estimation of the probability density function of the P-RF
data. In the DIRSIG simulation, there are three SIGINT
sensors to generate the P-RF data. These sensors are
placed orthogonally, one in north, and one in west, and
one in the nadir. The generated 2-D histograms are then
fed into the fusion networks in order to facilitate the
homogenous fusion between the three P-RF sources.
As seen in Figures 4 and 5, the 2-D histograms show
visually different patterns due to the different waveforms
transmitted by vehicles. To illustrate the differences in
histograms, the histogram value was plotted in the
z-dimension to provide a clear visual difference. The histogram
of P-RF in Simulation 1 (see Figure 4, left, single
IEEE A&E SYSTEMS MAGAZINE
53
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
