IEEE Aerospace and Electronic Systems Magazine - December 2020 - 12

Augmented Reality for Immersive and Tactile Flight Simulation

Figure 11.
Detection of ArUco tags and overlay with simulated VR gauges.
Moving counter-clockwise from the bottom left, ArUco tag identifiers 0, 1, and 2 are overlaid with the tachometer, altimeter, and
air speed indicator.

ing, they disappear when moving the head quickly due to
camera motion blur. Also, when the head is moved slowly,
there is approximately one frame (1/60 s = 17 ms) of lag,
causing the gauges to be slightly misaligned with the tags.

GEOMETRIC VERSUS GREEN-SCREEN BASED OVERLAY
As described in section " AR Flight Simulator Concept, "
AR flight simulator imagery can be accomplished using a
geometric model or green-screen detection. This section
shows HMD images from the AR simulator for these two
techniques.

Figure 13.
AR simulator view when taking a quick glance to ensure the right
hand is on the throttle control for geometry-based (top) and
green-screen (bottom) overlay. Left and right panes show images
sent to the left and right eyes with the HMD.

Figure 12 illustrates the basic forward view of the simulator for geometry-based (top) and green-screen (bottom)
overlay. The example illustrates how geometric overlay
requires a more complete physical cockpit than greenscreen detection for good aesthetics. Also, whereas geometric overlay " chops off " the user's hand when placed
over scenery display areas, green-screen detection allows
the hand to show over correctly.
Figure 13 illustrates using the throttle with the right
hand for the two overlay methods. In each case, a quick
glance is all that is required to verify the right hand is
touching the correct control, after which, the user can look
up and actuate the throttle by feel alone.

AR PROCESS TIMING

Figure 12.
Forward simulator view for geometry-based (top) and green-screen
(bottom) overlay. Left and right panes show images sent to the left
and right eyes with the HMD. In geometry-based overlay, the hand
disappears behind window scenery, whereas green-screen overlay
produces the correct result.

In the AR Process code, time stamps are stored at key
locations to allow for detailed profiling of the code.
Table 2 shows CPU time required for each thread for a
continuous 1-h and 10-min run of the AR simulator with
green-screen detection enabled. Note that the CPU has
eight cores, allowing all threads to run in parallel. The column " Time/Frame " indicates how much time on average
is used on a CPU core to process one frame. The column
" Max FPS " is the reciprocal of the first column, quantifying the maximum FPS that could be supported by that
thread. Finally, " FPS " gives the average frames per second that were actually processed by the thread, which is
close to the theoretical 60 FPS supported by the stereo
camera input.

IEEE A&E SYSTEMS MAGAZINE

DECEMBER 2020

IEEE Aerospace and Electronic Systems Magazine - December 2020

Table of Contents for the Digital Edition of IEEE Aerospace and Electronic Systems Magazine - December 2020