IEEE Robotics & Automation Magazine - June 2020 - 114

the obstacle. Using these safe-/unsafe-labeled initial-final
position pairs, a conservative NN was trained to make safety decisions about untrained initial-final position pairs at
runtime. Figure 12 shows the safe and unsafe initial-final
position pairs and corresponding NN decisions, using the
same color coding as the previous cases.
The safe navigation approach was validated in an environment with three obstacles and two fans blowing air in
the +y direction, as seen in Figure 13. Obstacles in Figure 13
are represented as circles having a radius equal to the actual
obstacle size plus the size of the UAV. In Figure 13(a), the
intermediate goal positions queried by the NN are shown
using the same color code as the simulation results. The
UAV queried 20 points to avoid the first obstruction until it
found a safe intermediate goal and repeated this operation
until it reached its final destination. Using this NN-based
framework, it takes a few milliseconds to search for a safe
target. The experimental results of the proposed approach
are compared with the ones in which the NN is not utilized
to make safety decisions about intermediate goal positions.
As expected, in Figure 13(b), the UAV fails to complete its
mission safely, as it crashes and gets very close to the other
obstacles. These experiments were executed without the
obstructions, which are overlaid in the figure for reference.

However, many challenges remain that need to be addressed to fully integrate these technologies in safetycritical operations. In this section, we provide an
overview of these challenges and offer some possible
solutions and directions for future research on assured
runtime monitoring.
Discussion
A first challenge typical of LECs centers around how to
select the appropriate training set. The accuracy of any
machine learning technique depends largely on the type,
amount, and heterogeneity of the data used during the
training phase. A poorly trained NN results in poor performance, leading to unsafe or overconservative behavior
of autonomous systems. Similarly, overfitting can introduce overhead and poor prediction. To deal with these
issues, it is possible to perform sensitivity analysis [18] and
nonconformity analysis [19] on the system prior to training to better interpret and select data. It is also possible to
leverage knowledge about the system dynamics to perform
verification before the deployment of the LEC, as suggested in this article.
Verification provides safety guarantees for the outputs
of such LECs; however, it does not provide any robustness
guarantees against changes between training and testing
conditions. Currently, state-of-the-art verification techniques, beyond machine learning applications [20], require
knowledge about system model and bounded conditions.
However, no guarantees can be provided if, for example,
the disturbance acting on a system is above or below the
bounds for which training was performed. This challenge
becomes even more evident when dealing with real systems. In fact, typically (including in our setup in this article), verification considers a specific model that may and
will change from the actual model of a real system. Even
precise system-identification techniques are not able to

Conclusions
The recent interest in LECs and their rapid introduction in
our society, in particular in autonomous systems technologies, reveal considerable potential and benefits, especially in
terms of computation overhead and decision-making applications. However, new challenges have emerged due to the lack
of models and the complete reliance on data that do not provide the assurance and guarantees necessary for their deployment in safety-critical operations.
The framework for verification discussed in this article moves toward this assurance-driven design of LECs.

-1

y (m)

Safe Final Position
Unsafe Final Position
Safe NN Decision
Unsafe NN Decision
Initial Position
Obstacle
-1
-1

-1

0
1
x (m)

-1

0
1
x (m)

-1

0
1
x (m)

-1

0
1
x (m)

-1

0
1
x (m)

Figure 12. The safe and unsafe final positions and NN results from different initial locations in the experimental setting. The NN here
is composed of four input nodes (an x-y initial position and x-y final position pair), one hidden layer of 40 nodes, and one output for
the safety decision. The NN performed with 0% FPs and roughly 16% FNs.

114

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020