IEEE - Aerospace and Electronic Systems - October 2022 - 28

Networks of UAVs of Low Complexity for Time-Critical Localization
state using new measurements and by applying the
Bayes' rule.
Kalman [36] and particle filters [29] are common methods
for solving sequential estimation problems. Probabilistic
mapping is also useful to reconstruct a binary map of the
environments [37]. In cooperative scenarios, all these methods
can be used in a distributedmanner making use offactor
graphs, belief propagation or distributed particle filtering
(PF) [38], [39].
POLICY AND CONTROL ENGINE
In the following, we discriminate between methods that
rely on accurate models of the environment, solved using
classical optimization techniques, from methods that learn
from data and experience using artificial intelligence/
machine learning (AI/ML) tools. Then, we briefly overview
both categories and, especially for time-critical applications,
we foresee the possibility to adopt hybrid
techniques that can fuse model-based and model-free
(data-driven) approaches in order to overcome situations
that cannot be efficiently solved by one category alone.
Table 1 provides a summary of the comparison
between architectures and optimization methods for control
design. In addition, we remark that the following models
are described from a high-level point-of-view. The
hardware implementation and the low level control are
out of the scope of this article.
Model-based optimization: Classically, this problem is
solved by using model-based optimization, e.g., non-linear
programming, where a single cost function or a limited set of
them is minimized either in a greedy fashion or by closedform
solutions, allowing for taking quick decisions.
For example, in [5] and in [40] the UAV navigation
problem is described as the minimization of the posterior
covariance matrix (sequential mean squared error) on target
estimation subject to UAV anti-collision and mobility
constraints. In this case, the cost function to be minimized
can be analytically derived because observation and transition
models are both known and Gaussians. Then, to
solve the problem, a projected steepest gradient descent
algorithm is used whose major steps are [40]:
1) analytical derivation ofthe gradient ofthe cost function
with respect to the UAV positions;
2) computation of an initial solution with the steepest
gradient descent of the cost function;
3) derivation of the projection matrix to constrain the
initial solution onto the tangent space of the potentially
violated constraints;
4) corrections to compensate for the nonlinearity effect
of the constraints (nonlinearity correction);
5) derivation of the projected control for UAVs [41].
28
Unfortunately, when the dynamics of the environment
are only partially known, model-based approaches might
not be sufficiently accurate. In this context, ML techniques,
i.e., model-free methods, become essential for solving
a large number of heterogeneous and, sometimes,
competing tasks (e.g., mission duration, storage capacity,
and communication requirements) [21].
Data-driven approaches: The interaction between a
UAV and the environment can be described under the
umbrella of Markov decision processs (MDPs) or as partially
observable MDP if the state is unknown, as shown
in Figure 2. An MDP is defined by a tuple containing the
space of possible states/observations, actions and rewards,
and the probability of state transition. In the case of multiple
UAVs, the formulation might be extended using the
notions ofteam stochastic game [28], [42].
Notably, UAVs act as local agents, and the state represents
the available knowledge about the environment
(e.g., the position and velocity of a target in a tracking
application). Each UAV, then, interacts with the environment
through the chosen actions (e.g., UAV movements)
and, in turn, it receives rewards (e.g., localization accuracy).
In this sense, a policy is a function that maps the
observations to the actions, and, hence, learning is the process
that allows inferring such a policy.
Typical machine learning (ML) approaches are usually
distinguished in supervised learning, which learns from a
training set of labeled examples, and unsupervised learning,
which extracts hidden features from unlabeled datasets.
However, they do not usually consider the whole problem of
an application-driven agent that interacts with an uncertain
environment to maximize a reward. Instead, in reinforcement
learning (RL), a thirdMLparadigm, the learning is carried
out through interactions ( " trial-and-error " ) and actions
are selected to maximize the sum of the discounted rewards
over the future ( " delayed rewards " ) [28], [43].
The policy can be estimated through apolicy search or
value-based learning approach. In policy search learning,
the policy can be represented by a neural network with
observations as inputs and actions as outputs, so that the
policy is directly inferred by approximations. Thus, learning
simply consists of adjusting the parameters of the neural
network to find the optimal input/output relationship.
Possible algorithms to perform such updates are represented
by policy gradient methods [28]. Unfortunately,
such methods might be stuck on local maxima.
Differently, value-based learning is a mapping of a
state-action pair into a value function, defined as the
expected return, i.e., a discounted reward sequence over a
certain time horizon. Since the policy should return an
action to the UAV, it assesses the values for every possible
action and decides either for the optimal action (exploitation)
or a random one (exploration).
Implementations of this policy include approaches
based on dynamic programming, Monte Carlo methods,
IEEE A&E SYSTEMS MAGAZINE
OCTOBER 2022
IEEE - Aerospace and Electronic Systems - October 2022

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - October 2022
