IEEE Robotics & Automation Magazine - September 2023 - 49

such cannot predict information that is not
already present in the input data set. The
remaining option is then RL, which is based
on a control loop (as shown in Figure 3),
where an actor predicts an action from the
current state, and this action is then given to
an environment, which updates the state
based on the action [15]. The action is determined
through a guided search-based method
[15], which determines the action that
maximizes a reward over time. In our case,
RL is an appealing solution as it naturally fits
into classical control-loop systems, needing
only an environment to experiment with different
actions, and a reward signal to determine
the optimal action for a given state [16].
At the time of this writing, the most stud "
UNFORTUNATELY,
DUE
THE NATURE
OF MODEL-FREE,
POLICY ITERATION
RL METHODS, THEY
ARE SUBOPTIMAL
IN CONVERGENCE
TIME WHEN COMPARED
WITH MODEL-FREE,
TEMPORAL
DIFFERENCE RL.
ied class of RL schemes is model-free temporal
difference RL, which updates its actor at
each iteration of the control loop and does not
require an existing system model to converge. However, these
procedures cannot be used in this article, due to
■ interstate differences being negligible at high frequencies
(i.e. control-loop frequencies higher than 1 Hz in this case
study), causing observation issues with the Markov decision
process on which temporal difference RL is based.
■ action delays induced by imperfect controllers and inertia,
resulting in dissociations among states, rewards, and
actions (i.e., an action taken at time T might affect the system
at time
Tn ).+
■ noise cancellation during action exploration at high frequencies,
causing the exploration to be nullified, preventing
the methods from converging.
This means that popular techniques such as the Deep
Deterministic Policy Gradient, Advantage Actor Critic, and
Proximal Policy Optimization cannot be used. Therefore, a
different class of RL approaches is applied in this article.
As such, the selected class of RL methods is model-free,
policy iteration RL [15], where the actor is updated only when
a task is completed or after a set number of iterations. These
schemes require an optimization algorithm to be applied to the
actor to converge toward an optimal solution. A few papers have
shown that these processes have similar performance when
compared to temporal difference RL methods [12]. More specifically,
this case study, which uses a neural network [11] as
an actor to predict gain because it has a continuous inputs and
outputs, is a universal function approximator that can be well
generalized to a given task and can be computed quickly. The
CMA-ES approach was the optimizer chosen [17] because it
is able to improve high-dimensional search spaces (i.e., higher
than 10); and nonlinear, nonconvex, and highly modal search
spaces, such as the ones encountered using neural networks.
Furthermore, the task itself is highly nonlinear and complex
due to dynamics at the wheel-ground interface. These characteristics
make neural networks with a CMA-ES optimizer an
ideal combination for determining a real-time, neural-based,
„
gain-estimation technique. Unfortunately, due
the nature of model-free, policy iteration RL
methods, they are suboptimal in convergence
time when compared with model-free, temporal
difference RL. This means that training the
actor must be done in simulation as the time
to convergence of the neural network might be
in the order of years of real-time experiments,
using multiple robots in parallel.
As such, the procedure has two phases: The
first is a training phase in simulation using
CMA-ES to determine an optimal neural network.
The second phase is where the trained
neural network, by itself, is defined as the gain
estimator model and is then used in real time
to predict optimal gain from the input state.
TRAINING OF THE NEURAL NETWORK
The CMA-ES method, guided by an obj function,
optimizes parameters of the neural network
(i.e., weights and biases), to determine a neural network
actor that can predict the optimal gains from a given input
state. The CMA-ES optimizer is able to train the actor-by
testing multiple candidates of neural networks in multiple
parallel environments-to determine a gradient vector, that
is, the neural network parameter space, that minimized the
obj function [17]. By employing this, the CMA-ES method
can move parameters of the neural network toward a local
optimum that minimizes the obj function, meaning that the
actor is capable of determining optimal gains using the given
input state, similar to gradient descent techniques [18].
The following obj function is used to train the neural network,
aiming at minimizing the lateral error and steering
angle along the trajectory to be followed:
N
obj
=+ d
=
|( )| .| ()|[ ],
05
n
@
1 / 6 yt Lt dtnF m
T n
(10)
where T is the total time, y is the lateral error, Fd is the front
steering angle, n is the current number of iterations, and N is
the total number of iterations. This obj function is similar to
the one used in previous works [5]; however, it does not
include the angular deviation factor as it prevented the CMAES
optimizer from reaching a more optimal solution. Furthermore,
the removal of this angular deviation from the obj
function did not increase oscillations, as previously thought.
Actor
Observation
ot
Reward
rt
Environment
FIGURE 3. The block diagram of a control loop using RL.
SEPTEMBER 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
49
Action
at
IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023
