IEEE Robotics & Automation Magazine - December 2021 - 12

Standard Control
Preliminary works on autonomous staircase traversal were
devoted to highly customized standard robot control. One of
the first was presented in [7], relying on the estimation of step
edges and gyroscopic data. A highly customized platform was
presented in [9], where, thanks to a sophisticated track platform,
the problem of staircase traversal was tackled, further
aided by an actively controlled pendulum. Unfortunately,
approaches that are highly customized to specific hardware
generally have limited potential.
To palliate the increased cognitive load experienced by
users that teleoperate such platforms, the authors of [8] propose
a motion planning framework where a robot exploits
side laser sensors to perceive the geometry of upcoming
obstacles and adapt its flippers so that they are tangential to
the surface. Defining the robot's negotiation limits, the
authors believe the system is able to negotiate any uneven terrain,
based on exhaustive sensory data. The framework in
[14] is comparatively superior, as the system is endowed with
passive flippers that push against a traversed obstacle to
improve traction, combined with an accident warning system
based on the estimation of the normalized energy stability
margin (NESM). Still, the robot is not autonomous, and the
overall approach is weakly generalizable.
Another approach relying on the precise estimation of
staircase configuration was proposed in [12], where a robot
could autonomously negotiate a set of steps by using proprioceptive
state estimation. That approach had excellent performance;
however, the system could negotiate only a staircase
known beforehand. One of the most elaborate analyses was
presented in [10] using a common tracked platform with an
additional 2 DoF due to a controllable arm. The derived analytical
solution is indicative of the high complexity of modeling
the interaction between a multi-DoF system and a
traversed surface. That system lacked congruent control of the
tracks, flippers, and arm and needed full knowledge of geometry,
kinematics, and physical characteristics, which can be
difficult to acquire.
Learning-Based Control
Learning-based approaches appear more suitable for developing
controllers when robot kinematics and dynamics are
harder to model. The authors of [15] identify two main categories
of learning-based methods, terrain traversability
analysis and an end-to-end approach, while [1] presents the
diversity of employed learning techniques along with challenges.
One of the main requirements is scalability to highdimensional
state and action spaces. Priors could bootstrap
learning; still, this demands more attention in the design of
the search policy algorithm to ignore irrelevant policy
properties. Generalization and robustness are relevant for
robots that are required to learn different tasks under varying
conditions.
To the best of our knowledge, earlier work on learningbased
control for actively articulated tracked robots is relatively
scarce. At the same time, in staircase ascent and descent, where it
12 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2021
is hard to model the dynamics of physical interaction, learning-based
techniques are particularly promising. Policy
endowment with properties such as bumpiness and stability
can be performed in different ways. Constraints may be
introduced in an RL optimization problem, as in [11], while
safety and reward can be associated in a single function [16].
Another more sophisticated approach was demonstrated in
[11], where the authors incorporated constraints into the
optimization problem of relative entropy policy search
using RL. However, the results were limited to the traversal
of a palette.
The authors of [13] use images for the navigation of a
tracked robot in indoor environments, improving the dueling,
double, deep Q-network (DQN) model with layer normalization
and noise injection, and propose an original
algorithm for its training. Yet, this approach is employed
only for navigation on 2D surfaces. The DQN was one of
the first deep RL algorithms and showed fascinating results
[17]. Over the years, RL algorithms have undergone significant
evolution, with the current state of the art composed
of actor-critic algorithms, which benefit from state-action
value estimation, as in pure, value-based DQNs, and directly
optimize policy parameters through gradient ascent.
Indicatively, the off-policy deep deterministic policy gradient
(DDPG) algorithm [18] moved a discrete action space
to a continuous one as an approximate DQN. Still, the major
improvement of the DDPG, which introduced twin-delay ed
DDPG (TD3), corresponds to enhanced DDPG with
delayed policy improvement, clipped double Q-learning,
and target policy smoothing. In parallel to the latter algorithm,
soft actor-critic (SAC) was proposed, which contains
the same features combined with a novel policy entropybased
maximization. An earlier on-policy algorithm is
proximal policy optimization (PPO), which yields more
conservative updates.
Robot Control Framework
Following a typical RL-based problem formalization, given a
state
s
t S! the agent selects actions a At
,
t
00 TT
-,,
,
11
! according to a
S ,
policy r and transits to the next state st 1+ ! receiving the
reward rR .! The array of consecutive state-action pairs
x = {, ,.., ,}sa sa creates a trajectory known as a rollout.
The set of sa strtt t+1 observed along a rollout is used
by RL algorithms to perform policy optimization. In the following,
we detail the chosen action and state spaces, present
the reward function and controller for staircase traversal, and
the manner in which KL divergence is used to compare policies
and their transfer to reality.
Problem Description
We consider a robot that can minimally control 3 DoF corresponding
to front and rear flipper angles and linear velocity,
occasionally complemented with an extra 2 DoF from a
robotic arm whose use can prove particularly helpful in staircase
ascent and descent. Therefore, the continuous 5-DoF
action space forms the following action vector:
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
