IEEE Robotics & Automation Magazine - June 2023 - 76

Ensemble Mixture [1], critic-regularized regression [133],
implicit Q-learning [63], and MuZero Unplugged [116].
Another approach to offline RL is to leverage techniques
developed in machine learning, such as the work by Chen et al.
[20], who utilize the transformer architecture conditioned on
the desired reward, past states, and action to produce future
actions that achieve a desired return. Other work addresses
offline data themselves. Yarats et al. [140], e.g., propose using
reward-free unsupervised data first and then annotating the
reward to learn an RL policy. There are also some efforts to
produce large offline datasets. Dasari et al. [26] introduce an
open database to learn models for vision-based robotic manipulation
that consists of 15 million video frames for seven different
robot manipulators.
PARALLEL LEARNING
Parallel learning deals with utilizing one and possibly more
heterogeneous hardware resources in the most efficient way
by means of parallelization [see Figure 5(b)]. Furthermore, it
addresses how to implement scalability and the necessary
robustness to handle different sizes of a learning process.
There are several formulations of parallel learning architectures
that were developed throughout the years, such as A3C
[88], IMPALA [32], Ape-X [44], D4PG [11], and R2D2 [58].
All the mentioned architectures are robust to parallel deployment
and show large improvements in the sample efficiency
and performance of the final policy over the baselines. There
are also some adjacent optimization paradigms, such as evolutionary
strategies [114], that can be scaled to large proportions
and are capable of producing competitive policies in
comparison to RL-based ones. Other work by Mania et al.
[80] proposes using a variant of random search, augmented
random search, that is also very scalable and derives nearoptimal
policies.
Other work makes use of hardware accelerators to permit
massive parallelization and thus facilitate the training process
[73]. In this vein, Makoviychuk et al. [79] present Isaac Gym,
a fully GPU-based simulator for RL that is capable of simulating
a high number of environments by using a single GPU.
Building on this, Rudin et al. [113] use Isaac Gym to train a
quadruped to walk in minutes over increasingly complex terrain.
Finally, the combination of other optimization techniques
with RL seems to be a promising approach. For example, Jaderberg
et al. [48] present a two-level optimization evolutionary
process targeting a population of RL agents. This framework
enables agents, conditioned on pixels only, to play a complex
3D game, matching human performance.
LEARNING FROM DEMONSTRATION
Although learning from demonstration [115] is a research
field on its own, many RL approaches make use of such techniques.
As an introduction to the field, we refer to Osa et al.
[99] and Billard et al. [15]. Aside from standard techniques,
such as behavior cloning [9] (learning offline data in a supervised
way) and Dataset Aggregation (training the policy to
mimic an expert in an online fashion) [112], novel approaches
are presented by Florence et al. [35], proposing to use implicit
behavior cloning and to let the policy be presented by an
energy-based model [67]. Laskey et al. [65] present the Disturbances
for Augmenting Robot Trajectories algorithm that
collects demonstrations with injected noise while adjusting
the noise level according to the trained policy.
Other approaches first train a teacher policy on unchanging
task setups via, e.g., RL and distill a policy capable of interpolating
among different task setups, such as [7], which chooses
neural dynamical policies [8] to present the teacher and students.
Others learn teacher policies on true state information
to then derive a student policy conditioned on a reduced or substituted
input space, e.g., Chen et al. [21], whose final visionbased
policies are able to reorient objects in the shadow hand
domain, and Lee et al. [68], who also distill a vision-based
policy and test it in their red-green-blue-stacking benchmark.
Other work makes use of classical optimization methods to
represent the expert and distill their demonstrations into trainable
policies. Wang et al. [131] use a hybrid learning process
of RL and imitation learning, with the Optimization-based
Motion and Grasp planner [130] as the expert.
Other than using a teacher, some work utilizes human
demonstrations. Akbulut et al. [2] introduce a new framework
called Adaptive Conditional Neural Movement Primitives,
combining supervised learning and RL to conserve old skills
learned from robot demonstrations while being adaptive to
new environments. James and Davison [51] present a coarseto-fine
discrete RL algorithm to solve sparse reward manipulation
tasks by using only a small amount of demonstration
and exploration data (work extended by [49] and [50]). Celemin
et al. [19] include human corrective advice in the action
domain through a learning-from-demonstration approach,
while an RL algorithm guides the learning process by filtering
out human feedback that does not maximize the reward.
Trainer
Sampler
(a)
(b)
Trainer
Teacher
(c)
FIGURE 5. The guided RL methods integrated as a learning strategy. (a) Offline RL from a recorded dataset (see the " Offline RL " section).
(b) Parallel learning with multiple samplers and trainers (see the " Parallel Learning " section). (c) Learning from demonstration with
distilled student policies (see the " Learning From Demonstration " section).
76 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
Student
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
