IEEE Robotics & Automation Magazine - June 2023 - 79

Domain adaptation based on vision was done by several
researchers. Bousmalis et al. [17] implement the former
idea via GraspGAN, where an adaptation module is trained
to convert synthetic images taken from simulation to more
photorealistic observations. James et al. [52] present randomized-to-canonical
adaptation networks, which learn to project
synthetic images derived from randomized simulations into
the style of the canonical simulation. Rao et al. [110] present
RL-CycleGAN, which converts synthetic images into more
realistic images. Liu et al. [75] introduce an approach called
real-sim-real that adapts the real-world state into a simplified
one by a segmentation model. Zhang et al. [143] propose adaptation
modules, which are trained independently of the deep
RL agent and can be deployed for different scenarios, e.g.,
indoor and outdoor navigation. Hoeller et al. [43] introduce a
navigation policy for ANYmal that can navigate in cluttered
environments with static and dynamic obstacles.
Other work investigates using an adaption module to handle
environmental factors. Peng et al. [104] introduce a framework
for training quadrupedal robots to imitate agile locomotion
skills from animals, where the learned policies can then be
transferred from simulation to the real world through a sampleefficient
domain adaptation process. Kumar et al. [64] present
a rapid motor adaptation algorithm that adapts in real time to
unseen real-world scenarios.
EVALUATION OF APPROACHES
In this section, we present a systematic evaluation of guided RL
approaches. As we show here, combining multiple methods
leads to improvements in all three guided RL dimensions,
namely, efficiency, effectiveness, and sim-to-real transfer, especially
when specific combinations are used. In the following,
we first describe the methodical approach and then present key
insights for both individual methods and combinations of those.
METHODICAL APPROACH
Table 2 provides an overview of the guided RL approaches
discussed in the " Description of Methods " section. According
to the three dimensions of guided RL
(see the " Concept of Guided RL " section),
we identify, for each of the discussed
dimensions, whether 1) the
overall training time has been reduced
(efficiency), 2) improved policy performance
has been achieved (effectiveness),
and 3) the trained policy has
been deployed to the real world (simto-real).
Specifically, we adopt the
achievements claimed by the authors
themselves along the three dimensions,
verified by means of figures, tables,
and specific text passages, respectively.
Furthermore, for more in-depth analysis,
the last column of the table shows
which specific guided RL methods
were used in each of the approaches.
Consequently, Table 2 provides a structured overview of the
approaches both in terms of achievements along the three
dimensions and used guided RL methods.
Based on those classified references, Figure 8 displays the
normalized contribution of the respective methods in terms of
efficiency, effectiveness, and sim-to-real. For instance, among
the covered papers adopting hierarchical RL, many approaches
have shown improvements in terms of policy performance,
and hence, this method seems to contribute significantly
toward increasing the effectiveness of the learning approach.
To increase the statistical significance of the evaluation, not
only papers of the corresponding method are considered but
also all papers adopting that method (see the " Guided RL
Methods " column in Table 2).
KEY INSIGHTS ON INDIVIDUAL METHODS
As the quantitative evaluation of references has shown (Figure
8), particular methods tend to lead to improvement in
terms of efficiency, effectiveness, and sim-to-real. The following
key insights can support selecting individual methods
to increase the probability of an approach being more efficient
and effective and reaching real-world deployment.
IMPROVING THE EFFICIENCY
As our findings show, parallel learning architectures, abstract
learning, and learning from demonstration data, in particular,
often lead to accelerating the RL training process. First, an
efficient parallelization of the algorithmic components allows
scaling the learning problem to different sizes [32], [44], [88].
Second, simplifying the learning task by means of task-specific
action spaces and hybrid model-based and model-free
approaches can improve the overall efficiency. Finally, training
based on expert demonstrations tends to be rich in information
and hence can accelerate policy training [7], [124],
[131]. Furthermore, the efficiency can likely be improved by
employing more instructive state representations, applying a
curriculum to gradually tackle difficult learning tasks, and utilizing
accurate simulation environments.
State Representation
Reward Design
Abstract Learning
Offline RL
Parallel Learning
Learning From Demonstration
Curriculum Learning
Hierarchical RL
Perfect Simulator
Domain Randomization
Domain Adaption
Efficiency (Minimize Training Time)
Effectiveness (Maximize Performance)
Sim-to-Real (Successful Deployment)
FIGURE 8. The evaluation of guided RL methods. Based on our literature review (see
Table 2), quantitative results show the relative contribution of the respective methods for
accelerating the training process (efficiency), improving the overall policy performance
(effectiveness), and successful real-world deployment (sim-to-real).
JUNE 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
79

IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023