IEEE Robotics & Automation Magazine - June 2023 - 81

space [82]. Two potential directions include latent action
spaces [106] and hybrid strategies [27], where model-free RL
and model-based approaches are interleaved meaningfully.
LEARNING STRATEGY
A key challenge with offline RL seems to be effectively processing
the collected training data to extract the necessary
information [1], [18], [39]. Current and future directions in
this domain focus on introducing offline RL datasets [1], [26]
and proposing novel algorithms to improve data usage [1],
[39], [133]. With parallel learning, a main challenge is to
design the information flow of the components to be parallelized
[32], [44], [88]. Directions include improving the robustness
of the parallel learning scheme [32], [11] while scaling to
larger architectures [11], [44], [88]. In cases of learning from
demonstration, potential challenges include accurate behavior
modeling and extrapolating the behavior to new situations [7],
[35], [124]. Current directions include generalizing behavior
beyond specific demonstrations and creating sophisticated
benchmarks for evaluating trained policies [2], [21], [68].
TASK STRUCTURING
Potential challenges with curriculum learning include selecting
the right sequence of subtasks to be trained and suitable
TABLE 3. The primary methodological approaches of guided RL.
TAXONOMY
Pipeline
KEY
MOTIVATION
Method
Source
(See the
" Evaluation of
Approaches "
Section)
Problem
formulation
State
representation
Reward design
World
knowledge
Expert
knowledge
Abstract learning Expert
knowledge
Learning
strategy
Offline RL
Expert
knowledge
Parallel learning Expert
knowledge
Learning from
demonstration
Task
structuring
Curriculum
learning
Expert
knowledge
World
knowledge
Hierarchical RL World
knowledge
Sim-to-real Perfect simulator Scientific
knowledge
Domain
randomization
Domain
adaptation
World
knowledge
Expert
knowledge
Effectiveness
and efficiency
Effectiveness
and efficiency
Efficiency and
effectiveness
Effectiveness
and efficiency
Efficiency and
effectiveness
Efficiency and
effectiveness
Effectiveness
and efficiency
Effectiveness
and efficiency
Sim-to-real
and efficiency
Sim-to-real
and effectiveness
Sim-to-real
and
effectiveness
FUNDAMENTAL
IDEA
(See the " Description
of Methods "
Section)
Employ states with
more instructive
representations
Shape or learn
dense reward
function
Substitute complex
actions spaces
with task-specific
ones
Learn policies from
recorded dataset
Deploy parallelization
of the learning
algorithm
Train a policy
based on example
trajectories
Iteratively solve
more complex
tasks
Decompose complex
task into hierarchy
of subtasks
Build more
realistic training
environment
Randomize visual
and dynamics
parameters
Transfer observations
among
domains
POTENTIAL
CHALLENGES
(See the " Discussion
of Challenges
and Directions "
Section)
Balance state richness
and computing
effort
Select taskspecific
reward
terms and parameters
Choose
appropriate
levels of
abstraction
Process the
collected training
data effectively
Design information
flow of parallel
components
Accurate behavior
modeling and
extrapolation
Select task
sequence and subtask
difficulty
Design appropriate
hierarchical structure
Accurately
model
robots, physics,
and environment
Determine randomization
parameters
and ranges
Select domains
and design adaptation
module
POTENTIAL
DIRECTIONS
(See the " Discussion
of Challenges
and Directions "
Section)
Combine multimodal
sensor
information
Bioinspired
shaping and
inverse RL
Hybrid RL and
model-based
approaches
Novel offline
RL algorithms
Robust learning
along with scalability
Generalize
behavior
beyond specific
demonstrations
Effective difficulty
progression and
automatic task
generation
More complex
tasks and longhorizon
tasks
Successful
zero-shot transfer
Informative randomization
and
automatic adjustment
No
overlap
between source
and target domain
Based on the introduced taxonomy (see the " Taxonomy " section), for each method, we summarize the key motivation (see the " Evaluation of
Approaches " section), fundamental idea (see the " Description of Methods " section), and potential challenges and directions (see the " Discussion of
Challenges and Directions " section).
JUNE 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
81
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
