IEEE Robotics & Automation Magazine - June 2023 - 78

representations, respectively. System identification [76] is about
building a precise mathematical model of a physical system. In
the context of robotics simulation, carefully tuning physical
parameters, such as friction, weight, and elasticity, can significantly
increase the realism of the simulator. Moreover, machine
learning approaches can be applied either offline [57] or, as
presented by Yu et al., in an online fashion by predicting the
dynamics model parameters in real time [141].
Accurately simulating the complex dynamics of modern
robots also imposes high demands on the choice of the physics
engine. Erez et al. [30] analyze quantitative measures of simulation
performance and speed related to solving the numerical
challenges of multibody dynamics present in robotics. Besides
choosing an appropriate physics engine, the physics simulator
has to support the need of the robotics use case. As the authors
of [25] conclude, for each robotics subdomain, different simulators
are preferred depending on the relevance of, e.g., sensors,
dynamic contacts, and friction modeling. Muratore et al.
[90] apply dynamics randomization and use a newly developed
algorithm to switch parameters of the domain randomization,
stopping overfitting to simulator dynamics. Lowrey et al. [77]
leverage real-world robot data to carefully identified robot
parameters [62], enabling an RL-trained policy to transfer
directly from simulation to reality. Heiden et al. [42] propose
a hybrid simulator with learned neural networks that switches
between analytical and learned computation of physical effects.
Xia et al. [137] present the Gibson environment, capable of realistic
visual perception for active agents based on real-world data.
Finally, an accurate representation of the environment can
significantly reduce the reality gap. Ramos et al. [109] present
BayesSim, a framework that offers adaptive Bayesian estimates
for simulation parameters via simulation-based inference,
while Golemo et al. [40] introduce neural-augmented simulation,
a method for augmenting robotic simulators with real
robot trajectories. Hwangbo et al. [46] present a neuronal net
trained on real data for ANYmal to convert policy action into a
torque value for the simulation model.
DOMAIN RANDOMIZATION
The idea behind domain randomization is to highly randomize
the simulation along a wide range of parameter distributions
[see Figure 7(b)]. Instead of carefully modeling the
real-world parameters in simulation, the real world simply
appears as just another variation of these distributions [144].
Depending on the parameters to be randomized, common
approaches deploy the randomization of either visual or
dynamics components. For a more thorough survey on randomization
simulations, the reader is referred to [92].
Tobin et al. [123] first introduced the idea of randomizing
rendering in the simulator to transfer neural networks to reality
for the purpose of robotic control. Mehta et al. [84] propose
active domain randomization, which learns a parameter sampling
strategy to leverage the randomization ranges that are the
most informative. OpenAI et al. [98] present automatic domain
randomization, which adjusts the domain randomization environment
parameters, depending on the policy success, for solving
a Rubik's Cube with a real robot hand. Prakash et al. [108]
present structured domain randomization, which creates context-aware
synthetic data by taking into account the structure
of a scene. Instead of randomizing visual components of the
simulator, Peng et al. [102] introduce dynamics randomization
that includes parameters such as link masses, joint damping,
and proportional-derivative gains, respectively.
Muratore et al. [91] introduce neural posterior domain randomization,
which adapts the simulator's parameters by using
only a few real-world rollouts to match the observed dynamics.
Tsai et al. [124] leverage a single human demonstration to identify
the simulator's distribution over dynamics parameters and
adapt the domain randomization to reduce the sim-to-real gap.
Ideas similar to visual and dynamics randomization have been
adopted in other works, where perturbances are introduced to
obtain more robust agents. For example, Wang et al. [128] consider
noisy rewards, while other recent works apply noisy sensor
signals [111] and random external forces [113] for effective
policy deployment in the real world.
DOMAIN ADAPTATION
Domain adaptation techniques aim to minimize the reality gap
by training adaptation modules, often represented by autoencoders,
capable of projecting one domain into another, e.g., realworld
camera images to simulation look-alikes [see Figure 7(c)].
The goal domain can be simulated environments, the real world,
and abstract latent spaces. Wang et al. [132] present a comprehensive
survey on this field of research.
Distribution
Simulation Parameters
Real World
Real-World
Parameters
Simulator
Source Domain(s)
Target Domain
(a)
(b)
(c)
FIGURE 7. The guided RL methods integrated for sim-to-real. (a) The perfect simulator to minimize the reality gap (see the " Perfect
Simulator " section). (b) Domain randomization to match the real-world parameter distribution (see the " Domain Randomization " section,
based on [144]). (c) Domain adaptation for matching source and target domains (see the " Domain Adaptation " section).
78 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
