IEEE Robotics & Automation Magazine - June 2023 - 66

100
20
40
60
80
Simple
Medium
Task Difficulty
FIGURE 10. The average elapsed time of different control methods
at different task difficulties.
consumes numerous computational resources but is meaningful.
For standard image augmentation, the texture and
lighting conditions are fine-tuned according to the actual situation.
The result shows that image augmentation can hardly
affect the performance of the model. However, we finally
decided to include the image augmentation process to
increase the robustness to natural disturbances. We use one,
five, and 10 frames of historical states of robot arms as network
input, and the results show that historical states can
improve performance, but excessive data increase the training
cost, so five frames are specified. The success rate of
sim-to-real transfer with the BC algorithm is significantly
inferior to that of the proposed algorithm, which proves the
effectiveness of the GAIL algorithm in the transfer process.
Besides, as shown in Figure 9, our agents manipulate more
rapidly and efficiently compared to scripts in most cases. The
average time consumed to complete the assembly task in each
mode under the three tasks of different difficulties is shown
in Figure 10, which shows that the RL algorithm based on
simulation training and domain transfer outperforms script
and keyboard operations on simple tasks and even approaches
DM on complex tasks.
CONCLUSION
Controlling robots to automate complex tasks has been an
important area of research. In this work, we demonstrate that
adapting an RL method based on training in a simulation
environment and sim-to-real transfer can outperform manually
written scripts. Our method implements three stages for
training robots. First, a physical simulator based on Unity3D
is elaborated to simulate real-world scenarios; second, largescale
distributed robot training is performed on physical simulation
to obtain an agent proficient in assembly tasks in a
simulation environment; and third, sim-to-real transfer is conducted
to adapt to real-world deployment. The experimental
results show that the proposed algorithm outperforms other
algorithms, and the real robot arm completes the assembly
task significantly faster than script and keyboard operations.
Despite this success, the work also reveals the potential of RL
66 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
Complex
DRL
MK
DM
SA
methods when robots are confronted with complex tasks. In
future work, we will attempt to improve the sample efficiency
to speed up the training process of the neural network.
ACKNOWLEDGMENTS
We gratefully acknowledge the financial support from The
National Key R&D Program of China (2021YFF0306405).
The corresponding author is Hong Wang.
AUTHORS
Daqi Jiang, School of Mechanical Engineering and
Automation, Northeastern University, Shenyang 110819,
China. Email: 1910103@stu.neu.edu.cn.
Hong Wang, School of Mechanical Engineering and
Automation, Northeastern University, Shenyang 110819,
China. Email: hongwang@mail.neu.edu.cn.
Yanzheng Lu, School of Mechanical Engineering and
Automation, Northeastern University, Shenyang 110819,
China. Email: 1810096@stu.neu.edu.cn.
REFERENCES
[1] A. Billard and D. Kragic, " Trends and challenges in robot manipulation, " Science,
vol. 364, no. 6446, Jun. 2019, Art. no. eaat8414, doi: 10.1126/science.aat8414.
[2] D. Silver et al., " A general reinforcement learning algorithm that masters
chess, shogi, and Go through self-play, " Science, vol. 362, no. 6419, pp. 1140-
1144, Dec. 2018, doi: 10.1126/science.aar6404.
[3] M. L. Lee, S. Behdad, X. Liang, and M. Zheng, " Task allocation and planning
for product disassembly with human-robot collaboration, " Robot. Comput.-Integr.
Manuf., vol. 76, Aug. 2022, Art. no. 102306, doi: 10.1016/j.rcim.2021.102306.
[4] A. X. Lee et al., " Beyond pick-and-place: Tackling robotic stacking of diverse
shapes, " in Proc. 5th Annu. Conf. Robot Learn., 2022, vol. 164, pp. 1089-1131.
[5] V. Mnih et al., " Playing atari with deep reinforcement learning, " 2013,
arXiv:1312.5602.
[6] V. Mnih et al., " Asynchronous methods for deep reinforcement learning, " in
Proc. Int. Conf. Mach. Learn., PMLR, 2016, pp. 1928-1937.
[7] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, " Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor, " in Proc.
Int. Conf. Mach. Learn., PMLR, 2018, pp. 1861-1870.
[8] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, " Proximal
policy optimization algorithms, " 2017, arXiv:1707.06347.
[9] T. Van Erven and P. Harremos, " Rényi divergence and Kullback-Leibler divergence, "
IEEE Trans. Inf. Theory, vol. 60, no. 7, pp. 3797-3820, Jul. 2014, doi:
10.1109/TIT.2014.2320500.
[10] J. Tan et al., " Sim-to-real: Learning agile locomotion for quadruped robots, "
2018, arXiv:1804.10332.
[11] J. Hwangbo et al., " Learning agile and dynamic motor skills for legged
robots, " Sci. Robot., vol. 4, no. 26, Jan. 2019, Art. no. eaau5872, doi: 10.1126/scirobotics.aau5872.
[12]
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, " Trust region policy
optimization, " in Proc. Int. Conf. Mach. Learn., PMLR, 2015, pp. 1889-1897.
[13] R. Jeong et al., " Self-supervised sim-to-real adaptation for visual robotic
manipulation, " in Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2020, pp. 2718-
2724, doi: 10.1109/ICRA40945.2020.9197326.
[14] A. Church, J. Lloyd, and N. F. Lepora, " Tactile sim-to-real policy transfer via realto-sim
image translation, " in Proc. Conf. Robot Learn., PMLR, 2022, pp. 1645-1654.
[15] V. Nair and G. E. Hinton, " Rectified linear units improve restricted
Boltzmann machines, " in Proc. Int. Conf. Mach. Learn., 2010, pp. 807-814.
[16] M. Jaderberg, K. Simonyan, and A. Zisserman, " Spatial transformer networks, "
in Proc. Adv. Neural Inf. Process. Syst., 2015, vol. 28, pp. 2017-2025.
[17] J. Ho and S. Ermon, " Generative adversarial imitation learning, " in Proc.
Adv. Neural Inf. Process. Syst., 2016, vol. 29, pp. 4572-4580.
[18] R. I. Hartley and P. Sturm, " Triangulation, " Comput. Vision Image
Understanding, vol. 68, no. 2, pp. 146-157, Nov. 1997, doi: 10.1006/cviu.1997.0547.
[19] D. Hershberger, D. Gossow, and J. Faust. RViz, 3D Visualization Tool for
ROS. (2019). Robot Operating System. [Online]. Available: http://wiki.ros.org/rviz
Average Elapsed Time (s)
http://dx.doi.org/10.1126/science.aar6404 http://dx.doi.org/10.1016/j.rcim.2021.102306 http://dx.doi.org/10.1109/TIT.2014.2320500 http://dx.doi.org/10.1126/scirobotics.aau5872 http://dx.doi.org/10.1126/scirobotics.aau5872 http://dx.doi.org/10.1006/cviu.1997.0547 http://wiki.ros.org/rviz
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
