IEEE Computational Intelligence Magazine - August 2019 - 19

Distributed Fusion-Based
Policy Search for Fast Robot
Locomotion Learning
©ISTOCKPHOTO.COM/FARBENTEK

Zhengcai Cao and Qing Xiao
College of Information Science
and Technology, Beijing University
of Chemical Technology,
Beijing, CHINA

Mengchu Zhou
Department of Electrical and
Computer Engineering,
New Jersey Institute of Technology,
Newark, USA

Digital Object Identifier 10.1109/MCI.2019.2919364
Date of publication: 17 July 2019

1556-603X/19©2019IEEE

Abstract-Deep reinforcement learning methods are developed to deal with challenging
locomotion control problems in a robotics domain and can achieve significant performance improvement over conventional control methods. One of their appealing advantages is model-free. In other words, agents learn a control policy completely from scratches
with raw high-dimensional sensory observations. However, they often suffer from poor
sample-efficiency and instability issues, which make them inapplicable to many engineering systems. This paper presents a distributed fusion-based policy search framework to
accelerate robot locomotion learning processes through variance reduction and asynchronous exploration approaches. An adaptive fusion-based variance reduction technique is
introduced to improve sample-efficiency. A parametric noise is added to neural network
weights, which leads to efficient exploration and ensures consistency in actions. Subsequently, the fusion-based policy gradient estimator is extended to a distributed decoupled
actor-critic architecture. This allows the central estimator to handle off-policy data from
different actors asynchronously, which fully utilizes CPUs and GPUs to maximize data
throughput. The aim of this work is to improve sample-efficiency and convergence speed
of deep reinforcement learning in robot locomotion tasks. Simulation results are presented
Corresponding Author: Zhengcai Cao (e-mail: caozc@nsfc.gov.cn)
and Mengchu Zhou (email: zhou@njit.edu)

AUGUST 2019 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

19



http://www.ISTOCKPHOTO.COM/FARBENTEK

IEEE Computational Intelligence Magazine - August 2019

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2019

Contents
IEEE Computational Intelligence Magazine - August 2019 - Cover1
IEEE Computational Intelligence Magazine - August 2019 - Cover2
IEEE Computational Intelligence Magazine - August 2019 - Contents
IEEE Computational Intelligence Magazine - August 2019 - 2
IEEE Computational Intelligence Magazine - August 2019 - 3
IEEE Computational Intelligence Magazine - August 2019 - 4
IEEE Computational Intelligence Magazine - August 2019 - 5
IEEE Computational Intelligence Magazine - August 2019 - 6
IEEE Computational Intelligence Magazine - August 2019 - 7
IEEE Computational Intelligence Magazine - August 2019 - 8
IEEE Computational Intelligence Magazine - August 2019 - 9
IEEE Computational Intelligence Magazine - August 2019 - 10
IEEE Computational Intelligence Magazine - August 2019 - 11
IEEE Computational Intelligence Magazine - August 2019 - 12
IEEE Computational Intelligence Magazine - August 2019 - 13
IEEE Computational Intelligence Magazine - August 2019 - 14
IEEE Computational Intelligence Magazine - August 2019 - 15
IEEE Computational Intelligence Magazine - August 2019 - 16
IEEE Computational Intelligence Magazine - August 2019 - 17
IEEE Computational Intelligence Magazine - August 2019 - 18
IEEE Computational Intelligence Magazine - August 2019 - 19
IEEE Computational Intelligence Magazine - August 2019 - 20
IEEE Computational Intelligence Magazine - August 2019 - 21
IEEE Computational Intelligence Magazine - August 2019 - 22
IEEE Computational Intelligence Magazine - August 2019 - 23
IEEE Computational Intelligence Magazine - August 2019 - 24
IEEE Computational Intelligence Magazine - August 2019 - 25
IEEE Computational Intelligence Magazine - August 2019 - 26
IEEE Computational Intelligence Magazine - August 2019 - 27
IEEE Computational Intelligence Magazine - August 2019 - 28
IEEE Computational Intelligence Magazine - August 2019 - 29
IEEE Computational Intelligence Magazine - August 2019 - 30
IEEE Computational Intelligence Magazine - August 2019 - 31
IEEE Computational Intelligence Magazine - August 2019 - 32
IEEE Computational Intelligence Magazine - August 2019 - 33
IEEE Computational Intelligence Magazine - August 2019 - 34
IEEE Computational Intelligence Magazine - August 2019 - 35
IEEE Computational Intelligence Magazine - August 2019 - 36
IEEE Computational Intelligence Magazine - August 2019 - 37
IEEE Computational Intelligence Magazine - August 2019 - 38
IEEE Computational Intelligence Magazine - August 2019 - 39
IEEE Computational Intelligence Magazine - August 2019 - 40
IEEE Computational Intelligence Magazine - August 2019 - 41
IEEE Computational Intelligence Magazine - August 2019 - 42
IEEE Computational Intelligence Magazine - August 2019 - 43
IEEE Computational Intelligence Magazine - August 2019 - 44
IEEE Computational Intelligence Magazine - August 2019 - 45
IEEE Computational Intelligence Magazine - August 2019 - 46
IEEE Computational Intelligence Magazine - August 2019 - 47
IEEE Computational Intelligence Magazine - August 2019 - 48
IEEE Computational Intelligence Magazine - August 2019 - 49
IEEE Computational Intelligence Magazine - August 2019 - 50
IEEE Computational Intelligence Magazine - August 2019 - 51
IEEE Computational Intelligence Magazine - August 2019 - 52
IEEE Computational Intelligence Magazine - August 2019 - 53
IEEE Computational Intelligence Magazine - August 2019 - 54
IEEE Computational Intelligence Magazine - August 2019 - 55
IEEE Computational Intelligence Magazine - August 2019 - 56
IEEE Computational Intelligence Magazine - August 2019 - 57
IEEE Computational Intelligence Magazine - August 2019 - 58
IEEE Computational Intelligence Magazine - August 2019 - 59
IEEE Computational Intelligence Magazine - August 2019 - 60
IEEE Computational Intelligence Magazine - August 2019 - 61
IEEE Computational Intelligence Magazine - August 2019 - 62
IEEE Computational Intelligence Magazine - August 2019 - 63
IEEE Computational Intelligence Magazine - August 2019 - 64
IEEE Computational Intelligence Magazine - August 2019 - 65
IEEE Computational Intelligence Magazine - August 2019 - 66
IEEE Computational Intelligence Magazine - August 2019 - 67
IEEE Computational Intelligence Magazine - August 2019 - 68
IEEE Computational Intelligence Magazine - August 2019 - 69
IEEE Computational Intelligence Magazine - August 2019 - 70
IEEE Computational Intelligence Magazine - August 2019 - 71
IEEE Computational Intelligence Magazine - August 2019 - 72
IEEE Computational Intelligence Magazine - August 2019 - 73
IEEE Computational Intelligence Magazine - August 2019 - 74
IEEE Computational Intelligence Magazine - August 2019 - 75
IEEE Computational Intelligence Magazine - August 2019 - 76
IEEE Computational Intelligence Magazine - August 2019 - Cover3
IEEE Computational Intelligence Magazine - August 2019 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com