IEEE Robotics & Automation Magazine - June 2023 - 62

post-Transformer MLP, and output distribution
layer. The critic network contains
only fully connected structures. The actor
and critic networks share the mutual architecture
for their observation networks, but
their parameters are not shared. The output
layer uses the exponential linear unit
(LU) activation function, and the other
layers use the rectified LU (ReLU) activation
function [15].
A Transformer that has achieved
"
IN THIS ARTICLE, MULTIHEAD
SELF-ATTENTION
MODULES ARE APPLIED
FOR EVERY TIME STEP.
„
breakthrough results in natural language
processing, image processing, and other fields can take full
advantage of deep learning, learning key features between data
and inferring in previously unseen environments [16]. In this
article, multihead self-attention modules are applied for every
time step. At this stage, we use the PPO algorithm, a state-ofthe-art
policy gradient DRL algorithm, to train the agent in the
simulation environment. In the process of environmental interaction,
the agent continuously updates the network parameters
to improve performance via
k =argmaxsa+, Er Ls a [( ,, k,)]
iii
+1
i
ik
(1)
where ki and k 1i + are new and old network parameters. The
loss function ()L i is shown in the following equation:
LE min ,, ,
where Ett
^hh ^ ^ h
CLIPii if f=- +
t
tt ttrA rAclip 11 h h@ (2)
6
^ ^
t
t t
is the empirical expectation over the time steps. rt is
the ratio of the probability under the current and updated policies.
f is a hyperparameter that limits the magnitude of parameter
updates. To simplify computation and converge faster, in
practice, the loss function simplifies to a simplified version
Ls a ii =minc
k
h
where
gA,e =)
^
h
^
^
1
1
+
-
e
e
h
h
A
A
A
A
$
1 0
.
(4)
Table 2 shows the RL settings, such as the PPO-clip hyperparameter
and task reward weights.
SIM-TO-REAL ADAPTATION BY IL
Due to the discrepancy between simulated and real environments,
agents trained in simulated environments often perform
poorly in the real world even if the input data structure is the
same. Since the acquisition of experimental data in the real
world will cause wear and tear of the robotic arm and waste a
lot of time, sim-to-real transfer has become the mainstream
method. If the agent is directly trained in a real environment,
the neural network needs to randomly initialize the parameters.
But the initial agent is so weak that it is difficult to obtain
an initial reward in the environment, especially in sparse
62 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
^ ,, ,, , ,,^e
i ^
r
r
i rrh kk
ii^
^
k
as
as
;
;
h
As ag As a
^ hhhm
(3)
reward and complex robot control tasks.
A supervised behavioral clone method
using trajectories in the simulated datasets
to train the agent was attempted, but this
suffers from a covariate shift, in which
case the input states are distributed differently
under the two datasets and cause
failure. Therefore, a high-efficiency solution
is to obtain a pretrained neural network
with the simulation of regular
properties through IL to make full use of
environmental space exploration.
In previous work, the method of obtaining a policy from
the simulation data generated by experts is usually IRL, in
which the cost functions of the expert data are first learned,
and then, various DRL methods are used to learn the expert
strategy. However, IRL is usually inefficient because even if
the cost function is well learned, the DRL approach still needs
to be used to train the policy. In this article, a modified GAIL
algorithm is proposed to learn policies directly from expert
data [17]. The performances of these algorithms are compared
in the " Experiments " section.
In the original IRL process, the goal is to find a cost function
that makes the expert outperform all other policies whose
cost function is regularized by }
IRL () `jargmax cH cs a
min
}
r} r=- +- +
-
E
c RSA
E
!
#
E [( ,)]
r cs a
r sa =
c
cr;=
()
r!P
()
E [( ,)]
r
(5)
where !r P is a policy whose probability of occurrence of
state action pair is measured as
t P(),sst
tr ; R3
expressed as Ec
[(sa , )]
(, )( )as t 0
=
and the discount expectation can be
rr=Rsa, t sa sa
} ^ h
GA c ='
E 6 ^^ ,
+3
E
r gcsahh@
if 1c 0
otherwise
.
(6)
Sequentially, a generative model G and a discriminative
classifier D are synchronously trained. The goal of the discriminative
model is to determine as accurately as possible
whether a sample is from real data or generated by a generative
network. The goal of the generative network is to make the discriminative
network as indistinguishable as possible from the
source of the samples. The two networks with opposite goals
are continuously trained alternately. When it finally converges,
if the discriminant network can no longer determine the source
of a sample, it is equivalent to the generation network that can
generate samples that conform to the real data distribution. In
this article, the simulation occupancy measure str
corresponds
to the raw data distribution, and the real-world occupancy measure
t
rr
corresponds to the data distribution, which is generated
by G. A min-max objective function like the GA network
is selected in this work.
i } EE 1^^ ^hhh
min ,, .
maxlogDsaD sa
r} r}i
+log
E66@@
(7)
(, )( ,). Different
} ()c values represent different IL algorithms. In this article,
} ()c is defined as
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
