IEEE Robotics & Automation Magazine - June 2023 - 61

directions. The comparisons between
the virtual and real environment and
the images taken by the cameras in the
virtual environment are shown in Figure
3. Each arm of the Baxter has seven
degrees of freedom, and the joint angles
within five frames are also used as
input. The control actions are the
14 joint velocities, which range from
minus one to one, and the two gripper
positions, which range from zero to 255.
Each episode lasts 600 steps at a
control rate of 10 Hz. The episode is
prematurely terminated once the robotic
arm exceeds a reasonable working
area or drops an object, and the DRL
agent receives the corresponding penalty.
When both arms are close to their
respective objects, the agent gets a grasp
approach reward if the current distance
is less than all of the previous values.
When both robotic arms successfully
grasp the corresponding objects, the
agent receives a grasp reward. After
grasping the objects, the agent gets an
assembly approach reward when the
end effectors of the two robotic arms
approach. If the relative position of the
two parts is just right for the assembly,
the agent gets a hover reward. When
the assembly is complete, the agent
gets a finish reward. The specific
parameters of the reward are shown
in Table 1. To intuitively display the
variation of rewards during the task,
a manual control through the keyboard
is conducted; the changes of the
reward functions and the key points of
the task are shown in Figure 4.
The neural network, summarized in
Figure 5, consists of the image observation
encoder, proprioception observation
encoder, actor network, and
critic network. The image observation
encoder inputs raw images from two
cameras and processes them through
ResNet encoders, activation, and flattening.
The proprioception observation
encoder inputs the joint angles of
the robot and processes them through
multilayer perceptron (MLP) with
activation. The two observation encoders
are concatenated and connected to
the actor network and critic network.
The actor network consists of the preTransformer
MLP, Transformer layer,
JUNE 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
61
16 Filters
32 Filters
Left Camera
ReLU,
3 × 3 Conv
512
3 × 3 Conv
Max Pool
16 Filters
128
3 × 3
Conv
Max
Pool
512
128
Right Camera
ReLU,
3 × 3 Conv
512
3 × 3 Conv
Max Pool
16 Filters
128
3 × 3
Conv
Max Pool
512
128
Proprioception
Value Heads
32
16
16
32 Filters
16 Filters
ReLU,
3 × 3 Conv
32
3 × 3
Conv
Max Pool
32 Filters
32
32 Filters
ReLU,
3 × 3 Conv
16
ReLU,
1 × 1
Conv
32 Filters
Flatten
Concatenate
16
32 Filters
Transformer
Q (X,a)
Policy Heads
16
32 Filters
ReLU,
3 × 3 Conv
32
3 × 3
Conv
Max
Pool
32 Filters
ReLU,
3 × 3 Conv
16
ReLU,
1 × 1
Conv
32 Filters
Flatten
32 Filters
ReLU
ReLU
ELU
π(x)
ReLU
ReLU
ReLU
ReLU
Convolutional Layer
Fully Connected Layer
Transformer Layer
80
FIGURE 5. The architecture of the neural network. Each box corresponds to a layer. The size and number of layers are indicated by the numbers in the figure.
128
128
512
128
128
32
32
1,024
32
32
16
16
1,024
16
16
16
16
512
8,192
8,192
1,024
512
512
ReLU
ReLU
512
256
ReLU
ReLU
256
512
ReLU
256
ReLU
ReLU
256
256
ELU
256
ELU

IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023