IEEE Robotics & Automation Magazine - June 2020 - 53

Orange Selector With a Robot Arm
This setup consists of a conveyor belt transporting "pears"
and "oranges," a 3-DoF robot arm located over the belt, and a
red-green-blue (RGB) camera with a view of the belt from
above (Figure 6). The image of the camera does not capture
the robot arm. The robot has to select oranges with the end
effector and avoid pears. The robot does not have any tool,
such as a gripper or vacuum gripper, to pick up the oranges.
Therefore, in this context, we consider a successful selection
of an orange when the end effector intersects the object. The
performance of the learning policy is measured using two
indices: 1) the orange selection success rate and 2) the pear
rejection success rate.
The observations obtained by the camera are from a different region of the conveyor belt than where the robot is
acting. Therefore, observations cannot be used to compensate for the robot position in the current time step; rather,
they are meaningful for future decisions. In other words,
the current action must be based on past observations.
Indeed, the delay between the observations and their influence on the actions is roughly 1.5 s. This delay is given by
the difference between the time when the object leaves the
camera range and the time when it reaches the robot's operating range, which is why this task requires learning temporal features for the policy.
The problem is solved by splitting it into two subtasks that
are trained separately:
1) Orange selection: The robot must intercept the orange coordinate with the end effector exactly when the orange coordinate passes beneath the robot.
2) Pear rejection: The robot must classify between oranges and
pears, so, when a pear is approaching, the end effector should
lift far from the belt plane; otherwise, it should get close.
These two subtasks can be trained sequentially. The
orange selection is initially trained through a procedure
in which there are some oranges being transported at a
fixed position on the belt while some others are placed
randomly. This is to avoid overfitting the policy to specific sequences. When the robot is able to track the oranges
within its reach, the pear rejection learning starts. For
that, pears are placed randomly throughout the sequences of oranges, and the human teacher provides corrections to the robot movement to make the end effector
move away from the pears when they are in the operation
region of the robot.

Figure 7 depicts the average learning curves for this task
after five runs of the teaching process. It is possible to see that
the pear rejection subtask is learned within 20 min with 100%
success, while the orange selection is a harder subtask that
only reaches roughly 80% success after 50 min. Effectively,
combining the two subtasks, the performance of the learned
policies is given only by the success of the orange selection
since the pear rejection was perfectly attained in all runs executed for this experiment.
Conclusions
This article introduced and validated an SRL strategy for
interactively learning policies from human teachers in
environments that are not fully temporally observable.
Results show that, when meaningful spatiotemporal features are extracted, it is possible to teach complex end-toend policies to agents using just occasional relative and
binary corrective signals. Moreover, these policies can be
learned from teachers who are not skilled at executing
the task.
The evaluations with the DAgger approaches and
D-COACH depict the potential of this kind of architecture to
work with different IIL methods, especially those based on
occasional feedback, which are intended to reduce the
human workload. The comparative results between
HG-DAgger and D-COACH with non-expert teachers
showed that, with the former, the policy remains biased,
with mistaken samples even if the teacher makes sure not to
provide more wrong corrections (given that HG-DAgger
works with the assumption of expert demonstrations), thus
making the policy harder to refine. On the other hand,
D-COACH proved to be more robust against mistaken corrections since all non-expert users were able to teach tasks
they were not able to demonstrate.
The previously discussed shortcoming of DAgger algorithms opens possibilities for future works intended to
study how to deal with databases that have mistaken examples. Another field of study is data-efficient movement
generation in animation [19], which, combined with our
method, would make it possible to learn (non)periodic

Selection/
Rejection Success Rate

real systems through two different tasks: 1) a real swing-up
pendulum and 2) a fruit-classifier robot arm. The real
swing-up pendulum is a very complex system for a human
to teleoperate. Its dynamics are faster than that of the OpenAI Gym simulated one used in the previous experiments.
The supplementary material provides more details of this
environment along with the learning curve of the agents
trained by the participants of this validation experiment.
Those results as well as the video show that non-expert
teachers can manage to teach good policies.

1
0.8
0.6
0.4
Orange Selection
Pear Rejection

0.2
0

0
0

2,000 4,000 6,000 8,000 10,00012,00014,000
Time Step
5

10

15

20 25 30 35
Time (min)

40

45

50

Figure 7. The orange selection/pear rejection learning curve.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

53
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
