IEEE Robotics & Automation Magazine - December 2022 - 154

data are then reshaped and stacked to
create an 80 × 80 lidar map.
3) Goal location: To inform the robot of
where to go, the final goal point and
its corresponding nominal path are
fed into the pure pursuit algorithm to
extract the subgoal point, which is
fed into the DRL-VO network.
This novel preprocessed data representation
is one key idea of the DRL-VO
control policy, allowing it to bridge the
simulated-to-real gap and generalize to
new scenarios better than other end-toend
policies.
DRL Network Module
The DRL-VO control policy uses an
early fusion network architecture to
combine the pedestrian and lidar data
at the input layer to obtain high-level
abstract feature maps. This early
fusion architecture facilitates the
design of small networks with fewer
parameters than late fusion works,
which is the key to deploying them on
resource-constrained robots. These
high-level feature maps are combined
with the subgoal point and fed into the
actor and critic networks to generate
suitable control inputs and a state
value, respectively.
Training
To ensure that the DRL-VO policy leads
the robot to navigate safely and efficiently,
the team at Temple developed a
new multiobjective reward function
that rewards travel toward the goal,
avoiding collisions with static objects,
having a smooth path, and actively
avoiding future collisions with pedestrians.
This final term, which utilizes the
concept of velocity obstacles, is key to
the success of the DRL-VO control policy.
With this reward function, the
DRL-VO policy is trained via the proximal
policy optimization algorithm in a
3D lobby Gazebo simulation environment
with 34 moving pedestrians, using
a Turtlebot2 robot with a ZED camera
and a 2D Hokuyo lidar.
Deployment
The Temple team directly deployed the
DRL-VO policy trained on a Turtlebot2 in
the BARN Challenge without any model
fine-tuning. To achieve this, the team had
to account for three key differences:
1) Unknown map: During development,
the DRL-VO policy used a
known occupancy grid map of the
static environment for localization,
which was not available in the
BARN challenge. To account for
this, the localization module
(amcl) was removed from the software
stack and replaced with a laser
odometry module.
2) Static environment: The DRL-VO
policy was designed to function in
dynamic environments. To account
for the fact that the environments in
the BARN Challenge were all static
and highly constrained, the pedestrian
map was set to all zeros.
3) Different robot model: The DRL-VO
policy was trained on a Turtlebot2,
which has a different maximum speed
and footprint compared to the Jackal
platform. In the BARN Challenge, the
maximum speed of the robot was
modified based on the robot's proximity
to obstacles. This led to the robot
moving slowly (0.5 m/s, the same
speed as the Turtlebot2) when near
obstacles and quickly (up to 2 m/s, the
maximum speed of the Jackal) when
in an open area.
Discussions
Based on each team's approach and the
navigation performance observed during
the competition, we now discuss lessons
learned from the BARN Challenge
and point out promising future research
directions to push the boundaries of
efficient mobile robot navigation in
highly constrained spaces.
Generalizability of LearningBased
Systems
One surprising discrepancy between
the simulation qualifier and the physical
finals is the contrasting performance of
the DRL-VO approach by Temple University,
which outperformed all baselines
and other participants in simulation but
suffers from frequent collisions with
obstacles in the real world. Despite the
fact that the organizers modified the
rules during the physical finals to allow
the teams to make last-minute modifica154
* IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
tions to their navigation systems, DRLVO
still did not perform well in all three
physical obstacle courses. The TRAIL
team believes this was due to two types
of gaps between the simulator and the
real world: 1) the real-world environments
were all highly constrained,
whereas the simulator environments
contained both constrained and unconstrained
environments and 2) the DRLVO
policy was learned on a Turtlebot2
model (which has a smaller physical
footprint than a Jackal). Most of the collisions
during the hardware tests were
light grazes on the side, so a robot with a
smaller size may have remained collision-free.
The
stark performance contrast
between simulation and the real world
suggests a generalizability gap for the
reinforcement learning approach. It
was not practical for the team to retrain
a new system on site during the competition,
given the impractically massive
amount of training data required for
reinforcement
learning. How to
address this generalizability gap and
make a navigation policy trained in
simulation generalizable to the real
world and different robot/sensor configurations
remains to be investigated,
even for such a simple static obstacle
avoidance problem.
Another potential way to address
such inevitable generalizability gaps is
to seek help from a secondary classical
planner that identifies out-of-distribution
scenarios in the real world and
recovers from them through rule-based
heuristics. In fact, for the last two physical
courses, the Temple team tried to
implement just such a " recovery planner "
as a backup for DRL-VO: when
a potential collision is detected (i.e.,
the robot faces an obstacle that is too
close), the robot rotates in place to head
toward an empty space in an attempt to
better match the real-world distribution
to that in the simulation during training.
Although such a recovery planner
did help in some scenarios, it was difficult
for it to cover every difficult scenario
and navigate through the entire
obstacle course.
Indeed, the Temple team spent time
during the 30-min timed sessions to
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
