IEEE Robotics & Automation Magazine - June 2022 - 69

Building Robot Operating System Packages Using Conda
The sidebar of this RoboStack article is geared toward
developers who are concerned with building Robot Operating
System (ROS) packages using Conda, which brings unique
challenges. The key challenge with packaging ROS is that, rather
than building a single package with a handful of dependencies
(the case for most packages on conda-forge), ROS has
thousands of packages with hundreds of dependencies, some
of which are external dependencies (such as PCL and Bullet)
and others which are dependencies on ROS packages. This is
not only challenging because of potentially long build times
if packages are built one after the other, but also because of
application binary interface compatibility. The RoboStack project
made several key advances around package management to
overcome these challenges, all of which are applicable beyond
the use case for ROS. We detail these advances, along with other
information of interest for developers, in the following.
The first key advance is the Vinca tool, which automatically
generates Conda recipes for ROS packages. An advantage
of ROS packages is that dependencies are listed within
package.xml files. Typically, the rosdistro tool handles the
conversion from dependency names within the package.xml
files to package names that can be, e.g., installed using the
apt system package manager on Ubuntu. For RoboStack, a
similar mapping file that maps dependency names to package
names in conda-forge was created. Vinca then combines the
latest package information obtained by the rosdistro Python
application programming interface with this mapping file and a
list of packages that should be built to generate Conda recipes.
As some packages need Conda-specific changes, Vinca also
allows the use of patches that can modify the source code
before building the package. In the future, we aim for tight
integration with the bloom release automation tool that is
used in the ROS ecosystem to create Ubuntu packages.
The second key advance is the introduction of parallelism
in the build pipeline. Vinca has a built-in mechanism to turn
a directory of package recipes into an Azure Build Pipeline
definition. Figure S1 presents the dependency graph up to
the 'ros-core' metapackage. The packages are topographically
sorted, and the pipeline is then split into multiple stages,
where each stage needs to wait for the previous one to finish.
Per each stage, multiple workers run in parallel, with each
handling up to five packages. This makes best use of the 10
free Azure workers that run builds in parallel (each worker
needs to set up the build environment, which takes roughly
2 min-time that is saved when running multiple builds per
worker). The number of packages per stage and Azure workers
are configurable and will scale if more free workers become
available in the future. Once the worker is finished, it uploads
the packages to the RoboStack channel.
The third key advance is Mamba, a fully compatible, fast
drop-in replacement for Conda. The main motivation for Mamba
was to solve the performance issues of Conda, both in terms
of speed and memory usage. Conda's dependency resolver is
relatively slow, in particular for large software repositories (such
as conda-forge) where the entire repository index is parsed
using standard Python JavaScript Object Notation (JSON) parser.
The time taken to solve the dependencies of a new environment
containing ros-noetic-desktop-full is reduced to 15s using
Mamba, from 90s using Conda. Tens of thousands of Python
objects are created (as many as there are objects in the JSON
index file) and processed before being input in the Boolean
satisfiability problem solver. On the other hand, Mamba makes
use of openSUSE's state-of-the-art libsolv dependency resolver,
bypassing most of that processing to resolve dependencies. The
core parts of Mamba are implemented in C++ for maximum
efficiency. Further, Mamba makes use of multithreading to
download repository data and package files in parallel.
The fourth key advance is boa, a fast package builder used
to build Conda packages. Boa is reusing a lot of the condabuild
infrastructure but replaces some parts. Specifically, boa
ros-core
rosconsole-bridge
ros-comm
roswtf
rosnode
rostopic
rosbag
message-filters
topic-tools
rostest
roslaunch
rosparam rosout
roscpp
rosconsole
xmlrpcpp
rosunit
roscpp-core
rosbash
roscpp-serialization
roscpp-traits
rostime
rosboost-cfg
cpp-common
catkin
Figure S1. A dependency graph up to the ros-core metapackage. Each node represents a package, and each edge represents a dependency
upon another package. ros-core depends on more than 175 packages, whereby approximately 100 packages are pulled in from condaforge
(such as Boost and Numpy), and the remaining packages are dependencies on other ROS packages, such as rosbag and rostopic.
JUNE 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
69
rosclean
rosmaster
rosgraph
roslz4 mk
rosbuild
roslib
ros-environment
rospack
rosmake
genpy
cmake-modules
rospy
pluginlib
roslang
roscreate
roslisp
rosgraph-msgs
class-loader
std-srvs
shape-msgs
diagnostic-msgs
std-msgs
message-runtime
message-generation
gencpp geneus genlisp
genmsg
gennodejs
trajectory-msgs
visualization-msgs
geometry-msgs
actionlib-msgs
nav-msgs
rosbag-storage
rosservice
rosmsg
sensor-msgs
stereo-msgs
common-msgs
IEEE Robotics & Automation Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2022
