IEEE Robotics & Automation Magazine - September 2015 - 65

collected in the mentioned experiments stress the lack of equity in experimental conditions. These issues affect the reliability of the trials and constitute a problem that should be
overcome by establishing GEM and practices. However, even
with a standard procedure available, within a marine context
(but also generally in robotics), replicability cannot be guaranteed. Therefore, the need is still present for a smart tool
in charge of suitably and actively conducting experiments to
be developed.
Experimental Design
A standard guideline for the design and execution of path-following tasks and some assumptions are defined in the following.
● An experiment (or batch) is formed by n runs, corresponding to n possibly different paths that are executed sequentially in time. Ideally, runs are executed in such a way
that they are independent, but this needs to be verified
through extensive simulation (see the "Early Simulation
Results With DeepRuler" section).
● Each run is divided into four sequential phases: 1) approach, 2) forward path, 3) turn, and 4) backward path.
● A rectangular area R 1 # R 2 for executing the experiments
is defined by the human operator. This area is divided into
three subareas: a middle area L # R 2 for executing the path
back and forth (at least one repetition) and two lateral areas
for the execution of the approach and turn phases. A suitable maneuver for the approach and turn phases is defined
by: 1) a straight line connecting the end of the currently
followed path with 2) a semicircumference of radius r, providing the reference for the turn phase, and 3) a straight
line of length l, connecting the semicircumference to the
target path reference to be followed. The latter is responsible for achieving repeatability. Its direction is chosen to
head the vehicle on the tangent to the target path in its
starting point (see Figure 1).
● Performance is measured only while executing the actual
path (during neither approach nor turn phases).
The human operator is required to set values for R 1, R 2,
L, r, and l, paying attention to maneuverability restrictions
and ensuring that l is long enough for the vehicle to enter the
target path at its tangent and r is big enough for the vehicle
to be able to follow the semicircumference (i.e., r is equal to
or greater than the minimum turning radius of the vehicle).
For the moment, the n target paths in a batch are chosen
for a small dimensional parametric class of functions, such as
sinusoids, circles, and ellipses. Generalization to other classes
of target functions is being considered.
Proposed Methodology and System Development
For the abovementioned reasons, an appropriate software
framework named DeepRuler (design execute and evaluate
path-following for robotic unmanned vehicles and experiment replicability) has been designed and developed. This
framework allows one to easily design experiments and to automatically execute them, both running a simulator or driving
a real robot. Details about DeepRuler are provided in the

"The Automated Approach" section and the "Software Design
and Features" section, while the "Early Simulation Results
With DeepRuler" section reports preliminary results obtained
employing a simulated robot.
The Automated Approach
The best way to achieve a good level of reproducibility and repeatability of experiments is to completely automate the process of generation and execution of the experiment.
To this aim, the DeepRuler framework has been designed
with a threefold objective. The first (design) is to guide the
user through the process of the definition of the path-following experiment. The second (execute) is to conduct and supervise the execution of the experiment itself. The third
(evaluate) is to automatically measure, directly on the field,
the robot performance through predefined or custom metrics. DeepRuler is then responsible for the following.
● Helping the User in the Definition of the Experiment: In an
offline phase, the experiment definition is carried out thanks
to a step-by-step windowed configurator that guides the user
through the setup process. The user is asked to input the parameters that define the experiment, such as the working
area, the shape of the paths, the parameters of each path, the
structure of the telemetry, the metrics to be computed, and
so on. The configurator generates a file describing the modality of the experiment that can be reused each time that an
experiment has to be reproduced exactly in the same way. In
addition, during this phase the user can customize the
framework in a manner that will be described in the
"Software Design and Features" section.
● Controlling the Robot; Sequencing the Various Experiment
Phases: DeepRuler does not directly drive the robot through
motion commands, rather it acts as a finite-state machine
(FSM) providing high-level commands to the robot controller. In the current release, DeepRuler employs only two
commands: 1) a GOTO command, used only once at the
beginning of the experiment to let the robot reach the initial
position and orientation and 2) a NEWPATH command
that provides the points of a path, used each time that the
robot has to perform a new run phase (approach, forward,
turn, and backward).
● Collecting the Telemetry Coming From the Robot: During
the execution of the experiment, the robot continuously
sends its telemetry to the framework which in turn collects
it, according to a predefined policy, for subsequent processing. The variables to be collected are defined during the
configuration of the experiment and they mandatorily include information about the position of the robot and all
the variables needed to compute the metrics.
● Computing the Metrics at the End of Each Run: The collected data will be used at the end of each run to compute the
metrics selected during the configuration.
The experiment execution can be monitored by the user
thanks to a human computer interface (Figure 1) which reports all of the relevant data and shows a real-time plot of the
position (actual and historical) of the robot.
September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015