IEEE Robotics & Automation Magazine - December 2023 - 83

robotic simulation. Then, a simulated robot with an eye-inhand
camera can be used to generate an image dataset of an
object from a predefined set of viewpoints.
In the plant model generation pipeline, as illustrated in
Figure 7, simulated 3D models are generated using real objects.
First, we generate a single realistic model of an object. For example,
a realistic pepper model is generated by applying photogrammetry
onto a set of photographs of a real pepper. Models
of leaves and flowers can be extracted from the photographs
as planar objects. These objects are then provided with a third
dimension using 3D mesh manipulations. Other parts, such as
the stem and plant pot, can be modeled manually and enriched
with realistic textures for better domain adaptation.
These simulated models are then used in the procedural
generation to create an entirely realistic and diverse synthetic
dataset by employing smart augmentation techniques. These
include a pseudorandom choice of plant parts and visualizations,
positional and morphological parameters, and background
images [8]. During the simulation rendering process,
autogenerated labels are produced for object detection, localization,
and semantic segmentation. We filter the pixel-perfect
labels to more closely resemble the level of detail of manually
labeled data [9].
SEMIAUTOMATED ANNOTATION
To overcome the limitations of synthetic data, a semiautomated
annotation method was developed in [6], where a human
operator teaches a robotic system to generate a set of labeled
samples by manually labeling a single sample. The setup consists
of an RGB-D camera mounted in an eye-in-hand configuration,
as in the MS-D system, along with a set of real-world
examples of the object to be detected. The proof of concept
was shown in a strawberry flower detection use case, where a
robot recorded a plant with visible flowers as a detection
target from a set of predefined poses along a precomputed
trajectory. The result of this process is a database of unlabeled
samples containing an RGB image, a depth image,
and the global pose of the camera, thanks to the calibrated
eye-in-hand RGB-D camera.
The collaboration relies on a human operator who annotates
the targeted object (in this case, a flower) by drawing
a minimum bounding rectangle around it in a single RGB
image. The 3D position of the labeled object is obtained by
applying the bounding rectangle to the organized point cloud,
and the corresponding 3D position is located in all the other
recorded samples. Projecting the identified 3D points into
their respective RGB images segments the object of interest
and provides a potentially large set of automatically annotated
samples of an object recorded from different perspectives by
relying on a single manual annotation by a human user. One
of the main benefits of this approach is relying on the same
distribution of input data during the training and deployment
phases. Models trained on higher-quality images tend to fail in
real applications, due to a significant domain gap. Exploiting
the structured and constant environment around the dual-arm
robotic manipulation system, we use the same sensory setup
for both training and validation, resulting in a robust perception
module regardless of hardware limitations.
ROBOT COOPERATION VERSUS MOBILE MANIPULATION
To evaluate the SPECULARIA system, we envision an opposing
robotic setup having a single more complex (and more
expensive) mobile manipulator as a representative of the stateof-the-art
robotized greenhouse approach. The systems are
compared in the same greenhouse, which consists of several
rows of growth tables with bell pepper plants. Special care
was taken to ensure that this layout was fair in the sense that
both systems could reach and handle all the plants. For the
multirobot system, two workstations are provided on the sides
of the manipulator, where two mobile robots can operate. The
unmanned ground vehicles (UGVs) bring the plants to the
workstations of the static robot manipulator arm. The underlying
assumption is that the manipulator can treat only the front
half of a plant. To reach behind the plant, the manipulator
requires cooperation from a UGV to rotate the plant. The
opposing system, which relies on the mobile manipulator,
must therefore go around to reach the other side of the plant.
Consequently, the proposed greenhouse is spread out so that
the mobile manipulator can reach both sides of the plants.
To enable the heterogeneous robot teams to work together,
we deploy the developed mission planning and coordination
Synthetic
Pipeline
Photos - Real World Collaborative
AI
3D Models
Labeled
Detection Dataset
Rendered
Image
Semantic
Masks
Detection
Boxes
FIGURE 7. Pipelines developed for training dataset generation in
SPECULARIA activities. The synthetic approach relies on Meshroom
for photogrammetric 3D model generation. Blender is used as
the backbone of the rendering and labeling procedure. Examples
of synthetic data for deep learning model training used and publicly
released under the SPECULARIA project are shown as results of
the synthetic pipeline. Semiautomated dataset
labeling on realworld
images is designed as a collaborative procedure, where a
human operator and a cobot equipped with an RGB-D camera cooperate.
Examples of the dataset generated through a semiautomated
labeling pipeline, along with automatically generated labels,
are shown as results of the collaborative pipeline.
DECEMBER 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
83
IEEE Robotics & Automation Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2023
