IEEE Robotics & Automation Magazine - June 2023 - 99

in charge of one team, checking the progress of each stage and
scoring the parts-fastening achievements. For the completeness
parameter, the judges evaluated the assembly capability
of the robot. If a defect occurred during assembly, the operator
intervened and corrected it. The completeness score was subtracted
in proportion to the number and duration of operator
interventions. The speed score was evaluated for the assembly
time, and the speed score was subtracted over time from
45 min from the start of assembly.
COMPETITION RESULTS
Our team (Team
SK )Y2
was awarded first place to complete
both missions with the least number of interventions among the
four participating teams. The competition consisted of mission 1,
in which the chair was to be assembled following the original
assembly instructions, and mission 2, in which the chair was to
be assembled by following the modified assembly instructions.
The competition results are shown in Figure 2(b) and (c). In the
first mission, all processes, except for the last short screw bolting,
were performed without operator intervention, and the
assembly was completed in 38 min. In the second mission, an
additional intervention was required owing to a failure to recognize
the position when fastening the screw bolts. The second
mission was completed in 52 min.
AI-BASED ALGORITHM FOR READING AND
COMMANDING ASSEMBLY INSTRUCTIONS
In this section, we introduce an algorithm that converts the
IKEA Stefan chair assembly instructions for humans into
assembly commands for robots. The IKEA instructions
describe the connections between the frame parts by using
visual elements, such as part diagrams, arrows, and highlights.
These elements do not provide numerical data. Therefore,
it is necessary to convert the visual information into
instructions for robots.
We generate assembly commands for robots using AI-based
algorithms. We have divided the process into three stages: recognition,
analysis, and application. In the recognition step, the
visual instructions in the 2D assembly manual are extracted by
describing information, such as the poses of the parts and the
relationships between parts. In the analysis step, a connection
relationship tree is generated using the visual information on
each instruction page. This tree contains the assembly information
as a hierarchy of parts. Finally, in the application step,
a planning domain definition language (PDDL) is generated
using a connection relationship tree. Based on the hierarchy
information presented in the connection relationship tree, the
PDDL expresses an assembly process, such as the order of
assembly of parts, connections of connectors, and assembly
method between parts.
RECOGNITION OF ASSEMBLY INSTRUCTIONS
In this section, we introduce a recognition framework that
can read and understand the required assembly actions
from the assembly instruction. As illustrated in Figure
3(a), we split the assembly instruction into three individual
functional regions: details, warning, and assembly
regions. The details region contains items and explanatory
text. The warning region helps with the selection of the
correct parts when similar parts exist. The assembly
region defines the assembly actions required for the current
frame.
To understand an image, the proposed recognition framework
is divided into two tasks: 1) the detection task, to detect
components and object parts and the number of components
and parts; and 2) the action estimation task, to estimate
assembly actions among components and identify the parts
required for the current frame. To achieve the detection task,
we combined the object detection, text recognition, and detection
correction approaches.
Team 1
Team 2
Team 4
Mission 1
Time:
38 min
Failure:
Once Occurred
Team 3
Team 4
(Team SK2Y)
(b)
Team 4
Mission 2
Time:
52 min
Failure:
Twice Occurred
(a)
(c)
FIGURE 2. (a) A screen capture of live broadcast of the competition (Team SK2Y is indicated by the red boxes). (b) The result of the first
mission for Team SK2Y. Assembly was completed in 38 min without human intervention. (c) The result of the second mission for Team
SK2Y. Assembly was completed in 52 min with only one intervention.
JUNE 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
99

IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023