IEEE Robotics & Automation Magazine - June 2023 - 100

We adopted Faster-RCNN of the Detectron2 framework
[6] to locate and label all items, including speech balloons,
text, and cross signs [7]. The training and testing data sets
were generated as follows. 1) We collected text and signs,
diagrams, speech balloons, and cross signs from IKEA
product manuals available on IKEA's home page and augmented
them through rotation, scaling, and flipping. 2) We
rendered and automatically generated Stefan's parts and
their labels from the 3D model created using Blender [8].
Consequently, to distinguish the items in the same category
but having different serial numbers, we employed MORAN
text recognition [9] and adopted the checkpoints trained by
the authors to convert text regions of interest into item serials.
To correct the part labels, we examined the item shapes
in the warning region.
For the action estimation task, poses of the parts in the
assembly regions are required and should be estimated. We
used several keypoint-based pose estimation methods [10], [11],
only to realize that these methods failed to recognize low-texture
objects, such as drawing objects in IKEA's manuals. The
pixel-wise information in the IKEA manual is inadequate
to recognize poses. To solve this issue, we have proposed a
novel partition-based pose estimator to predict the poses of
parts. It is possible to use structural information of the object
rather than pixel-wise information because divided parts contain
structural features, such as holes, outlines, and angular
shapes, rather than keypoints of pixels. We detected several
object partitions, which were easier to recognize compared to
the corresponding keypoints. Figure 3(b) and (c) compares the
results obtained using the conventional pose estimation method
and the proposed novel partition-based pose estimation
method. From the estimated poses, the locations of connectors
on the parts could be calculated. In the last step, we examined
all connector pairs and selected the pair with the minimum
distance between connectors.
CONNECTION RELATIONSHIP TREE INFERENCE
In the analysis step, we combined the relative connections of
the recognition step into one object. For efficient inference,
we proposed a connection relationship tree by using the recognition
results. The visual elements on each page explain
how to assemble the frame parts hierarchically. The tree
structure is appropriate for expressing hierarchical characteristics
because it is a set of nodes connected by parent-
child relationships without cycles. These tree properties are
advantageous for analyzing the instructions of assembly
Details
Warning
Assembly
Keypoint-Based Pose
Estimation Result
(b)
Partition-Based Pose
Estimation Result
(a)
(c)
FIGURE 3. (a) Recognition result of assembly instructions. We extracted the visual information of the assembly instructions using the
Faster-RCNN and the MORAN recognition technology. In addition, we utilized a novel pattern-based pose estimator that predicts the
pose of parts for the action estimation task. (b) and (c) Comparison results of pose estimators. (b) The conventional keypoint-based pose
estimator does not recognize the posture of the left frame, but (c) the proposed estimator based on the partition correctly recognizes the
posture of the left frame.
100
IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
101350

IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023