IEEE Robotics & Automation Magazine - June 2022 - 37

The Proposed Robotic System
image in the previous layer of I ,C and I L5
five layers ago of I .C
System Overview
As depicted in Figure 3(a)-(e), the developed robotic system
consists of three parts: the robotic system, three FFF 3D printers,
and a removal area. First, a UR10e (Universal Robots,
Denmark) is used as our robotic manipulator, with a two-jaw
parallel gripper (GEP2016IO-05-A, Zimmer Group, Germany).
Second, three FFF 3D printers (Original Prusa I3 MK3S,
Prusa Research, Czech) are installed, with each having one
Raspberry Pi (RPi) 4 Model B. Each RPi is connected with a
8MP RPi camera (Camera Module v2). In each printer, there
is a flexible and removable heatbed steel sheet that is magnetically
connected to the print bed. The Raspberry Pi is communicated
with the host controller (an Ubuntu 16.04 computer)
via Wi-Fi to enable remote access. OctoPrint (https://
octoprint.org/; https://github.com/OctoPrint/OctoPrint) is
used in each RPi to control the printers with Octoclient
(https://github.com/hroncok/octoclient), while Octolapse
(https://github.com/FormerLurker/Octolapse) is utilized to
make stabilized, time-lapse images of our print. In addition,
two customized mechanical guides are mounted with each
printer to ensure the correct replacement of the heatbed steel
sheet. Third, a removal area is designed to remove the prints
from the heatbed steel sheet. More details will be discussed in
future sections of this article.
DDR
The DRR algorithm comprises three components: 1) failure
detection, 2) robotic manipulation for removal of the print,
and 3) robust replacement of the heatbed steel sheet. The
entire flow is presented in Figure 4(a)-(c).
Detect: Failure Detection
The hybrid IC and DL vision (HIDV) algorithm is proposed
to detect the failures. In this article, we focus on the detection
on three common types of failures, namely, spaghetti,
detachment, and air printing. More specifically, an IC [16],
[17] is employed to handle detachment and air printing. In
addition to an IC, the YOLO-based TSD [10], [15] is used
specifically for spaghetti. The pseudocodes of the HIDV
algorithm are shown in " Algorithm 1: Hybrid IC and DL
Vision Algorithm. "
Image Comparison to Detect
Detachment and Air Printing
First, the proposed IC algorithm is inspired by [16] and [17].
The flow of the IC algorithm is shown in Figure 5. During the
printing, an image frame is taken for each layer of the print. In
each step of the calculation in the IC algorithm, there are four
images being considered. The four images are
I ,BG
I ,C I ,P and
I ,L5 which denote the background image, current image,
image from the previous layer, and image from five layers ago,
respectively. At the begging of the print, an image is taken as
the background image. In Figure 5, the detachment is illustrated
as the example for the current image
I .C
IP is the
is the image from
There are four procedures in each step of calculation in an
IC, namely, gray scale, absolute difference, threshold, and normalized
root mean-square error (NRMSE). More specifically,
the first procedure converts color images into gray scale. In
the second procedure, given two images, absolute difference
aims to detect the motion of the objects [19]. Assume Ik
the value of the kth frame in the image sequences, and the
absolute differential image is defined as [19]
II .
dkkk k11(, =-I
-I
,BG
I ,C
I ,P and I L5
) ;;
represent I ,0
I ,k
(1)
In this example, assume the current frame is the kth frame,
hence,
Ik 1- , and Ik 5I
,C
I ,P and )I L5
and the background image (),IBG
I (, )tC BG
,
respectively. In the second procedure, the absolute difference
is employed to obtain the difference among the three images
(i.e.,
which
are (HTML translation failed) and I (, )dL BG5
. The three absolute
differential images are binarized, using a binary threshold
in the third procedure, and got binary images
and I (, )tL BG5
, I (, )tP BG
k 1- NRMSE(,
,
RMSE(,
kk 1)
kk )1-
==
1
/ k
N i
N
1
Algorithm 1: Hybrid IC and DL Vision Algorithm
Given k as number of layers with image Ik
Obtain the image in the current layer II
Ck
!
if k 0= then
Register I0
else
Save images Ik
Obtain ,I L5
CL
IP and IC
Calculate (, )CL5
tions using IC [16], [17]
if
CL
UU2 (, )( ,) UU2 P max
The Detachment occurs
CP
CP
else
Check the printing conditions using DL
Import TSD [10], [15] model MTSD
if detection 02 using TTSD
The Spaghetti occurs
then
end if
end if
end if
if Spaghetti, Detachment or Air-Printing occur then
Stop printing.
else
Activate robot to execute Remove and Replace
Continue printing.
end if
Execute the next layer if no errors occur.
JUNE 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
37
U and U then check the printing condi55L
max and (, )( ,)
(, ,)CP
then
else ifUU1 (, )( ,) and (, )( ,)
The Air-Printing occurs
55L min UU1 P min
then
as the background image ,IBG
else if k 5# then
Save images Ik
II0
BG
!
is defined [16], [20] as
(()( ))
k-1
,
. Finally, NRMSE is used to compare the structures
of two chosen images. Assuming the two images are k
and
Ii Ii 2
(2)
to be
https://octoprint.org/ https://octoprint.org/ https://www.github.com/OctoPrint/OctoPrint https://www.github.com/hroncok/octoclient https://www.github.com/FormerLurker/Octolapse
IEEE Robotics & Automation Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2022
