IEEE Spectrum July, 2017 - 18

RESOURCES_hANDS ON

JUL 2017

NORTh AMERICAN

makes it much easier to use, especially after looking at Jeremy Howard and Rachel
Thomas's excellent starter course on machine learning.
I started building my neural net system in
earnest. Within several days I had the sorter
working and handling more than a few
classes of parts for the first time.
The next step was to get a training set for
my neural network that was large enough to
make working with over 1,000 types of Lego
pieces possible. At first this seemed like an
insurmountable problem: I could not figure
out how to photograph and label enough
sample parts by hand. Even the most optimistic calculations had me working for six
months or longer flat out.
Then I realized I didn't need to make the
training set all by myself. The machine takes
and labels images. All I need to do is spot
the ones where the computer was wrong
and relabel the image correctly. As the neural net learns, there are fewer mistakes, and
the labeling workload decreases.
The first day I managed to label a starter set
of about 500 assorted scanned pieces. Using those parts to train the net, the next day
the machine sorted 2,000 more parts. About
half of those were wrongly labeled, which I
corrected. The resulting 2,500 parts were
the basis for the next round of training. Another 4,000 parts went through the machine,
90 percent of which were labeled correctly!
So, I had to correct only some 400 parts. By
the end of two weeks I had a training data set
of 20,000 correctly labeled images.
Some classes are still underrepresented in the training set, so I need to increase
the number of images for those. I'll probably
just run through a single sample batch consisting of nothing but those parts. Once the
software is able to reliably classify across
the entire range of parts in my garage, I'll be
pushing through the remainder of those two
tons of bricks. And then I can finally start selling off the results! -JacquEs MatthEIJ
LEGo EX MachIna: A slow conveyer belt
pulls pieces from a hopper [top]. A camera
captures images of individual pieces [second
from top] and a neural net identifies them
[second from bottom]. Air puffs from valves
[bottom] knock sorted pieces into bins.

SPECTRUM.IEEE.ORG

An extended account of the sorter
construction is available on Mattheij's
blog at Jacquesmattheij.com.
↗ Post your coMMEnts at http://spectrum.ieee.org/
lego0717

Jacques mattheiJ (4)

hopper before dropping them onto a much
faster transport belt. This belt moves parts
past a $30 magnifying camera connected
via USB to a PC for identification.
Once identified, the parts have to be moved
from the transport belt to the correct bin. After
some experimenting I settled on putting air
nozzles next to the belt. A well-timed puff
knocks the part into the desired bin.
The biggest challenge of course was identification. First, I had to capture a good image of each piece. There were all kinds of
gotchas here. For instance, parts may be
longer than a single image frame, parts can
be a color that is extremely close to that of the
background, and so on. It was several weeks
before I could reliably stitch and crop scans
automatically, so that I was generating images containing one complete piece of Lego.
Then came the really hard part: identifying
each piece. Lego bricks come in thousands
of distinct shapes and over 100 colors (you
can roughly tell how old someone is by asking them what Lego colors they remember
from their youth). Initially, I tried to classify this
zoo with the OpenCV computer vision library.
Using things like contour matching and circle detection, the system could tell the differences among the basic Lego bricks, but not
much more than that.
Next, I tried Bayesian classification: I chose
distinguishing features and built software
detectors for those. I came up with around
18 features, which included things such as the
height of the part, whether or not it had any
holes, how many studs were visible, and so on.
Building and testing the detectors took
quite a while, but eventually I was able to identify pieces with impressive accuracy. But the
system was too slow to keep up with the machinery. After a few other failed approaches,
and six months in, I decided to try out a neural
network. I settled on using TensorFlow, an immense library produced by the Google Brain
Team. TensorFlow can run on a CPU, but for a
huge speed increase I tapped the parallel computing power of the graphics processing unit in
my US $700 GTX1080 Ti Nvidia video card.
TensorFlow has an arduous learning curve,
but eventually I was pointed to Keras, a
Python language library by François Chollet.
Keras acts as a wrapper for TensorFlow and

http://www.Jacquesmattheij.com http://spectrum.ieee.org/ http://SPECTRUM.IEEE.ORG

Table of Contents for the Digital Edition of IEEE Spectrum July, 2017