IEEE Consumer Electronics Magazine - May 2018 - 25

orientation of all of the filtered FAST features using the
intensity centroid [19], and computes the descriptors using
rotated Binary Robust Independent Elementary Features [6],
where the coordinates of the elements in any feature set of
the binary test are rotated according to the orientation of the
keypoints. Given the keypoints detected by the ORB algorithm, the LDB algorithm incorporates the first-order gradients, in addition to using the average intensity, and computes
the initial descriptors by concatenating the binary strings
from all of the partitions generated by multiple gridding
choices. That is, each image patch is partitioned in multiple
ways, e.g., 2 × 2, 3 × 3, and 4 × 4. Then, an AdaBoost-based
bit selection scheme was employed to improve the efficiency
for matching and storage while reducing the correlations
between dimensions.
To correctly update the coordinates of the tracked food, the
set of descriptors in the reference image should be matched
with the newly captured scene image. If there are enough
matching feature points between the two images, the homography matrix between the most recent frame and the reference
frame is estimated. The mapping process is defined by
H 11 H 12 H 13 x 1
x2
y
=
f 2 p f H 21 H 22 H 23 pf y 1 p + X s = HX r,
z2
H 31 H 32 H 33 z 1

(1)

where H is the generated homography matrix, X s is the coordinates of the scene corners, and X r is the coordinates of the
reference corners. The proposed system automatically compensates for the user's change of view by receiving the result
of a series of coordinate transformations.
With food tracking, the nutritional data are labeled on a
tracked grocery item as shown in Figure 2. After recognizing
the carrot successfully, the dietary information pertaining to
carrots was retrieved from the online USDA database and
overlaid in the area of the blue box, which is the position of
the carrot, by updating the food location on every frame.

To better demonstrate how the
proposed system works, we
streamed the display from Google
Glass to an Android phone.
RIS PERFORMANCE AND EVALUATION
We evaluated the proposed system in the environment of a
real supermarket, comparing the recognition accuracy of the
food in the real scene against that of the food in the public
image data set FIDS30 [26]. We selected ten types of food
from the FIDS30 data set, each of them with 15 images. Correspondingly, we collected 15 images for each type of food in
a supermarket. Each photo contained at least one food item in
its entirety. The recognition rates for the individual food
items are shown in Table 1.
The overall accuracy of the RIS on images from the real
scene was 75.9%, while that of the RIS on photos from the
image database could reach 87.9%. This is because the images in the FIDS30 database are always presented with simple
backgrounds, usually plain white. However, in real scenes,
the background is always more complicated. For instance,
there might be several foods in the same view, or the image
could be partially occluded, or simply have a low illumination level [22]. The photos in the FIDS30 database are always
well illuminated, whereas images in a supermarket are less
well lit. These sources of noise in a real supermarket will
affect the performance of the RIS. For example, it will usually recognize food with a rough surface, such as a cantaloupe,
as another item that has a similar color and texture, such as a
rock. Also, with lower illumination levels and more complicated backgrounds, the RIS can mistake foods with similar
colors, such as oranges and lemons, for each other. Additionally, dark-colored items, such as grapes, cannot be easily recognized with low illumination levels. Note that foods with a

EXPERIMENTS AND RESULTS
To explore the usability and limitations of the
developed system, we conducted four corresponding experiments. The first one was to
validate the accuracy of the RIS in different
situations, such as against simple or complicated backgrounds. The second was designed
to test the overall speed of the whole system.
The third focused on AR stability and how AR
technology benefits the proposed system. A
final investigation showed the tradeoff
between AR performance and RIS accuracy.
To better demonstrate how the proposed system works, we streamed the display from
Google Glass to an Android phone to visualize
the complete process. Figure 3 gives an overview of a simple demonstration of how the
system works.

FIGURE 2. The nutritional information on a tracked grocery item. The blue box
represents the location of the food as tracked by the proposed system. Information for several nutrients are reported per 100 g.

MAy 2018

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - May 2018