IEEE Consumer Electronics Magazine - May 2018 - 23

Monitoring System to measure food temperature and the
weight changes of the contents of a patient's plate; it consists
of a digital camera, weighing scale, infrared thermometer,
radio-frequency identification (RFID) reader, and a user
RFID transponder card. The system aims to improve dietary
data collection through routine monitoring of food temperature in a hospital setting. There are also many inventions
aimed at daily, long-term use.
Passler et al. [15] built a sensor system containing two
microphones to record chewing and swallowing sounds. It
detects food intake activity based on the signal energy of in-ear
and environmental sounds, classifying them as chewing and
swallowing using a hidden Markov model. A wearable sensor
platform consisting of a microphone and a camera is designed
to classify chewing activity based on sound features and video
sequence analysis, logging both the order in which food types
are eaten and the rate of consumption [12]. This could quantify
detailed information on subjects' food intake, which manually
acquired food records cannot perform. Kalantarian et al. [9]
proposed another solution, a piezoelectric-sensor-based wearable necklace. It is capable of estimating the volume of meals
and classifying solid and liquid foods. This reminds the user of
his or her food consumption, helping to establish a regular and
balanced diet and tracking the person's eating habits.
Computer vision and AR techniques are very popular in
food intake assessment. They analyze the collected images or
video to recognize food and estimate its volume. Then, using
both of these food nutrition retrieval techniques, food portions can be gauged. These techniques have the potential to
help users to know whether food items are healthy and if they
might eat too much. As a result, they then could make wise
decisions regarding food and form healthy dietary habits.
Shroff et al. [23] used a mobile-phone-based system called
DiaWear to monitor calories for patients who needed to manage their diet. The calorie estimation and image recognition
are both performed on a remote server. The previous dietary
assessment system [10] has the capability of performing food
identification and quantity approximation, although it has the
limitation that food quantity estimation must be conducted
with a checkerboard. Moreover, this work does not contain a
nutrition retrieval function.
Puri et al. [16] also used a cell phone to collect images of
the plate of food and then performed food-type recognition
with classifiers, volume estimation with 3-D reconstruction,
and the sending of quantitative nutrition information to a cell
phone. However, this system works only for certain types of
food. More important, the image analysis is performed offline
at a remote processing site, and sending quantitative nutrition
information via text message is an unintuitive and inconvenient way to display food information to users. Stutz et al.
[25] presented an AR solution using mobile phones. It overlays a 3-D model of the food, estimates the volume, and then
computes and displays the portion size. This is an evolution
for food portion estimation, which implements AR to
achieve real-time analysis. However, the food category is
manually selected.

One technique for CBIR is RIS,
which our proposed system
implements using a sample image
captured by Google Glass.
Waltner and his team developed a mobile application with
an AR-based dietary recommender using traditional classification algorithms, such as random forest [27]. It could be
used in a supermarket and helps users make decisions on
individual dietary-management issues. With the application
proposed by Yoshiyuki et al. [10], the user points a smartphone camera toward a food item before eating it. Then the
system recognizes the food within the defined bounding
boxes, and the nutrition information is shown on the screen.
Later, Ahn et al. [3] developed an AR-assisted mobile grocery-shopping application that offers real-time, customized
recommendations of healthy products to users, assisting
shoppers in making healthier decisions.

SYSTEM OVERVIEW
In this section, we discuss the design of our AR-based foodmonitoring system. It consists of three main components: food
recognition, retrieval of nutrition information, and image
tracking and visualization. The objective of the proposed
scheme is to assist users in making wise choices and to monitor their purchasing preferences. The typical use of the
approach is shown in Figure 1.
First, the system implements content-based image retrieval
(CBIR) via image recognition. Then, it fetches the relevant nutritional information on the targeted food from the online database
maintained by the U.S. Department of Agriculture (USDA).
Implementing CBIR for each frame is extremely time consuming. Therefore, instead of running CBIR for each incoming
frame, the proposed system tracks the specific food and overlays
the corresponding nutrient data on it. The system then sets the
coordinates of the dietary information overlay based on the
results of the image tracking for the most recent frame. Finally,
the nutrient facts are displayed on each expected food item.

RIS
One technique for CBIR is RIS, which our proposed system
implements using a sample image captured by Google Glass.
The image is uploaded to Google's hash server and analyzed
to find different combinations of global identifiers and local
features, such as color, points, lines, textures, and scaleinvariant feature transform, among others [31]. Those distinct features are used to formulate a query, in the form of a
hash. Then, the generated query is matched against the
images in Google's back end to locate those with visually
similar elements. Jsoup (Java Hypertext Markup Language
Parser) is used to parse the webpage and to extract the image
name and description.
MAy 2018

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - May 2018