Signal Processing - January 2017 - 69

human-computer interaction personal (particularly around
web search), personalization has remained largely broken
until recently, not only for PDAs but also for general human-
computer interaction due to the four main gaps.
1) Data: The system had a limited amount of data to properly
model the user and his or her interests. It understands the
user based on the experience it provides and the feedback
loop it uses.
2) Computing: The limited computing power and machine
learning were not adequate for modeling the complexity of
user behavior.
3) Interest: There has been conflict of interest between the
user, the platform, and those who pay for the user's attention. The system has not been necessarily prioritizing the
user's interests over these other actors.
4) Content/action: The system does not support the actions
the user wants to perform or does not have the content to
serve the user's interests.
During the past seven years, two primary changes have
occurred that allowed PDAs to be personal: 1) the increased
number of device sensors on mobile phones and 2) the quality
and quantity of the user data and digital artifacts (on the device
and/or in the cloud) coming from the web services and applications the user accesses. As a result of these changes, it is now
possible to represent a user along four axes:
1) user profile: user's name, age, gender, parental status, profession, employer, home, work, people in his or her close
circle (people graph), favorite places, files, documents,
music, photos, and interests explicitly provided by the user
2) digital activity: digital artifacts (e.g., calendar, e-mail,
social media activity, web searches) on applications and
web services
3) space: physical location of the user
4) time: time at which a specific digital or physical activity
takes place.
These four dimensions, when considered together, blend the
physical with the digital world and open up new possibilities for
powerful inferences and deep user understanding. Inevitably,
managing and protecting privacy and security of the user data
and information is a major concern, and what has been done
in that space is critical, but it is outside the scope of this article.

Mobile device sensors
The computational power and capabilities of mobile phones
are increasing every year. The number of built-in sensors on
smartphones (e.g., Samsung Galaxy) more than tripled during
the past five years [71]. Smartphone sensors measure motion/
orientation, GPS coordinates, and many other user and environmental conditions. For example, a device's gravity sensor
provides data to infer complex user gestures and motions,
such as shake, swing, or rotation. The rich high-precision data
coming out of these sensors are made available through application programming interfaces (APIs) and are used in numerous applications and scenarios. The information is sensitive,
as it is personal and contextual. Making it available opens up
new research areas like fitness and health applications or

(a)

(b)

Figure 1. The proactive flight cards. (a) Summary and suggestions for the
trip. (b) Flight details for the first leg.

opens up new solutions to already existing problems [6]. User
experiences that currently exist can also be enhanced by the
available data. For example, using activity detection, the PDA
can hold the incoming call or send a short message service
(SMS) text message to the caller if the user is biking or it can
turn up the volume if the user is climbing stairs/walking.
The three main mobile platforms (Android, iPhone operating system, Windows) support four broad categories of sensors
on mobile devices.
1) Motion: This sensor set includes accelerometers, gravity
sensors, gyroscopes, and rotational vector sensors. They
measure acceleration and rotational forces along three axes.
They measure movement and orientation of the device.
2) Environmental: These sensors measure various environmental
conditions, such as ambient air temperature and pressure, illumination, and humidity. This category includes barometers,
photometers, magnetometers, and thermometers.
3) Position and location: These sensors measure the physical
position and location of a device. This category includes
orientation sensors and magnetometers. The magnetometer
can determine the rotation of the device relative to magnetic north. It can also detect magnetic fields around the
device. GPS and Wi-Fi (not really a sensor in the traditional sense) determine the location of the device.
4) Proximity: This sensor detects whether the phone is brought
near the face during a phone call. This functionality disables
the touch screen, preventing inadvertent input to the phone
from the user's face and can also save battery power.

System architecture
The scenarios that the PDAs support can be divided into two
main categories: 1) proactive and 2) reactive assistance. The
conceptual agent architecture designed to support these two
modes of assistance is shown in Figure 2. The system

IEEE Signal Processing Magazine

January 2017

Table of Contents for the Digital Edition of Signal Processing - January 2017