IEEE Consumer Electronics Magazine - May 2018 - 17

Marker Processing

Audio Notification

RenderScript
Process

Image Sequence
Mobile Phone
Application
Sensor Values

Native C
Process

User Interface

Values Analysis

TTS
Locating System

Text Information

Obstacle
Detection

Vibration

FIgurE 9. A block diagram of the mobile phone application.

mobile pHoNe ApplicATioN
There are three inputs for the proposed application (Figure 9),
i.e., image sequence, sensors values, and the interface control.
During localization, images will be streamed to the marker
processing. To improve the performance of the proposed system, RenderScript, which is a parallel computing framework
in Android application programming interface, was used
because the segmentation and intersection are pixel-wise processing. If the QR code can be successfully recognized, the
stream of images will be stopped. The localization provides
text information from the QR code, and the user can get audio
feedback from the text-to-speech (TTS) that contains the location and orientation of the nearby locations. If the marker is
detected but the QR code cannot be decoded, the system
will provide TAG data and the direction of the marker to
make it easier for the user to locate the marker.
Next, the signals from the ultrasonic transducers are the
input of the obstacle detection function, which are defined
as the distance to an object within a range. The obstacle
detection analyzes these values to determine the direction of
the obstacle. Among three multidirectional transducers, the
front-facing sensor is given the highest priority, as the user
would tend to walk straight. When an obstacle is too close,

the application will alert the user to avoid collision through
audio and tactile feedback.
Another input is the application interface on the user's
mobile phone. As the target of this study is mainly aimed at
individuals with visual impairment, the user interface for
the proposed system utilizes a voice assistant from the
Android system that provides speech feedback on the item
chosen by the user. Additional interfaces, such as the camera view and other information, are also provided for
debugging purposes.

EXPErIMEnTaL rESuLTS
THe RuNTime of THe pRoposed
mARkeR pRocessiNg
To validate the performance of the proposed marker, the average runtime of each processing step is measured on four
mobile phones with different specifications on the basis of
ten samples. The specifications are presented in Table 1.
Based on the runtime comparisons of the proposed system
on various implementation platforms (presented in Table 2),
the system consumes an average runtime of 78.5 ms for processing one frame, or 12.74 frames/s (fps), with low deviation

Table 2. Runtime comparisons of the proposed
system for a single frame on various phones.

Table 1. The specifications of the tested
mobile phones.
Index

CPU

GPU

RAM

Process

Phone A

Phone B

Phone C

Phone D

Phone A

Snapdragon 800 and
Quad-core 2.3 GHz

Adreno
330

2 GB

6.1

Segmentation

8 ms

5 ms

11 ms

Phone B

Snapdragon 801 and
Quad-core 2.5 GHz

Adreno
330

2 GB

6.1

Intersection

13 ms

12 ms

7 ms

16 ms

Clustering

25 ms

23 ms

20 ms

Phone C

Snapdragon 808,
Quad-core
1.44 GHz, and Dualcore 1.82 GHz

Adreno
418

2 GB

6.1

Centering

5 ms

4 ms

3 ms

2 ms

Matching

5 ms

3 ms

QR decoding

28 ms

27 ms

22 ms

18 ms

Exynos 5433,
Quad-core 1.3 GHz,
and
Quad-core 1.9 GHz

MaliT760
MP6

3 GB

TAG decoding

4 ms

3 ms

Total

88 ms

85 ms

68 ms

73 ms

Phone D

5.1

MAy 2018

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - May 2018