Signal Processing - September 2017 - 136

across these systems were the design of the transceiver's antenna
and the algorithms used for filtering non-line-of-sight signals.
Interestingly, and similarly to Wi-Fi-based technologies,
there is a significant variation in the performance of UWBbased solutions. Even though all teams leverage the exact same
transceiver, localization accuracy ranges anywhere between
0.23 and 3.22 m for this type of system. This variation underlines the importance of properly designing the antenna front
end and the ability to effectively handle non-line-of-sight signals during the location computation.
Technologies based on the ToF of acoustic signals were able
to realize comparable accuracy to UWB-based systems. In
particular, Lazik and Rowe in 2015 were able to attain an error
of 0.31 m, a significant improvement compared to the error of
2.09 m that the same team achieved in the 2014 competition.
The error reached by other systems based on similar technology was much higher, ranging from 1.2 m (Hammer et al.) all
the way to 2.86 m (Flores et al.) in 2016.
Measuring the phase offset of Wi-Fi signals has been getting increasingly popular in the academic community. During
the competition, we were able to evaluate three such implementations (Reimann et al. and Schmid and Lee in 2014, and
Symington et al. in 2015). All of these systems require customized hardware to enable high-resolution phase measurements.
The implementation from Reimann et al. has been by the far
the most accurate, reaching an error of 0.72 m. More interestingly, Reimann et al. only deployed six out of the maximum
number of ten anchors allowed. This was also the system with
the best performance during the 2014 competition. It was surpassed in accuracy by only the ultrasound system by Lazik and
Rowe, most UWB-based solutions, and lidar-based systems.
Bluetooth beacons are one of the easiest ways to deploy an
indoor location solution but one of the hardest ones to accomplish fine-grain accuracy. Even though only a few such systems
were evaluated in this competition, the system by Chen et al.
in 2015 accomplished an impressive localization error of only
1.37 m. This system was using PDR along with BLE beacons,
and even though not as accurate as UWB or ultrasound technologies, the achieved accuracy is impressive given the large
evaluation area in the 2015 competition. As a comparison,
Dentamaro et al. in 2014 attained a 2.72-m error using Wi-Fi
fingerprinting and BLE beacons in a much smaller space with
a very high density of Wi-Fi access points.

Lab versus reality
Indoor localization approaches are usually evaluated in highly
controlled environments (i.e., a research lab). This type of
evaluation could positively bias the performance of the system.
During the 2014 competition, we attempted to quantify this bias.
In particular, we asked each participating team to report the localization error that it had previously attained in its own experiments, and we compared this error to the one achieved in the
competition. Most teams had worse accuracy by approximately
1.5-4 m. From the infrastructure-based teams, only Reimann
et al. achieved the same error as the one reported in their own
experiments. Infrastructure-free teams mainly accomplished the
136

same or, in a few cases (Beder and Klepal, Li et al.), better performance compared to their own experimental evaluation. We
believe that this was due to the large number of Wi-Fi access
points that were leveraged in the evaluation study. Given that the
evaluation area was relatively small (300 m2), all ten access points
could be successfully sniffed from every location in the evaluation area, creating an ideal setup for Wi-Fi-based approaches.

Impact of furniture setup on accuracy
Furthermore, we leveraged the two rooms in the 2014 evaluation area to quantify the impact of furniture setup on the accuracy of the systems. Even though both rooms had furniture,
we purposely changed the furniture setup in only one of these
rooms after the teams had calibrated their systems. By comparing the accuracy achieved by the systems in these two
rooms, we were able to quantify the impact of the furniture
setup on the localization accuracy.
With the exception of Li et al., the rest of the infrastructurefree approaches reported higher location errors in the room
where the furniture setup was modified. The error increase
varied anywhere between 0.47 and 0.94 m. Surprisingly, even
infrastructure-based approaches seem to be affected by the
changes in the furniture setup. The top four teams in this category, with the exception of Adler et al., exhibited an increase
in location errors in the modified room that varied anywhere
between 0.11 and 2.99 m.

Lessons learned
This evaluation study allowed us to closely observe and evaluate
multiple teams deploying various technologies in an unfamiliar
area. Even though the competing teams did not cover every single research and industry effort in the indoor location field, we
believe that the submissions are representative of the most popular indoor location technologies. Therefore, based on the analysis of the results and our experience organizing this event, we
believe we can safely extract a set of high-level conclusions.

Accuracy versus cost
After more than a decade of intensive work in this area, the
indoor location problem remains unsolved. There does not
seem to exist a technology or a combination of technologies
that can recreate the experience that GPS offers outdoors in
the indoor environment. Even though UWB and ultrasound
approaches can comfortably achieve localization accuracy
well below 0.5 m, their deployment overhead remains high.
Custom hardware needs to be carefully deployed in every area
where indoor location services are needed. Given that this
hardware needs to be hardwired to power to facilitate
maintenance over time, this increases the cost and effort
required to deploy accurate indoor location solutions.
However, fingerprinting-based approaches might not
require the deployment of custom hardware, but they rely on
laborious, manual data collection processes. Even worse, fingerprinting is a tedious process that needs to be frequently
repeated to ensure that the localization accuracy is resilient to
temporal signal variations and environmental changes, such as

IEEE SIGNAL PROCESSING MAGAZINE

September 2017

Table of Contents for the Digital Edition of Signal Processing - September 2017