Signal Processing - September 2017 - 137

furniture setup and human motion. Furthermore, even under
somewhat ideal conditions, fingerprinting-based approaches
are not able to provide localization accuracy below 2 m, making these technologies unsuitable for a variety of applications
that require submeter accuracy.
From directly observing all the teams during the set-up day,
it became clear that the installation/profiling cost of current
approaches is prohibitively high. All teams were given 7 h to
deploy their hardware and/or profile a relatively small area of a
building. Even though one would think that 7 h should be way
more than enough time for the teams to set up their systems,
this wasn't the case. On average, it took each team 5 h to set
up its approach in the designated evaluation area, with many
teams leveraging the full 7 h of set-up time. For a couple of
teams, 7 h was not enough time to fully position their systems.
This is particularly concerning given the fact that the teams did
not have to worry about any practical issues that any commercial installation would impose (i.e., aesthetics, properly hiding
the deployed equipment, and so forth). In addition, the whole
process of installing custom hardware and profiling the space
was quite intrusive. We don't believe that any business owner
would like to perform either of these two tasks while real customers are in the business.
When considering the massive size of deployment candidate
sites (i.e., shopping malls) and how intrusive, time consuming,
and labor intensive the processes of positioning hardware and
profiling the space are, realistic indoor location deployments
that can achieve centimeter-level accuracy seem infeasible at
this point. Reducing the overhead and manual labor required
by the different indoor location technologies is of paramount
importance for their success.

Redesigning indoor location evaluation
The way indoor location technologies are evaluated and compared can be rather tricky. Even though various metrics have
been proposed in the literature (i.e., average location error,
root-mean-square error, 95th percentile, and so on), there are
variations in the real world that are not being properly captured by these metrics. For instance, not all evaluation points
are equal. There are easy points that almost any indoor location approach can easily handle, and there are points that are
really hard to accurately localize. As a result, the way evaluation points are selected and weighted in the evaluation metric
becomes crucial.
The design of appropriate evaluation metrics can have a significant impact on the results of the competition and the conclusions that are made. In particular, there is an opportunity
for researchers to contribute to future competitions (and to the
research community in general) by developing metrics that are
fair across a set of competitors that participate using a heterogeneous set of platforms and technologies.

Designing a better indoor localization competition
Organizing an indoor localization competition over the past four
years gave us a great deal of insight on how an ideal competition
should be organized. First, one of the major issues that teams had

to deal with was the RF interference caused by the numerous
custom RF solutions that were simultaneously deployed. This
interference made calibration a tedious task, and in some cases,
it prevented contestants from properly calibrating their systems.
Ideally, and assuming no realistic time restrictions, each team
should be allocated a time slot during which only this team's
system is active, enabling hassle-free system calibration.
Second, instead of using relatively small evaluation areas
consisting of a few rooms on a single floor, a significantly larger area with a mixture of open and office-like spaces across
multiple floors should be leveraged for such an evaluation.
Third, the evaluation of systems in this competition has
been point-based, ignoring the ability of these systems to perform continuous localization as the human subject moves in
space and time. A way to capture and quantify the ability of
indoor location systems to capture the continuous path that the
human subject follows would be of great value.
Fourth, to ensure that all systems are evaluated under identical
environmental conditions (i.e., number of people in the room,
interference, and so forth), all systems should be simultaneously evaluated at a given evaluation point. Also, to capture
temporal variations, all systems should be evaluated across
different time windows and days as well.
Fifth, it is very hard to capture the effectiveness of an indoor
localization algorithm with a single metric. Ideally, competing
systems should be compared across a wide variety of localization accuracy metrics, and several other aspects of each
system, such as deployment overhead, set-up time, and more,
should be quantified in detail.

Conclusions
The Microsoft Indoor Localization Competition described in
this article was an experiment that aimed to bring multiple
indoor location technologies under the same roof and directly
compare their accuracy and overhead requirements. The overwhelming participation clearly demonstrated that indoor
location remains a hot topic. It also demonstrated the need from
the research and industry community in this area to have a
venue for demonstrating its latest results and comparing its performance to other teams in a reliable way. Based on the passion
the teams demonstrated and the fun they had during the event,
we believe that more experiments like this one need to take
place or even be established as recurring (i.e., yearly) events.

Acknowledgment
We would like to thank all of the teams that devoted time and
much hard work into deploying their systems during the
Microsoft Indoor Localization Competitions.

Authors
Dimitrios Lymberopoulos (dlymper@microsoft.com) received his undergraduate degree in computer engineering from
the Computer Engineering and Informatics Department at the
University of Patras, Greece, and his Ph.D. degree from the
Electrical Engineering Department at Yale University, New Haven,
Connecticut, in 2008, where he designed and implemented

IEEE SIGNAL PROCESSING MAGAZINE

September 2017

137

Table of Contents for the Digital Edition of Signal Processing - September 2017