IEEE Technology and Society Magazine - Winter 2013 - 62

human behavior and incentives
as it did about technology. The
winning teams devised strategies
that successfully provided incentives for individual participation,
and this aspect was arguably
more important than their technical approach to the problem. But
nuclear weapons aren't flagged
with red balloons, and underground nuclear tests are rare. So
the PTM approach faces a substantial challenge in obtaining and
sustaining public engagement in
the verification arena.
For data collection, one approach
might be to bundle together a variety of sensors that pertain to "public service" monitoring. This might
include some combination of CO2
and other environmental sensors,
seismic monitoring, and various
biological and chemical sensors. A
modular hardware standard would
allow individuals to deploy whatever combination of devices best
suited to their particular combination of interests and ability to
invest.
On the data and image analysis
side, it seems to us unavoidable that
only those individuals and organizations that have a sustained interest in a particular problem will
invest the time and effort needed
to provide a valuable PTM contribution. But the Galaxy Zoo project (discussed below) has provided
an example of how a structured,
supervised, and reward-based
image analysis framework can
succeed.

Some Relevant Examples and
Models for PTM
A number of existing systems provide relevant examples and lessons
for PTM. These include:
■ The quake-catcher network,
which exploits accelerometer
data from laptops and mobile
devices. This example was discussed above.
■ Public analysis and inspection
of satellite images, obtained
from both Earth-observing
62

|

and astronomical imaging
systems.
Although there are some interesting examples of non-governmental use of downward-looking
satellite images, we point to the
Galaxy Zoo project as an example
of widespread supervised public
engagement in image analysis and
interpretation - essentially crowdsourced data analysis.
we submit that the Galaxy Zoo
project3 is a very promising example of public participation in image
analysis, with clear applicability
to the verification challenges of
the decades ahead. The Galaxy
Zoo project had the goal of using
minimally trained citizen scientists to classify (with visual inspection) the characteristics of galaxies
using digital astronomical images
obtained with the Sloan Digital
Sky Survey. The Survey obtained
exquisite-resolution images of the
entire Northern sky, and amassed a
total of over 27 TeraBytes of imaging data and object catalogs.
The response was enormous.
The Galaxy Zoo project attracted
250  000 online participants, from
170 countries. This public wave of
amateur astronomers succeeded in
classifying over 100 million galaxies, by visual inspection of every
single image. The public participation in this project is comparable
to the 300  000 individuals who
have helped construct wikipedia. Numerous scientific results
have been drawn from the galaxy
classifications performed by this
entirely volunteer community of
world citizens. This demonstrates
the tremendous leverage that can be
attained with a guided and informed
interaction of interested, connected
citizens with public domain data.
The Galaxy Zoo project has been
so successful that understanding its
effectiveness has become a research
endeavor in its own right [12].
One key to the project's success
was the shrewd use of computers

for those aspects of image processing that computers do well,
and then presenting people with
a resulting subset of the pixels in
a way that allowed them to efficiently and effectively address the
question of interest.
we have attempted to distill
some lessons from the Galaxy Zoo
project, in the context of future
large-scale PTM image analysis:
1) Incentives: The project must
engage the public with real
data, on a topic of interest, and
provide effective incentives for
ongoing participation.
2) Clarity: The project must provide an elegant yet effective
user interface, with clearly
defined tasking.
3) Education and training:
Online tutorials and intuitive
data access tools4 are essential.
4) Assessment: For data analysis and interpretation tasks
executed by the public, participants should pass simple proficiency tests before embarking
on the project tasks. The
results can then be used to
assign appropriate statistical weights to the judgments
rendered by each (calibrated)
participant.
5) Redundancy: Multiple individuals should carry out independent analyses, across a wide
geographical range. The statistical properties of these results
can then be used to assess and
validate the system. This also
avoids any simple gaming and
denial and deception attempts.
6) Enjoyable: The projects should
be fun, with good visualization
and interaction tools.
7) Collaborative: The project
team should provide multiple
mechanisms for structured and
loosely supervised interaction between and among participants, thereby allowing the
group's experts to mentor less
experienced individuals.

3

4

See http://www.galaxyzoo.org.

See http://zoo1.galaxyzoo.org/Tutorial.aspx.

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

|

wINTEr 2013
http://www.galaxyzoo.org http://zoo1.galaxyzoo.org/Tutorial.aspx
Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - Winter 2013
