Signal Processing - July 2017 - 111

P (YS X S) ! P (YT X T ) . A more in-depth explanation of these
to perform the required annotations. By doing this, we create a
discrepancies can be found in [21].
new or additional labeled data set L cs, and then ASA models
An idealized solution to mitigate these differences is to
can learn from the increased labeled data set Ll = L , L cs.
obtain access to all possible variations by acquiring data on
Manual annotation is, however, costly in terms of time and
a massive scale. However, it is either practically impossible to
money. Therefore, strategies to reduce these costs are of
anticipate all variations or such data would require exhaustive
particular importance.
annotation. In such unmatched scenarios, transfer learning
Crowdsourcing is one method to gather the needed data in
(TL) [21] is regarded to be a highly promising technique to take
a cost-efficient manner. In crowdsourcing, human intelligence
advantage of the knowledge from the source
tasks (HITs) such as data annotation are disdomain for the target domain.
tributed via the Internet to a large number
Another emerging trend
Finally, it is important to note that all
of potential workers (annotators). The users
for crowdsourcing is
of the aforementioned techniques for each
perform the tasks for usually low compensathe gamification of the
challenge can be performed either individution. The assumption behind crowdsourcing
ally or jointly. This is illustrated in Figure 1,
is that the use of nonexperts is less onerous
service, which is used to
where possible combinations that can occur
and more rapid than the use of experts. Furintroduce a sense of fun
are indicated through the use of the N(Y)
into what are often simple thermore, the aggregated opinion of many
symbol, which denotes no or yes. For examnonexperts has been shown to approach the
and recurring tasks.
ple, crowdsourcing can be used no matter
quality of the opinion offered by comparawhether the labeled target data are available
tively fewer experts [15], [23], [24].
or not. Likewise, AL can be executed on either unlabeled tarPopular crowdsourcing platforms include Amazon Mechaniget data or unlabeled source data. All of the key techniques
cal Turk (MTurk), CrowdFlower, and Crowdee. MTurk is
mentioned in this section are reviewed in detail in the followlikely the most popular crowdsourcing platform for ASAing sections.
related tasks. While MTurk provides access to a larger number
of potential annotators, it is considered relatively expensive
Contributions of this article
when compared to other platforms [15]. The CrowdFlower
The literature shows a few surveys relevant to the topic of this
platform is steadily increasing its market share. When comarticle. Deng et al. [22] offered a comprehensive overview of
pared to Mturk, it provides customers with a steady number
machine-learning paradigms for speech recognition systems.
of contributors and has a higher degree of quality control.
Wang et al. [20] provided a TL survey for speech and lanAn emerging trend, as implemented by Crowdee, involves
guage processing, drawing the conclusion that TL has the
moving the platform from the web to a mobile platform.
potential to overcome the data-mismatch challenge. None of
Participants associated with this platform have the potenthese surveys, however, perform a complete analysis of the
tial to undertake a task at any time and place.
sparse, unreliable, and unmatched data challenges or provide
Another emerging trend for crowdsourcing is the gamificaa comprehensive overview of the corresponding approaches.
tion of the service, which is used to introduce a sense of fun
Extending from a previous abstract [12], this article is the
into what are often simple and recurring tasks. This is also
first to offer a thorough and in-depth overview of the most
interesting from an ethical point of view, aiming to improve
prominent and state-of-the-art techniques in this direction,
working conditions of crowd workers. The iHEARu-play
including crowdsourcing for efficient data labeling; spokenplatform, for example, offers annotators a chance to perform
term detection/discovery to facilitate learning when there are
labeling, or prompted recording tasks, in return for scores and
no labeled data; data augmentation, speech synthesis, URL,
prizes, which are computed on the correctness and workload
SSL, AL, and CL to enable learning when only a limited
of their annotations [25].
amount of labeled resources are available; data selection and
Generally, the procedure of crowdsourcing speech resourcbalancing techniques to facilitate learning from unreliable or
es can be broken into four stages. The first step is to define the
unbalanced resources; and TL and data agglomeration to learn
project parameters, such as an appropriate platform, quality
unmatched resources.
control strategy, budget, and time scale. The second step is to
Rather than simply enumerating a list of associated papers
prepare the data. The third step is to distribute tasks. This genand techniques, the focus of this article is on the analysis of
erally involves splitting the whole task into many small units
the various data conditions and on how to better explore data
and then assigning each unit to several annotators. The final
under the different conditions. In doing this, ASA researchstep is to aggregate and evaluate the resources (e.g., speech
ers and developers, new and established, can profit from
data or annotations).
the approaches introduced and discussed for the aforemenFor speech processing, crowdsourcing has been widely
tioned applications.
employed for a range of tasks, including speech data collection/acquisition, speech annotation, speech perception,
Efficient data labeling: Crowdsourcing
assessment of speech synthesis, and dialog system evaluaThe most straightforward solution to address a shortage of
tion [15], [26]. With particular respect to speech annotation,
labeled data is to organize a group of workers (i.e., annotators)
many studies have shown crowdsourcing's benefit in terms
IEEE SIGNAL PROCESSING MAGAZINE

July 2017

111

Table of Contents for the Digital Edition of Signal Processing - July 2017