Signal Processing - July 2017 - 108

speaker identification/verification, which targets obtaining
devices, such as smartphones, wearable devices, and tablets.
the speaker's identity from speech signals
Furthermore, as these devices often have microphones, social
■ computational paralinguistics, which attempts to distill
media apps, and Internet connectivity, they can be considered
nonlinguistic information mainly concerning the speaker's
distributed sensors or entryways for speech collection and proshort-term states (e.g., emotions), medium-term states
cessing. Thus, the advance of Internet technologies and the
(e.g., health condition and attitude), and long-term traits
ubiquity of smart devices can drastically reduce the cost and
(e.g., personality, age, and gender) from spoken speech.
time associated with collecting and processing speech data.
A serious obstacle to the broad application of ASA is the
Cloud computing, or Internet-based computing, is expected
lack of sufficiently labeled data in terms of both quantity and
to provide an on-demand computing resource. Thus, it gives
quality. For example, many available coman opportunity to store, access, and analyze
putational paralinguistics corpora contain
the volume of speech data generated by the
A serious obstacle to the
only a few hours of audio data at most [3].
distributed devices mentioned previously.
broad application of ASA
Similarly for ASR, many of the world's lanCloud computing has been shown not only
is the lack of sufficiently
guages are in a low-resource setting, where
to minimize the costs associated with an
labeled data in terms of
the electronic speech resources and linguisever-increasing demand for greater compuboth quantity and quality.
tic expertise are lacking. According to a
tational resources but also to reduce the cost
2010 United Nations Educational, Scientifassociated with infrastructure maintenance
ic, and Cultural Organization report [4], approximately 2,500
and user access. Motivated by these advantages, most major
languages are in danger of becoming extinct. In this scenario,
speech technology providers have already shifted their primait is exceptionally difficult to obtain a large-scale amount of
ry research and application attention from embedded systems
transcribed speech data to perform reliable ASR.
to cloud computing platforms.
The requirement for large-scale labeled data is not new
in machine leaning. Prevailing paradigms are often conductGeneralized automatic speech analysis:
ed in a supervised manner, and a substantial increase in the
Problem statement and notation
amount of available training data usually brings encouraging
The aforementioned technologies provide great potential to
performance improvements [5]. Because of the advancement
generate and process a large amount of speech data. However,
of deep-learning technologies [6], [7], this need for data has
there are three main challenges-data sparsity, unreliability,
become more compelling than ever. Deep-learning models
and nonmatching (Figure 1)-that limit the dissemination of
are often designed with millions of parameters, and, if trained
these data in research and industry. Before formally defining
with insufficient amounts of data, are vulnerable to being
these challenges, we first overview the generalized mathemattrapped in a locally optimized minimum, resulting in overfitical problem statement and notation commonly used in both
ting to the training data [6]. When sufficiently trained, howASA and throughout the remainder of this article.
ever, deep models reach unprecedented levels of performance.
First, let us define a domain D = {X, P (X )} that comFor example, Amodei et al. [7] utilized approximately 12,000
prises a feature space X and a marginal probability distriand 9,000 h of speech data to model English and Mandarin
bution P(X), where X denotes a set of feature vectors, i.e.,
X = {x 1, f, x n} ! X; while P(X) indicates the distribution
ASR systems, respectively, by employing deep-learning models with more than 35 million trainable parameters, achieving
of X in X. In the case that each feature vector x consists of
a performance breakthrough that exceeds the capability of
d attributes, i.e., x = {x 1, f, x d}, X is a d-dimensional space.
even human perception. Sufficient and reliably labeled data,
The most commonly used feature space X for ASA is arguwhen available, provide the opportunity to train robust ASA
ably the Mel-frequency cepstral coefficients (MFCCs) that
models whose resulting recognition is largely invariant in the
are extracted via filtering a speech frame by a bank of nonlinface of the abundance of acoustic variations naturally present
ear bandpass filters (Mel filters) whose frequency response
in speech data.
is based on the cochlea of the human auditory system [8].
Other exemplary feature spaces include the i-vector repreOpportunities
sentation often used for speaker identification/verification [9],
Traditionally, tasks such as data collection and annotation have
and mixed brute force feature representations, such as the
been performed by small groups of experts in a laboratory setbroadly used ComParE feature set, which contains 6,373
ting. This conventional work paradigm is often tedious, time
static features (i.e., statistical functionals including mean
consuming, and costly. However, the ongoing information and
and variance) of low-level descriptor (LLD) contours (i.e.,
communication technologies revolution and related technoloMFCCs) often used in tasks such as recognition of emotion
gies, such as the Internet of Things (IoT) and cloud computing,
from speech [10].
are providing us with opportunities to exploit larger amounts of
We further define a generic ASA task F = " Y, f ($) , that
speech data in more effective ways than ever before.
consists of a label space Y and a predictive function f (·) (or
The IoT, as a global infrastructure of the information socia conditional distribution P (Y | X )). The goal of this task is to
ety, is expected to offer advanced services (i.e., data collection)
build an effective and robust predictive function f (·) that is
by interconnecting a wide variety of contemporary recording
capable of learning transformation rules from the feature space
■

108

IEEE SIGNAL PROCESSING MAGAZINE

July 2017

Table of Contents for the Digital Edition of Signal Processing - July 2017