Signal Processing - July 2017 - 118

data set ^ Ll = L , L )sslh, which is sequentially employed to
refine the previous model and retest the remaining unlabeled
data. This process is repeated several times to incrementally
upgrade the initial model.
Self-training is simple and can be easily applied to
an existing model. However, it is open to the risk of error
accumulation, which is introduced by the selection of misclassified data in early learning iterations. Commonly used
techniques to mitigate such a detrimental effect include
1) using an additional development partition to determine the
stopping point of learning, 2) using generalized expectation
maximization to assign weights to the automatically labeled
data based on the prediction confidence [74], and 3) retesting previously selected data for subsequent reevaluations and
selections, such that the mislabeled data in previous iterations
are possibly corrected in future iterations with an improved
model [91].
Another commonly used inductive SSL paradigm in ASA
is cotraining. Compared with self-training, cotraining attempts
to exploit the mutual information between two learners (trained
on different views or feature domains X 1 and X 2). That is,
each learner uses its own predictions to teach not only itself,
but also the other learner [92].
Successful cotraining relies on two assumptions: sufficiency and conditional independence [92]. Sufficiency infers
that each view is sufficient for classification on its own, i.e.,
the two hypotheses f1: X 1 7 Y and f2: X 2 7 Y are good
enough for recognition. Conditional independence denotes
that the views are conditionally independent, given the class
label, i.e., P (y i x) ! P (y i x 1) P (y i x 2) . Although these two
assumptions are restrictive, the work presented in [76] shows
the capability of cotraining for retrieving emotional information in unlabeled data via separating the acoustic feature
set into two pseudo views (i.e., not completely conditional
independence) in the speech domain. Similar verification of
cotraining has also been reported for other computational
paralinguistics tasks [76]. Additionally, a more general
framework called multiview learning requires less restriction in terms of conditional independence than cotraining
and has been successfully applied in speech recognition
by using several types of acoustic features and randomized
decision trees [77].
More recently, SSL research in ASA has started to explore
the advantages of deep-learning techniques [75], [93]. A typical implementation is ASR for a low-resource language [75],
[93]. First, an initial DNN is trained in an unsupervised manner
using multilingual data to learn the generalized representation of speech. Next, this model is fine-tuned as a seed model
by using limited amounts of monolingual data from the lowresource language. The seed model is then employed to decode
the untranscribed utterances, with the predicted hypotheses
being regarded as the training transcripts for the next iteration. Various discriminative criteria (e.g., maximum mutual
information or minimum cross entropy) can be adopted to
obtain the prediction confidence scores for each frame, word,
or utterance [75], [93]. Similar to traditional self-training and
118

cotraining, the data (i.e., frame, word, or utterance) predicted
with high confidence are assumed to be of high quality and are
then incorporated to update the initial DNN or GMM-HMM
acoustic model.
Apart from the inductive approaches, a graph-based transductive approach can also be integrated into DNN-based
speech recognition systems at either a late or early stage [78].
For the late-stage integration, a graph is first constructed over
the labeled and unlabeled data sets, where the node represents a data instance and the edge indicates the similarity
between a data instance pair. Then, using a graphic-based
learning algorithm, a new set of posterior distributions for
each instance of unlabeled data is produced. After that, the
posteriors are converted into a graph likelihood and are integrated with the original acoustic scores given by the DNN for
a subsequent rescoring of the unlabeled data [78]. A major
drawback of this late integration approach is a substantially
increased computational cost, as the graph has to be reconstructed after each learning iteration. To overcome this problem, an early-stage integration algorithm has been proposed
[78]. This algorithm employs a graph embedding approach
in which the data in the graph is transformed into a compact feature vector, which is then used as additional input for
the DNN.

Active learning
Similar to SSL, AL attempts to improve recognition models
by exploring unlabeled data. However, unlike SSL, which performs automatic machine (model) annotation, the focus of AL
approaches is to efficiently select the most informative data S
in the unlabeled collection U for manual annotation. Partly
because of the growing amounts of data to be handled and the
popularity of crowdsourcing (see the "Efficient Data Labeling:
Crowdsourcing" section), AL strategies for ASA are currently
more important than ever.
One of the central goals of AL is to determine the informativeness of unlabeled data, a process known as query strategy.
The following sections briefly review the most commonly used
strategies with relevance to ASA, which include the uncertainty sampling, query by committee, and metaquery strategies.

Uncertainty sampling
This strategy uses confidence measures as a criterion to select
the most informative data. The basic idea is to use a pretrained
model (an active learner) to determine the uncertainty of predictions for a specific ASA task. The instances with the least
certain predictions are then sent to an oracle (a human) for the
annotation.
Formally, the selected data can be expressed as
xl = argmin Q c (x; i),
x !U

(11)

where i indicates the model parameters trained on the labeled
data set L and Q c denotes the confidence measure function.
When using a probability model (e.g., Bayesian networks),
this function is usually estimated using either the posterior

IEEE SIGNAL PROCESSING MAGAZINE

July 2017

Table of Contents for the Digital Edition of Signal Processing - July 2017