IEEE Signal Processing - May 2018 - 101

feedback projections from higher areas to lower areas in the
cortex. Indeed, most cognitive processes rely on both feedforward (bottom-up) and feedback (top-down) operations. One
possible role for these projections is to convey predictions
about the upcoming sensory information. A neural signature of
such syllabic predictions was uncovered using magnetoencephalography (MEG) [34]. On the computational side, influential
models of speech perception dating back to the 1970s have
offered views that strongly oppose whether lexical predictions
can bias prelexical perceptual decisions. More recently, these
models have been recast within a Bayesian framework, whereby the brain computes posterior probabilities about phonemes,
syllables, and/or lexical units [35], [36]. According to the predictive coding theory, bottom-up signals only transmit the error
between the predictions and the actual sensory information
[37]. This allows for massive reduction in the size of the information passed on from sensory to higher areas, especially in
the context of speech, which contains many redundancies at
various levels.
Speech acquisition and production models, such as the directions into velocities of articulators (DIVA) [38], contain auditory
feedback and somatosensory (tactile) feedback, e.g., if the tip of
the tongue has correctly touched the alveolar ridge during the
t-sound production. Another speech production model, hierarchical state feedback control (HSFC) that posits internal error
detection and correction processes, can also detect and correct
speech production errors prior to articulation [39].

Biologically inspired speech representations
Biologically inspired systems have been proposed to enhance
the spectral representations of speech. The proposed models
approximate the peripheral and the central auditory systems,
with high-dimensional short-time vectors (e.g., vectors 3,840
in long [40], and 6,766 in [41]).

mammalian auditory pathway beyond the auditory nerve [47].
Finally, within the central auditory system, tuning properties of
auditory neurons in A1 are well described by sparse spectrotemporal filters, consistent with sparse encoding of acoustic
information [48].
Psychoacoustical and neurophysiological results indicate
that spectrotemporal modulations play an important role
in sound perception. Speech signals, in particular, exhibit
distinct spectrotemporal patterns that are well matched by
receptive fields of cortical neurons. Methods that capture
spectrotemporal modulations are believed to improve the performance of the speech recognition systems. Accordingly, the
Gabor-shaped localized spectrotemporal features were extensively deployed by scientists and engineers at the University
of California, Berkeley, for robust speech recognition systems [98]. The Gabor filters can model the shape of receptive
fields of cortical neurons in the primary auditory cortex [49],
[50]. The localized features are obtained by two-dimensional
(2-D) convolution of an auditory (MEL) spectrogram with the
Gabor filters.
The Gabor filters are defined as the product of a complex
sinusoidal function s ^n, k h with n and k denoting the time and
frequency index, respectively, and a short-time window function w ^n, k h . Spectrotemporal Gabor features may improve
recognition results in all acoustic conditions. For example,
automatic speech recognition in one-speaker conditions with
reverberation and noise resulted in large relative word error
rate (WER) improvements of at least 52% [43].

Temporal encoding systems

For natural audio signals like speech and environmental sounds,
gammatone atoms have been derived as expansion functions that
generate a nearly optimal sparse signal model [45]. Furthermore,
gammatone functions are established models for the human
auditory filters employed in the cochlea. Recent advances
exploit this property in developing a sparse gammatone signal
model that can predict the annoyance of background noise in listening to the speech signals as perceived by humans. The number of gammatones required to encode the noise is directly
correlated with the perception of noise intrusiveness [46].

A computational model of self-generated neural oscillations in
[51] demonstrates a proof of concept, such that neural oscillations can reliably signal syllable boundaries and that detected
syllable boundaries can improve the recognition of linguistic
units in a parallel neural pathway. In such a model, coupled
excitatory and inhibitory neurons intrinsically synchronize
around 6 Hz and automatically lock to edges in speech amplitude that convey the syllabic flow. The model is based on an
interconnected network of leaky integrate-and-fire neurons, in
essence, the network of ten excitatory and ten inhibitory neurons. Neural oscillations automatically lock to slow speech
fluctuations that convey the syllabic rhythm; a putative syllable boundary is declared for each inhibitory spike burst-i.e.,
whenever there were at least two inhibitory spikes occurring
within a window of 15 ms corresponding to the timescale of
integration of cortical neurons.

Spectrotemporal features

Phonological features

Sparse representation is useful in learning about the structures
in the spectrogram representation of sound, i.e., harmonics, formants, onsets, and localized patterns [47]. These sparse acoustic features resemble neuronal receptive fields reported in the
inferior colliculus (IC), as well as auditory thalamus and cortex; sparse modeling of neurons exhibits the same tradeoff in
spectrotemporal resolution as observed in the IC. This model is
able to predict the receptive fields of neurons in the ascending

Linguistic and neurocognitive studies recognize the phonological features as the essential and invariant representations used
in speech temporal organization. Cernak et al. [52] hypothesized that phonological speech representation inferred using a
deep-learning approach can form the basis of information
flow in the phonological network of the dual-stream model, as
shown in Figure 2. The phonological posterior probabilities
estimated by a feed-forward neural network (NN) convey

Perception of noise intrusiveness

IEEE Signal Processing Magazine

May 2018

101

Table of Contents for the Digital Edition of IEEE Signal Processing - May 2018