IEEE Signal Processing - May 2018 - 99

is particularly difficult because the different units (phonemes,
syllables, and so on) have varying durations and operate at different timescales.
The cognitive speech processing introduced in this article
is specifically based on a dual-stream cortical circuit [8], and
it presents a paradigm shift from perceptual (auditory) speech
processing toward cognitive (auditory/peripheral plus cortical/
central) sparse speech processing. Perception is extensively
studied in general auditory and speech processing [9]. Biologically, the cochlea and the auditory cortex contribute to
speech perception. Auditory sensations reach perception only
if received and processed by a cortical area.
Cognitive speech processing covers, in particular, the temporal aspects of speech processing. As shown in Figure 1, the
auditory cortex must deal with the different timescales pertaining to speech; one prominent hypothesis is that speech is
first parsed into chunks corresponding to syllables and phonemes, and then each chunk is categorized [10]. Studies have
indicated during speech processing that the brain generates a
cortical oscillation in the i range (3-8 Hz) that correlates to
the syllable rate, and faster c -range oscillations (25-40 Hz)
that correspond to the more transient acoustic properties.
As a result, the fine (phonetic) structure of the speech (the
energy bursts underlying consonants) have signal modulation
above 40 Hz. Although psychology and speech engineering
already have a long history of competing theories of speech
perception [11], recent experimental and theoretical developments in neuroscience support the idea that this cortical temporal sampling is thought to play a key role in human speech
processing [12].
The principle of this multiresolution temporal sampling has
been studied in the context of speech compression. The basic
idea is based on packaging information into units of different
temporal granularity, i.e., phonemes and syllables, in parallel.
As an example, the incremental phonetic vocoder-cascaded
speech recognition and synthesis systems-extended with
syllable-based asynchronous information transmission mechanisms was recently proposed [13]. The principles of asynchronous processing are fundamental to cortical perception
processing; asynchronicity exists in visual perception [14], in
audiovisual perception [15], and in asynchronous evolution
of various articulatory feature streams of speech recognition
[16]. In this article, cognition is assumed to be information
processing in the central nervous system after the peripheral
auditory system, any broader definition that relates to abstract
concepts such as memory, meaning, mind, and intelligence,
are avoided.

Human cognitive speech processing
Human speech coding
The functional anatomy of speech and language was learned
by observing its dysfunction. In the late 19th century, Carl
Wernicke observed that fluent aphasia, a communication disorder in which people utter fluent but meaningless speech with
impaired comprehension, was associated with damage in the

superior temporal gyrus (STG) [8], [17]. On the other hand,
damage to Broca's area causes nonfluent aphasia, which results in intact comprehension but partial loss of the ability to
produce written and spoken language [18].
The first stages of acoustic treatment (the peripheral auditory system, the front end), before reaching the auditory cortex
have been well characterized. This process is summarized in a
popular computational model [7], which lists precortical steps
as the following:
1) a decomposition of the acoustic signal into filter banks
through wavelet transform in the cochlea
2) a high-pass filter, nonlinear compression, and low-pass filtering by hair cells in the auditory nerve
3) an enhancement of the frequency selectivity (and rectification)
through a lateral inhibitory network in the cochlear nucleus
4) a further low-pass filtering in the midbrain.
Subsequent stages in auditory perception involving auditory
cortex [spreading from primary auditory cortex and A1 (i.e.,
the part of auditory cortex that receives direct input from the
thalamus)] and other cortices are the subject of intense research
currently. These are the main brain areas where cognitive speech
processing occurs.
Human auditory coding has evolved into highly efficient
coding strategies to maximize the information conveyed to the
brain (and between brain areas) while minimizing the required
energy and neural resources [19]. Sparse coding schemes and
hierarchical processing are central to A1 information extraction and transformation and are present from peripheral to central auditory structures [12], [20], [21]. In turn, the principles
of sparse and hierarchical (deep) structures in representation
learning of sound have led to advancements in speech processing techniques [22].
Investigations into electrophysiological recordings demonstrate that no more than 5% of neurons of A1 fire above 20
spike(s) in response to acoustic stimulation. This observation
suggests that the auditory responses are sparse and highly
selective [23], which permits more accurate representations and
a better discrimination of auditory stimuli [24].

A dual-stream model
Increasingly, speech (production and perception) data, as well
as promising functional neural data from the brain activity during speech, are being used to devise cognitive models of
speech and language production and perception. Figure 2
shows one prominent example, the simplified dual-stream cortical circuit linking cortical network architecture with speech
processing, which leads to a different paradigm in cognitive
speech coding (CSC). The first cortical stages of auditory processing take place in the auditory cortex and more anterior
parts of the STG. Spectrotemporal analysis on the precortical
input allows for the unveiling of spectrotemporal patterns (e.g.,
formants, place of articulation), the decoding of the associated
phonemes [27], and further phonological-level processing [25].
Damage of this brain area results, for example, in speech agnosia, an incapability to comprehend spoken words despite intact
hearing, speech production, and reading ability.

IEEE Signal Processing Magazine

May 2018

Table of Contents for the Digital Edition of IEEE Signal Processing - May 2018