IEEE Circuits and Systems Magazine - Q4 2019 - 35

■ Effective learning of representation: Compared

to other machine learning algorithms, deep
learning based techniques can learn and create
more comprehensive and informative representations from raw audio signals. Deep learning
based approaches applied in audio generation
systems can model high-dimensional, highly
correlated features efficiently. This reduces the
demand for hand-crafted feature engineering,
which is the most time consuming aspect for
speech synthesis systems. More complex sets of
features are allowed for training and evaluation
in practical tasks.
■ Powerful modeling of relationships: With the
various types of activation functions, NNs have
the ability to model nonlinear and complicated
relationships between inputs and outputs. The
unique multi-layered architecture with nonlinear operations integrates feature extraction with
acoustic modeling. NNs can also model multiple
simultaneous acoustic events within one frame to
create waveform files with high fidelity. This is a
vital property for dealing with natural signals sampled from real-world scenarios and for producing
fluent conversations or background sounds.
■ Flexible setting of networks: Unlike many other
machine learning methods, specific tasks in audio generation applying deep learning can be
processed with a flexible architecture with diverse combinations of DNN modules. This offers
a better representation ability with increased
flexibility in the parameter configuration. Additionally, there are fewer restrictions placed on
the format of the inputs and allows wider commercial deployments.
B. Future Issues
Even though deep learning technology has assisted the
exploitation for more audio generation applications,
there are still some issues to be resolved for a more advanced realization of the "intelligible chatbot":
■ Design of audio corpora: With an increasing
amount of sound receivers embedded in portable
devices, more realistic and coarse data collected
are used in the acoustic feature extraction and
modeling processes. Moreover, when only a limited or an imbalanced amount of labeled audio
signals is available, the performance of a deep
neural network will inevitably degrade. Further
efforts are required for designing elaborate audio corpora with diverse acoustic environments.
More audio datasets composed of specific types
of acoustic signals need to be created for recreFOURTH QUARTER 2019

ational and professional generation tasks like music composing and video post-processing.
■ Audio generation systems for practical scenarios: With the rapid development of computational
resources and deep learning algorithms, time cost
on training has been effectively reduced by machines equipped with GPUs. However, it is still unworkable for most low-power portable devices like
mobile phones. To relieve the computational burden, advances in hardware are necessary for the
next generation intelligent equipments for both
industry and academia. From the viewpoint of algorithms, several possible future directions warrant further investigation, such as more efficient
models adaption for cross-lingual TTS systems
and speaking style transferring. Emotional speech
synthesis applying deep learning approaches is
another potential application in real-world scenes.
VI. Conclusion
In this paper, the most commonly used deep generative and discriminative algorithms were reviewed. A detailed statement for the application of deep learning to
speech/acoustic signal processing, especially audio generation, was provided for readers in the machine learning community. Additionally, the benefits introduced by
deep learning methods and several issues of further research were discussed in detail.
Deep learning methods have shown remarkable improvement for pattern recognition and generation problems. Other machine learning approaches have been
outperformed by deep learning methods due to their
powerful capability of feature representation and complex model understanding. By adopting up-to-date computational equipments like GPUs, deep learning techniques demonstrate impressive performance for yielding
human-like speech. It is indeed the case that there is still
a long way to go before the realization of a general intelligence or strong AI. However, there is wide confidence
that deep learning will receive more enthusiasm and motivation for broader applications in the future.
Acknowledgment
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPU used
for this research.
Yuanjun Zhao received the Bachelor degree and Master degree in Electronic and
Communication Engineering from the
Harbin Institute of Technology (HIT) in
2013 and 2015, respectively. Now he is a
Ph.D. candidate in the School of Electrical,
IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019