IEEE Computational Intelligence Magazine - August 2018 - 55

Review
Article

Tom Young
School of Information and Electronics, Beijing Institute
of Technology, Beijing, China
Devamanyu Hazarika
School of Computing, National University of Singapore,
Singapore
Soujanya Poria
Temasek Laboratories, Nanyang Technological University,
Singapore
Erik Cambria
School of Computer Science and Engineering,
Nanyang Technological University, Singapore

Recent Trends in Deep Learning Based Natural Language Processing
Abstract

eep learning methods employ
multiple processing layers to learn
hierarchical representations of
data, and have produced state-of-the-art
results in many domains. Recently, a
variety of model designs and methods
have blossomed in the context of natural language processing (NLP). In this
paper, we review significant deep
learning related models and
methods that have been
employed for numerous
NLP tasks and provide a
walk-through of their
evolution. We also summar ize, compare and
contrast the various models
and put forward a detailed
understanding of the past, present
and future of deep learning in NLP.

I. Introduction

Natural language processing (NLP) is a
theory-motivated range of computational techniques for the automatic
analysis and representation of human
language. NLP research has evolved
from the era of punch cards and batch

Digital Object Identifier 10.1109/MCI.2018.2840738
Date of publication: 18 July 2018

creasingly focusing on the use of new deep
learning methods (see Fig. 1). For decades,
machine learning approaches targeting
NLP problems have been based on shallow models (e.g., SVM and logistic regression) trained on very high dimensional and
sparse features. In the last few years, neural
networks based on dense vector representations have been producing superior
results on various NLP tasks.This
trend is sparked by the success
of word embeddings [2], [3]
and deep learning methods [4]. Deep learning
enables multi-level automatic feature representation learning. In contrast,
traditional machine learning
based NLP systems liaise heavily
Image lIcensed by Ingram PublIshIng
on hand-crafted features. Such handcrafted features are time-consuming and
processed in less than a second [1]. NLP
often incomplete.
enables computers to perform a wide
Collobert et al. [5] demonstrated that
range of natural language related tasks at
a simple deep learning framework
all levels, ranging from parsing and partoutperforms most state-of-the-art apof-speech (POS) tagging, to machine
proaches in several NLP tasks such as
translation and dialogue systems.
named-entity recognition (NER), seDeep learning architectures and almantic role labeling (SRL), and POS
gorithms have already made impressive
tagging. Since then, numerous complex
advances in fields such as computer vision
deep learning based algorithms have
and pattern recognition. Following this
been proposed to solve difficult NLP
trend, recent NLP research is now inprocessing, in which the analysis of a
sentence could take up to 7 minutes, to
the era of Google and the likes of it, in
which millions of webpages can be

Corresponding Author: Erik Cambria (cambria@ntu.edu.sg)

august 2018 | ieee Computational intelligenCe magazine

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2018