IEEE Computational Intelligence Magazine - February 2020 - 27

the neighborhood together for an image, and the word embedding
technique puts words with a similar meaning closer to each other
in a word space. However, the structured data with categorical
features may not have continuity. The traditional method is to use
one-hot encoding. For example, when we have 1,000 base stations,
we need a 1000-dimension vector to represent a base station, i.e.,
one single 1 and all the others 0. Naively applying neural networks does not work well due to the following reasons:
❏ When we have many high cardinality features, one-hot
encoding often results in an unrealistic amount of computational resource requirement. In our example, the dimension
of input data is at least 179 + 9, 720 + 684 = 10, 583. Moreover, the convergence of the deep neural networks cannot
be guaranteed.
❏ Different values of categorical variables are treated independently, and some informative relations are ignored.
In our example, the time interval of two entries cannot
be captured.
❏ As different categorical variables have diverse dimensions, the
importance of some variables with the lower dimension will
be weakened. In our example, the dimension of the user feature is 9,720, and will dominate the feature representation.
To tackle this problem, we propose a novel deep neural network architecture [27] as illustrated in Fig. 2. It consists of four
layers. For the input layer, we utilize the one-hot encoding for
each type of data, e.g., user, time and website. The first hidden
layer is entity embedding layer. The one-hot encoding of each
type of data will be fully connected to the corresponding entity
embedding layer. All the output of the first hidden layer will be
fully connected to one dense layer. The output layer is the classification result. In comparison with the traditional deep neural
network, one type of embedding layer will transform the spare
one-hot encoding data to a low-dimension representation, and

classification performance using deep neural networks and traditional machine learning methods, by taking the interest prediction as an example. Similar analysis can be extended to other
mobile access pattern predictions.
B. Traditional Machine Learning Methods

Traditional machine learning algorithms consist of two major
steps, including feature extraction, and (un)supervised feature
learning. For the feature extraction, there are three types of features that are commonly used for mobile access pattern prediction, including:
❏ Original feature: There are three original features that we can
extract for each entry, including the timeslot (when this entry
occurs), the geo-location (where this entry occurs), and the
sojourn time (How long a user connects to a website).
❏ Temporal feature: Given a mobile user, we can extract two
temporal features for each entry, including the time interval
between two consecutive entries, and the time interval
between two consecutive entries for the same website.
❏ Spatial feature: Give a mobile user, we can extract two spatial
features for each entry, including the distance interval between
two consecutive records and the gyration of this user.
For the supervised learning step, there are many classification models, such as support vector machine, decision tree, random forest, and AdaBoost. We choose the random forest
method with 10 trees to build the classification model, which
achieves the best performance among these prevailing methods.
C. DNN-Based Method

In principle, a neural network can approximate any continuous
function and piecewise continuous function. Currently, the
unstructured data can be transformed into continuous variables.
For example, the convolutional neural networks group pixels in

Prediction

Softmax Function:
Classification

Linear Function:
Regression

Fully Connected Layer
Number of Users One-Hot-Encoding
Feature
Learning

Preprocessing

Raw Data

Fully Connected Layer

Linear

Embedding

Linear

Embedding

Time
19:24:35

User
[0, ..., 1, ..., 0]

Geo-Location
[33.14, 119.79]

Website
[0, ..., 1, ..., 0]

User A

[1, 0, 0]

User B

[0, 1, 0]

User C

[0, 0, 1]

Example of One-Hot Encoding

FIGURE 2 DNN architecture for mobile access pattern prediction. It consists of four layers, including one input layer, two hidden layers and one
output layers. For the two hidden layer, we first utilize one entity embedding layer to transform the spare one-hot encoding into a vector with
lower dimension for each type of input, and a dense layer to fully connected the neurons in the entity embedding layer.

FEBRUARY 2020 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - February 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2020