Computational Intelligence - May 2013 - 22

manikins [13], a tool to rate levels of arousal and valence in
discrete or continuous scales [14]) and introduce the use of
DL algorithms for preference learning, namely, preference
deep learning (PDL). In this paper, the PDL algorithm proposed is tested on emotional manifestations of relaxation, anxiety, excitement, and fun, embedded in physiological signals (i.e.,
skin conductance and blood volume pulse) derived from a
game-based user study of 36 participants. The study compares
DL against ad-hoc feature extraction on physiological signals,
used broadly in the AC literature, showing that DL yields
models of equal or significantly higher accuracy when a single
signal is used as model input. When the skin conductance and
blood volume pulse signals are fused, DL outperforms standard
feature extraction across all affective states examined. The
supremacy of DL is maintained even when automatic feature
selection is employed to improve models built on ad-hoc features; in several affective states the performance of models built
on automatically selected ad-hoc features does not surpass or
reach the corresponding accuracy of the PDL approach.
This paper advances the state-of-the-art in affective modeling in several ways. First, to the best of the authors' knowledge, this is the first time deep learning is introduced to the
domain of psychophysiology, yielding efficient computational
models of affect. Second, the paper shows the strength of the
method when applied to the fusion of different physiological
signals. Third, the paper introduces PDL, i.e., the use of deep
ANN architectures trained on ranked (pairwise preference)
annotations of affect. Finally, the key findings of the paper
show the potential of DL as a mechanism for eliminating
manual feature extraction and even, in some occasions,
bypassing automatic feature selection for affective modeling.
II. Computational Modeling of Affect

Emotions and affect are mental and bodily processes that can
be inferred by a human observer from a combination of contextual, behavioral and physiological cues. Part of the complexity of affect modeling emerges from the challenges of finding
objective and measurable signals that carry affective information (e.g., body posture, speech and skin conductance) and
designing methodologies to collect and label emotional experiences effectively (e.g., induce specific emotions by exposing
participants to a set of images). Although this paper is only concerned with computational aspects of creating physiological
detectors of affect, the signals and the affective target values
collected shape the modeling task and, thus, influence the efficacy and applicability of dissimilar computational methods.
Consequently, this section gives an overview of the field
beyond the input modalities and emotion annotation protocols
examined in our case study. Furthermore, the studies surveyed
are representative of the two principal applications of AI for
affect modeling and cover the two key research pillars of this
paper: 1) defining feature sets to extract relevant bits of information from objective data signals (i.e., for feature extraction),
and 2) creating models that map a feature set into predicted
affective states (i.e., for training models of affect).

22

IEEE ComputatIonal IntEllIgEnCE magazInE | may 2013

A. Feature Extraction

In the context of affect detection, we refer to feature extraction as
the process of transforming the raw signals captured by the hardware (e.g., a skin conductance sensor, a microphone, or a camera)
into a set of inputs suitable for a computational predictor of affect.
The most common features extracted from unidimensional continuous signals-i.e. temporal sequences of real values such
as blood volume pulse, accelerometer data, or speech-are simple
statistical features, such as average and standard deviation values,
calculated on the time or frequency domains of the raw or the
normalized signals (see [15], [16] among others). More complex
feature extractors inspired by signal processing methods have also
been proposed by several authors. For instance, Giakoumis et al.
[17] proposed features extracted from physiological signals using
Legendre and Krawtchouk polynomials while Yannakakis and
Hallam [18] used the approximate entropy [19] and the parameters
of linear, quadratic and exponential regression models fitted to a
heart rate signal.The focus of this paper is on DL methods that can
automatically derive feature extractors from the raw data, as
opposed to a fixed set of hand-crafted extractors that represent
pre-designed statistical features of the signals.
Unidimensional symbolic or discrete signals-i.e., temporal
sequences of discrete labels, typically events such as clicking a
mouse button or blinking an eye-are usually transformed
with ad-hoc statistical feature extractors such as counts, similarly to continuous signals. Distinctively, Martínez and
Yannakakis [11] used frequent sequence mining methods [20]
to find frequent patterns across different discrete modalities,
namely gameplay events and discrete physiological events. The
count of each pattern was then used as an input feature to an
affect detector. This methodology is only applicable to discrete
signals: continuous signals must be discretized, which involves a
loss of information. To this end, the key advantage of the DL
methodology proposed in this paper is that it can handle both
discrete and continuous signals; a lossless transformation can
convert a discrete signal into a binary continuous signal, which
can potentially be fed into a deep network-DL has been successfully applied to classify binary images, e.g., [21].
Affect recognition based on signals with more than one
dimension typically boils down to affect recognition from
images or videos of body movements, posture or facial expressions. In most studies, a series of relevant points of the face or
body are first detected (e.g., right mouth corner and right
elbow) and tracked along frames. Second, the tracked points are
aggregated into discrete Action Units [22], gestures [23] (e.g., lip
stretch or head nod) or continuous statistical features (e.g.,
body contraction index), which are then used to predict the
affective state of the user [24]. Both above-mentioned feature
extraction steps are, by definition, supervised learning problems
as the points to be tracked and action units to be identified
have been defined a priori. While these problems have been
investigated extensively under the name of facial expression or
gesture recognition, we will not survey them broadly as this
paper focuses on methods for automatically discovering new or
unknown features in an unsupervised manner.



Table of Contents for the Digital Edition of Computational Intelligence - May 2013

Computational Intelligence - May 2013 - Cover1
Computational Intelligence - May 2013 - Cover2
Computational Intelligence - May 2013 - 1
Computational Intelligence - May 2013 - 2
Computational Intelligence - May 2013 - 3
Computational Intelligence - May 2013 - 4
Computational Intelligence - May 2013 - 5
Computational Intelligence - May 2013 - 6
Computational Intelligence - May 2013 - 7
Computational Intelligence - May 2013 - 8
Computational Intelligence - May 2013 - 9
Computational Intelligence - May 2013 - 10
Computational Intelligence - May 2013 - 11
Computational Intelligence - May 2013 - 12
Computational Intelligence - May 2013 - 13
Computational Intelligence - May 2013 - 14
Computational Intelligence - May 2013 - 15
Computational Intelligence - May 2013 - 16
Computational Intelligence - May 2013 - 17
Computational Intelligence - May 2013 - 18
Computational Intelligence - May 2013 - 19
Computational Intelligence - May 2013 - 20
Computational Intelligence - May 2013 - 21
Computational Intelligence - May 2013 - 22
Computational Intelligence - May 2013 - 23
Computational Intelligence - May 2013 - 24
Computational Intelligence - May 2013 - 25
Computational Intelligence - May 2013 - 26
Computational Intelligence - May 2013 - 27
Computational Intelligence - May 2013 - 28
Computational Intelligence - May 2013 - 29
Computational Intelligence - May 2013 - 30
Computational Intelligence - May 2013 - 31
Computational Intelligence - May 2013 - 32
Computational Intelligence - May 2013 - 33
Computational Intelligence - May 2013 - 34
Computational Intelligence - May 2013 - 35
Computational Intelligence - May 2013 - 36
Computational Intelligence - May 2013 - 37
Computational Intelligence - May 2013 - 38
Computational Intelligence - May 2013 - 39
Computational Intelligence - May 2013 - 40
Computational Intelligence - May 2013 - 41
Computational Intelligence - May 2013 - 42
Computational Intelligence - May 2013 - 43
Computational Intelligence - May 2013 - 44
Computational Intelligence - May 2013 - 45
Computational Intelligence - May 2013 - 46
Computational Intelligence - May 2013 - 47
Computational Intelligence - May 2013 - 48
Computational Intelligence - May 2013 - 49
Computational Intelligence - May 2013 - 50
Computational Intelligence - May 2013 - 51
Computational Intelligence - May 2013 - 52
Computational Intelligence - May 2013 - 53
Computational Intelligence - May 2013 - 54
Computational Intelligence - May 2013 - 55
Computational Intelligence - May 2013 - 56
Computational Intelligence - May 2013 - 57
Computational Intelligence - May 2013 - 58
Computational Intelligence - May 2013 - 59
Computational Intelligence - May 2013 - 60
Computational Intelligence - May 2013 - 61
Computational Intelligence - May 2013 - 62
Computational Intelligence - May 2013 - 63
Computational Intelligence - May 2013 - 64
Computational Intelligence - May 2013 - 65
Computational Intelligence - May 2013 - 66
Computational Intelligence - May 2013 - 67
Computational Intelligence - May 2013 - 68
Computational Intelligence - May 2013 - 69
Computational Intelligence - May 2013 - 70
Computational Intelligence - May 2013 - 71
Computational Intelligence - May 2013 - 72
Computational Intelligence - May 2013 - 73
Computational Intelligence - May 2013 - 74
Computational Intelligence - May 2013 - 75
Computational Intelligence - May 2013 - 76
Computational Intelligence - May 2013 - 77
Computational Intelligence - May 2013 - 78
Computational Intelligence - May 2013 - 79
Computational Intelligence - May 2013 - 80
Computational Intelligence - May 2013 - Cover3
Computational Intelligence - May 2013 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com