Computational Intelligence - May 2013 - 53

model [14], they utilized an improved
SOM model named Habituated SelfOrganizing Map (HSOM) to cluster
Sensors
similar tasks and novelty is calculated by
External Stimuli
a habituation function.
Stateexternal
To summarize, in existing works,
Curiosity Appraisal
curiosity has been integrated into
Learnt
Learning of
Determination of
Emotions
Knowledge
agents' learning module and decision
Stimulation Level Reinforcement State-Action
Memory
module to enhance their perforMapping
mance. However, these agents can
Mapping from
Energy
hardly be perceived to be believable
Stimulation
Intrinsic
Level to Emotions
by a human observer. There are two
Constraint Agent's
main reasons for this: (1) existing
Actions
models lack a comprehensive psyUpdate
Emotions
chological theory as background, and
Influence
on Action Strength
(2) agents perceive environment on
the machine language level (featureUpdate
Actuators
based knowledge representation)
rather than on the human language
level (semantic knowledge represenFigure 1 Architecture of the curious peer learner.
tation). Hence, in this work, we
attempt to build a computational
[30]. Another function of curiosity-related emotions is their
model of curiosity based on human psychology and by
influence on the agent's knowledge acquisition ability. This is
adopting a semantic knowledge representation method.
inspired by human nature, where our learning ability can be
regulated by different emotion states [31].
IV. An Overview of Our Approach
An overview of the key innovations in our approach is given
as follows:
V. The Curious Peer Learner
First, to mimic a human student, a virtual peer learner
In this section, we present the proposed virtual peer learner
should perceive the VLE at the same level as a human stuwith curiosity-related emotions, referred to as curious peer
dent does. Hence, instead of feature-based knowledge reprelearner. Architecture of the curious peer learner is shown in
sentations, most commonly utilized in existing works, we
Fig. 1. It can be observed that the curious peer learner can
employ a semantic knowledge representation, that can easily
sense external states (e.g., in a learning zone) and receive exterbe interpreted by humans and is more suitable for designing
nal stimuli (e.g., learning tasks). The external stimuli can trigger
virtual peer learners. In this work, we adopt Concept Map
the curious peer learner to perform curiosity appraisal. The
(CM), a semantic knowledge representation stemming from
curiosity appraisal requires learnt knowledge stored in the
the learning theory of constructivism. It has been widely
agent's memory and consists of two steps: determination of
applied in classrooms for knowledge organization [25] and
stimulation level and mapping from stimulation level to emomany educational softwares for modeling the mind of stutions. Emotions derived from the curiosity appraisal process
dents [26], [27].
serve two functions: (1) as reinforcement value for the learning
Second, the measurement of stimulation level incorporates
of state-action mapping, and (2) as influence on action
three dimensions of information proposed by Berlyne [17],
strengths (e.g., the depth of learning). Actions (e.g., explore)
including novelty, conflict and complexity. The calculation of
derived from the learning of state-action mapping module are
stimulation level is based on an extension and transformation of
performed by actuators, and update intrinsic constraints of the
Tversky's ratio model [28].
agent (e.g., energy). In the rest of this section, detailed working
Third, we explicitly model three curiosity-related emotions:
mechanism of each module will be introduced.
boredom, curiosity and anxiety. They are appraised based on
Wundt's theory [18], by adopting two thresholds to divide the
A. Memory and Knowledge Representation
spectrum of stimulation into three emotion regions.
We adopt Concept Maps (CMs) to represent the semantic
Finally, curiosity-related emotions are utilized as intrinsic
knowledge in both learning tasks (knowledge to be learnt) and
reward functions to guide the virtual peer learner's learning
the agent's memory (knowledge already learnt).
of behavior strategy. This is inspired by the frequently adopted
A CM is a graph-based representation that describes semanassumption in intrinsically motivated reinforcement learning
tic relationships among concepts. It can be represented by a
that human decision-making process consists of maximizing
directed graph with nodes and edges interconnecting nodes.
positive emotions and minimizing negative emotions [29],
We formalize the symbolic representation of CMs as follows:

may 2013 | IEEE ComputatIonal IntEllIgEnCE magazInE

Table of Contents for the Digital Edition of Computational Intelligence - May 2013