IEEE Computational Intelligence Magazine - November 2022 - 28

I. Introduction
W
ith the continuous development of robotics,
robots have become more sophisticated in
mechanical construction. However, in the
field of intelligent systems, especially in the field
of cognitive development robotics, how to improve the intelligent
system to a higher level that is even comparable to
humans' cognitive capabilities? How to enable humans to cognitively
interact with the system in a more natural, friendly and
human-like way? It is still facing great challenges, even not in a
rapidly changing environment, and this is also the primary goal
of cognitive system research [1], [2], [3], [4], [5]. The realization
of such cognitive interaction requires the integration of
multiple disciplines such as sociology, relevance theory, cognitive
science, neuroscience, development and cognitive psychology.
Relevance theory [6] explores in depth the process of
communication and interaction between humans. It mentions
that people's interactions not only exchange thoughts but also
establish their cognitive environment. Take, for example, a
speech-impaired but well-hearing person interacts with a
common person. The common person says " hello " . Although
the speech-impaired person is unable to speak, he can turn
words like " hello " into relevant concepts and then respond
with gestures that express the related concepts which the common
can understand. This happens because they have established
the relations between the concepts in their cognitive
environment, or they have shared the cognitive environment.
It is necessary for the cognitive robot to establish a cognitive
environment in order to quickly infer human actions, languages,
or observed objects, thus improving the human-robot
interaction experience and pushing cognitive robotics into
many aspects ofour daily life.
Associative memory has been widely applied to the cognitive
environment ofdeveloping robots [7], [8]. A bidirectional
associative memory (BAM) is a hetero-associative memory
that has a two-layer nonlinear network. This bidirectional
recall capability of BAM facilitates the application of association
memory to a variety of human-robot interaction tasks.
In [9], the BAM model is deployed for robots to associate gestures
with object attributes. A maximum-likelihood-criterionbased
BAM network is proposed to associate between template
and searching area for tracking footballs in a robot football
match [10]. BAM can be extended to multidirectional
associative memory (MAM) with multi-modal information,
such as object, gesture and facial expression information [11].
Masuyama et al. use the quantum-inspired complex-valued
MAM network for associating cognitive information (i.e.,
facial expression, object, gesture, voice and biometric information),
emotion states and robot behaviors [12].
Associative memory is studied using spiking neural networks
(SNNs) due to their natural superiority in dealing with
spatial-temporal patterns [13] presents the associative memory
based on SNN for realizing the association ofdifferent patterns
ofblack dots on a screen for robots. Tang et al. propose a brainlike
system based on cognitive memory and mapping for navigation
and spatial memory tasks [14]. Hampo et al. associate signals
and predefined reactions by an associative memory
network in the form ofan SNN for real-world reasoning on a
28 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2022
mobile robot [15]. Although some works have focused on using
SNN-based associative memory (not bidirectional) to develop
cognitive robots, there is still scarce of using spiking BAM to
develop cognitive robots. A few studies focus on developing
algorithms for spiking BAM and have laid a good foundation
for developing cognitive robots with spiking BAM. Zamani
et al. propose a spiking BAM network with temporal coding
[16].Johnson creates a spiking BAM network by converting
a non-spiking BAM [17]. The above spiking BAM networks
cannot be directly developed on cognitive robots because their
encoding method could not effectively represent some semantic
information (e.g., vision or actions), which results in insufficient
learning ofthe spiking BAM network.
In this paper, a cognitive robotic model which learns associations
between vision and action semantics with a spiking
BAM network is proposed. First, vision semantic information
is obtained through the deeply salient shape detection (DSSD)
method. And a novel coding method is proposed for gesture
recognition and converting semantics into spiking patterns.
Action semantic information is obtained from an SNN with
the proposed coding method. Next, a supervised spike-based
learning rule is used to train the spiking BAM network to
establish associations between different spiking patterns.
Experimental results show the proposed spiking BAM network
achieves computational efficiency, relatively high convergence
speed and recall accuracy. The proposed spiking
BAM network can improve the robot's ability to infer human
intentions in its natural interactions with humans. The contributions
ofthis work are summarized as follows:
1) A novel spike coding method based on XOR operation
and phase coding is proposed for converting input information
into binary spikes. The proposed coding method
makes the spike pattern of the input information more
evenly distributed and keeps enough spikes to achieve high
recall accuracy.
2) A cognitive robotic model is developed based on a spiking
BAM network. The spiking BAM network can establish
associative memories effectively by a supervised spike-based
learning rule named precise-spike-driven (PSD) [18].
3) A cognitive robotic model is further deployed to implement
a human-robot interaction system. Based on the proposed
system, robots can autonomously associate vision
information with the recalled memories of actions in order
to make a response correspondingly.
The rest of the paper is organized as follows. Section II
illustrates the architecture of the proposed cognitive robotic
model, including a brief view of different modules of the
model. Section III gives a detailed description of the proposed
coding method. Section IV presents an SNN-based method
for gesture recognition. Section V describes the spiking BAM
network. The experimental results and the application on a
real physical robot NAO are shown in Section VI. The conclusions
and future works are given in Section VII.
II. The Human-Robot Interaction System
This section illustrates the human-robot interaction system based
on the proposed spiking BAM model. The developed system is
IEEE Computational Intelligence Magazine - November 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2022
