IEEE Computational Intelligence Magazine - November 2022 - 30

3)The Spiking BAMModel
In the implementation of the spiking BAM model, two spiking
BAM networks are combined to learn two relationships:
one is the relationship between objects and their properties,
and the other is the relationship between objects and gestural
actions. Fig. 2 demonstrates the learned information relationships
in the learned spiking BAM model. When the pattern of
the white rectangle property is input to the BAM model, a
cup's pattern will be recalled from memories ofthe object and
its property. Through the association of memories between
object and action, the action pattern of the drink gesture will
be recalled. On the backward computing, when inputting the
pattern ofthe drink gesture, the object's pattern ofthe cup and
the property's pattern of the white rectangle will be retrieved.
The detailed steps for learning the spiking BAM network are
described in Section V.
FIGURE 2 Illustration of learning relationships with spiking BAM
networks. The solid arrows denote the forward computing, while the
hollow arrows denote the backward computing.
1)Deeply Salient Shape Detection (DSSD)
The DSSD method can automatically detect salient objects
and extract their properties from a complex environment. The
salient object detection method based on a deep convolutional
network from [19] is used as a plugin of the DSSD process to
extract the saliency maps of the objects. After a series of morphological
processes, the polygon estimation [20] is applied on
the convex hull map to get the salient shape according to the
edges. Three edges represent a triangle, four edges represent a
rectangle, five to seven edges represent a polygon, and more
than eight edges represent a circle. Due to the computing limitations
of the NAO robot, the DSSD model is deployed on
the cloud GPU server, providing the NAO robot with realtime
detection ability.
2) Spike-Based Gesture Recognition
The spike-based gesture recognition method tracks an
object in a subject's hand and uses the tracked motion trajectories
as human gestures. An SNN-based gesture recognition
method is proposed to get the semantic information
of human gestures. The detailed process is described in
Section IV.
III. XOR-Phase Coding
Coding methods that convert input information into binary
spikes are the foundation component for SNNs. An XORphase
coding method is proposed to encode gestural trajectories
and input/output patterns of the spiking BAM network.
Different from the phase encoding method proposed in [18],
the shift direction ofthe encoding unit's membrane oscillation
is determined by an XOR operation between the activation
state and the position state of the input information, rather
than the input information's activation value (e.g., binary
image's black/white pixel) only. The proposed XOR-phase
coding method makes the spike pattern of the input information
more evenly distributed and keeps enough afferent spikes
to achieve the high recall accuracy. Fig. 3 shows the architecture
of the XOR-phase coding method. Firstly, each element
of input information (e.g., binary image) corresponds to an
effective afferent neuron, where each effective neuron has an
activation state and a position state (each state value is equal to
0 or 1):
❏ Si
a represents the activation state of each neuron, which is
determined by the intensity value ofthe corresponding element.
Si
a ¼ 1 means i-th afferent neuron is activated.
❏ Si
p represents the position state of each neuron, where the
position states of different neurons represent the phase
oscillation of the input information. All the position states
have periodic switching in the index of afferent neurons,
FIGURE 3 Illustration of the proposed coding method. Afferent neurons firstly get their corresponding activation states and position states from the
input information. Then the corresponding firing states can be obtained through an XOR operation between activation states and position states. The
encoding neuron will fire a spike according to the firing state.
30 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2022

IEEE Computational Intelligence Magazine - November 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2022