IEEE Computational Intelligence Magazine - November 2022 - 36

TABLE V Relationships of vision-action semantics.
PROPERTY
OBJECT
Red circle
Blue circle
Red polygon
Orange circle
White rectangle
Yellow triangle
Basketball
Football
Sofa
Orange
Cup
Sign
ACTION
Play
Kick
Sit
Eat
Drink
Watch
TABLE VI Relationships between gestures and action
semantics.
Action
Play Kick
Sit
Gesture Down Up Right
Eat
Drink Watch
Left Near
Far
Fig. 12 illustrates detailed steps of the proposed spiking
BAM model. It consists of two spiking BAM networks and a
series of encoding, associative learning, and decoding processes.
The solid arrow points to the forward learning process.
The black box represent the results of the DSSD model and
the SNN-based gesture recognition method. The blue box
represents the semantic information, the green box represents
the binary code that transformed from the semantic information
according to the ASCII code (eight bits to encode one letter)
or decoded from the spike train, and the orange box
represents the spike train that encoded from the binarized data
or associated with the spiking BAM network. In the forward
learning process, the model receives the semantic information
ofobject properties from the DSSD model. The primary color
of the salient shape can be obtained from the color
similarity [27]. Fig. 13 shows the process of using the DSSD
model to detect salient objects' shape and color properties.
Then, the object property is encoded into a binary train using
the ASCII code. The binary train uses 8 bits to represent a letter
and has the fixed-length binary train (insufficient lengths
are filled with zeros). Here the number of effective neurons
Ne and the number of auxiliary neurons Na are set as 160 and
40, respectively. Next, the proposed encoding method is used
to encode the binary train into a spike pattern, which is fed
into the spiking BAM network to obtain the corresponding
spike pattern through association memories. Then, the semantic
information of objects can be obtained through decoding
operation. The decoded results are compared with the
FIGURE 13 Illustration the detection process of the DSSD model. (a)
shows original images without any processing. (b) shows masks after
extracting salient objects. (c) shows masks after threshold processing
and connected contour analysis. (d) shows the convex hull of the
obtained contours after the extraction of the major contours. (e) shows
the obtained mask after utilizing polygon approximation. (f) shows the
results of the number of mask's edges and the primary color of the
original masked image after calculation.
FIGURE 12 Illustration of the structure of the proposed model with two spiking BAM networks, where the solid arrows denote the forward
computing, and the hollow arrows denote the backward computing.
36 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2022

IEEE Computational Intelligence Magazine - November 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2022