IEEE Computational Intelligence Magazine - November 2021 - 73

represents the existence possibility of a
digit class. Most importantly, a routingby-agreement
mechanism is executed
between PrimaryCaps and DigitCaps to
enlarge the active features and ignore the
negative ones.
Denoting the output of the i-th
c a p sule in PrimaryCaps as u , ii
{, ,, },
12 32f
a prediction vector uji;
t
calculated by multiplying ui
matrix Wij sized 81 .6#
j {, ,, }.
= / t ji;
! 12 10f
sucjij
i
2
sj
v =
j
where cij
1 + sj
2
s
s
j
j
,
(2)
is the coupling coefficient calculated
using the routing-by-agreement
mechanism, as shown in Equation (3).
The parameter bij
is 0 at the beginning
and then dynamically updated using
Equation (4) in a routing-by-agreement
mechanism. Usually, there are 2 routing
iterations for parameter-update.
cij = /exp ()
z
exp()
bij
biz
t
bb vu=+ $
ij
The vector vj
ij jj i
;
,
,
(3)
A. General Framework
This paper aims to embed additional
data into some elements (such as weights,
layer output, and coupling coefficients)
of CapsNets during training. Denote the
embedding function for one receiver as
PPM = (, ),fKx
where Px
(5)
is the element used for
(4) mm t201Tt
!
f
of each capsule in
DigitCaps represents a digit class, and the
norm represents the existence possibility
of digit class. The nonlinear function in
Equation (2) ensures that short vectors
shrink to almost zero length and that long
vectors shrink to a length close to 1.
In the next section, the redundancy in
network parameters of CapsNets is used
for data hiding, and a data hiding scheme
with multiple receivers is proposed.
III. Proposed Method
In this section, a data hiding scheme is
proposed to transmit data with multiple
receivers via CapsNets as an example. It is
easy to embed additional data in a trained
CapsNets by directly modifying the network
weights, which is widely used for
data hiding in digital images. However,
the modifications will decrease the detection
accuracy of the original network. For
embedding, which will be discussed
in the next subsection, and K is the
embedding key possessed by the receiver.
Given a data vector
f
M [( ),m 1=
PMm mm [( ), (),, ()][ ,]
= pp pt12
! 01
(),, ()]{ ,} , the output
Tt
of the embedding function should be
close to M as much as possible since the
elements in PM
are decimals. Meanwhile,
the detection accuracy of CapsNets
should not be visibly decreased.
Without loss of generality, the embedding
function for the case of n receivers
is defined as
{, ,, }( ,,, ,).ff=
(6)
12 n
1
pmr(i )
0.5
pmr(i )
mr (i )
FIGURE 4 Redundant space of the output pimr
().
Redundancy
!
is
by a weight
Then, the output
vj of the j-th capsule in DigitCaps
is calculated using Equations (1) and
(2),
,
(1)
In this paper, a data hiding scheme is proposed to
transmit data with multiple receivers via CapsNets
as an example.
this reason, the authors propose embedding
additional data during the process of
training. Satisfactory detection accuracy of
CapsNets containing additional data can
be guaranteed this way. In the proposed
scheme, the parameters of data decoding
networks are generated using the embedding
keys possessed by the receivers.
Therefore, it is unnecessary to store and
transmit the decoders using covert channels.
Details are as follows.
Similarly, {, ,, }KK Kn12
f are the
embedding keys possessed by the n
receivers. For the outputs of the embedding
function, vectors {, ,, }
should be close to {, ,, },MM Mn12
= [( ), (),, ()]T
Mm mmrr rr
12
PP PMM Mn12
f
f
where M mm mt12rr rr
{, }, [( ), (),, ()]T
[, ], {, ,, }.rn01
01 pp pt12Pt
t ! f
=
f
f
!
!
To achieve the function, the total loss
L of CapsNets containing additional
data is defined as the weighted summation
of two parts, as shown in Equations
(7) and (8). Lx
original CapsNets [8], which is used to
guarantee satisfactory detection accuracy
on the MNIST dataset. LM
is used to
facilitate the embedding of additional
data, and b is used to adjust the weights
of the two parts. A large value of b is
advantageous to data extraction but disadvantageous
to detection accuracy, and
vice versa. The value of b will be discussed
in Subsection IV-A. In addition,
the extraction error e can be calculated
using Equation (9), where e 0= means
that the additional data are extracted
correctly and
round (·)
rounding operation.
LL .LM
=+b $
n
L
e
MM r
r
==
n
1
nt
r
==
((
))
1
1 / / roundp im i() . (9)mrr
r=1 i
n
t
Since the value range of pimr () is
mir
PP PPMM Mn12xfK KK [0, 1] while () is binary ( " 0 " or " 1 " ),
there is a redundant space of pimr (). As
x
/ PM .
1
2
stands for the
(7)
(8)
is the loss defined in the
mr (i )
Redundancy
NOVEMBER 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 73
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
