IEEE Solid-States Circuits Magazine - Fall 2021 - 67

We show that circulant matrices generated
from generator polynomials, as
in cyclic error-correcting codes, give a
set of solutions for the bi-adjacency
matrices of the layers that satisfy all
of the requisites in our problem statement.
From here on, we call these
graphs CSC architectures.
Al
Suppose the bi-adjacency matrix
## - of every factor()
01
lL
ized
layer l in our CSC graph has a
generator polynomial ()
pxl
that has
F terms, which generates a cyclic biadjacency
matrix of block length N. It
can be shown that the product of the
L bi-adjacency matrices attributed
to the L layers is a matrix
A (, )ijT
A ,T
where
represents the number of
paths between the Input node i and
Output node j of the CSC graph.
In the problem statement, the
connectivity should be equal to C
for all arbitrary pairs of Input and
Output nodes, and, thus,
AJC ,
T =
where J is an N-by-N all-one matrix.
The all-C matrix AT
can be attributed
to another generator polynomial
()
px CxT
= R =0
N-1
i
i, which constructs
a cyclic matrix as follows: the first
row of the matrix corresponds to
the coefficients of the polynomial-
represented as big-endian in this
article-and, then, every next row
is a cyclic right shift of its previous
row. The cyclic bi-adjacency matrices
in this article are not ideals, as
in ideal cyclic codes, and might have
nondistinctive rows. We provide two
different factorization sets of (),pxT
connectivity equal to one (CSCI) and
layers equal to two (CSCII).
CSCI
If
=
C ,1= and F, L, and N are such that
NF ,L
then ()= R =i 0 x can be
px
T
factorized as follows:
Z
[
\
]
]
N-1
i 0
=
l
]] == l
Dil
/ %
/
px xD F
.
F
i
=
-
1
l
pxl
() ,.
xp x
L
i
=
l
=
-
1
l ()
(6)
By assigning () and () to thepx1
px0
By assigning () as the generator
polynomial of the factorized
layer l, the CSC layers of the graph
are completely defined. Dl
is the
first and second layers of the graph, respectively,
the CSCII graph is defined.
In CSCII graphs,
EN ,NC2=
declares compression given CN /41
and a computation of ()O NN .
which
N-1
i
The second motivation of devising a CSC
architecture is to develop and employ
structurally compact models that require no
indexing for low-precision neural networks.
dilation parameter that indicates
the distance between elements of
value one in the first row of the
bi-adjacency matrix of layer l.
It
also represents the dilation between
the fanned-out edges in its corresponding
layer. For small values of
F near Napier's constant (e.g., two
or three) and, consequently, more
layers (e.g., log N2
or log N3
), the
least number of synapses results
but at the expense of elongating the
graph, which, by itself, will increase
the memory communication during
hardware implementation.
Figure 1(d) shows a CSCI graph
with three layers and generator polynomial
()
pxl
= R =
-
j
2 1 l
x
2NN2log
2 j
for the layer l.
For F 2= and C ,1= the numbers of
edges and MAC operations are equal
to
, which makes the comO
logNN2
putation complexity ().
We evaluate LeNet-300-100, which is
an early neural network proposed by
Yann LeCun, on MNIST, which is a data
set of handwritten digits, by replacing
FC layers with CSCI.
CSCII
If
satisfy NC ,F2
L ,2= and F, C, and N are integers to
N-1
= then ()= R =0
i
can be factorized as follows:
N==
Z
]
]
]
]
[
\
]
]
]]
NCx
px px ().( )( 1)
i=
/
1
px xD
px xD
Fi=
F
0()
1
1()
,.1.
i
/
/
=
-
1
Di
1
C
F
Di
0.
,
i
===
==
01
mod xN
1
(7)
YW ij jXW X
j
[] ()[] /ii [, ][ ], (8)
=
1
which is a computation of O( ).N2
If
W is factorizable into L matrices, or,
correspondingly, if a cascade of L CSC
layers with linear activations as well
as parameters N and F are weighted to
equate a factorizable FC layer, then the
equivalent
WW . Because of
Ll
T
=P =
1
l
the associative property of matrix-matrix
multiplication, the computation of
WXT
can start from the rightmost matrix-vector
multiplication and propagate
to the leftmost matrix. As such,
IEEE SOLID-STATE CIRCUITS MAGAZINE
FALL 2021
67
px CxT
i
Figure 1(e) shows a CSCII graph
with C = 2 and generator polynomial
()
pxl
= R =
-
j
4 1 l
x
2 j
for layer l. In the
" CSC Architectures for FC Layers "
section, we evaluate replacing FC
layers of AlexNet, which is a popular
neural network proposed by Alex
Krizhevsky, with CSCII architectures.
CSC Architectures for FC Layers
In this section, we denote the computation
of weighted matrices
trained using a CSC architecture.
We then explain a bottom-up algorithm
to seek appropriate CSC
architectures. For simplicity, we ignore
the term bias in all equations
in this section and assume the numbers
of nodes in the I/O of the FC
layers are all equal to N. As a result,
vectors/matrices in this section are
YI ,RN
!
WI ,RNN
!
# XI .RN
!
If layers
are not equally sized, we adjust
their I/O according to the " I/O Adjustment "
section.
A standard FC layer is representable
with a dense weight matrix W
whose operation on an input vector X
is equivalent to a matrix-vector multiplication
that generates a vector Y,
subject to
IEEE Solid-States Circuits Magazine - Fall 2021

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2021
