IEEE Circuits and Systems Magazine - Q1 2021 - 33

4-tuple " 2 n ! 1, 2 n + 1 ! 1 , takes advantage of the fact that
M in (9) is odd. Thus, [41] can be applied to perform the
comparison required to compute the max function in
(42) and (43). The fully RNS-based CNN accelerator in
[123], i.e., Res-DNN, is supported by the Eyeriss architecture [124] and the RNS arithmetic for the 3-moduli
set " 2 n - 1, 2 n, 2 n + 1 - 1 ,, preventing overflow by using
the BE algorithm referenced in II-B (extending 2 n to the
modulo 2 n + e ). A dynamic range partitioning approach is
adopted to implement the comparison operation in the
RNS domain [125], while for sign detection in the ReLU
unit, the MRC method is adopted [126].
Apart from the large amount of computing, CNNs also
require high memory bandwidth due to the large volume
of data that must be moved between the memory and
processing units. CNNs fully computed in the RNS domain are also proposed for PIM (RNSnet) [122]. RNSnet
simplifies the fundamental CNN operations and maps
them with the support of RNS to in-memory addition and
data access. The nonlinear activation functions are approximated using a few of the first terms of the Taylor expansion, and the comparison for pooling is implemented
only for addition by taking advantage of the characteristics of the traditional 3-moduli set " 2 n - 1, 2 n, 2 n + 1 , .
By following a different approach, the fact that
weights and activations in a CNN naturally have nonuniform distributions can be used to reduce the bit-width
of a data representation and simplify the arithmetic [12],
[127]. Inspired by the Kulisch approach [128], which removes the sources of rounding errors in a dot product
through an exact multiplier and wide fixed-point accumulator, products and accumulation can be performed
in both the logarithmic and the linear domains [127].
This approach works for the FP and fixed-point representations, reducing the data bit-width through nonuniform quantization.
Assuming that translating the number m. f from
the logarithmic to the linear domains is easier than
computing log 2 ^1 ! 2 x h (6), Exact Log-linear Multiply
Add (ELMA) [127] performs multiplications in the

logarithmic domain and then approximates results
to the linear domain floating-point representation for
accumulation. Translating m imposes a multiplication
by 2 m in the linear domain, a floating-point exponent
addition or a fixed-point bit shift. If the number of
bits of the fractional part f is small, then LUTs can be
used in practice to map from the logarithmic to the
linear domain and from the linear back to the logarithmic domain.
The tapered floating point, for which the exponent
and significand field sizes vary according to a third field,
such as the posit number system [129], has also been
adopted to design CNNs. The posit number system is
characterized by (N, s), the word length in bits (N ) and
the exponent scale (s). The results show that the accuracy of the multiply and add operations with the
proposed ELMA is approximately the same as that
of t he posit (8, 1) and leads to very efficient hardware implementations.
Fig. 24 shows the samples and the weights for
c o n volution represented in the log- domain [12]
ui=
a s xu i = Quantize ^log 2 ^ x i hh ( 3 - b i t w i d t h) a n d w
Quantize ^log 2 ^w i hh (4-bit width), respectively.
u ih
conv = / 2 xu i + wu i = / BitShift ^1, xu i + w
i

(44)

i

In (44), the accumulation is still performed in the linear domain, with BitShift(1, z) denoting the function that
bit-shifts 1 by an integer z in the fixed-point arithmetic.
To reach the method in Fig. 24, the accumulation is also
performed in the log-domain by using the approximation log 2 ^1 + x h . x for 0 # x 1 1. By considering the
u 1 and
accumulation of only two terms, i.e., pu 1 = xu 1 + w
u 2 , (44) can be rewritten as (45). (45) can be
pu 2 = xu 2 + w
easily generalized for any number of terms, i and pu i ,
leading to the complete computation in the logarithmic
domain of the convolution layers and the remaining layers of the CNN [12]. Moreover, an end-to-end training
procedure based on the logarithmic representation is
proposed in [12].

log2 w
4 Bits

log2 x
3 Bits
From Memory
Small Bandwidth

ReLU
(43)

log2 x
3 Bits
To Memory
Small Bandwidth

Figure 24. Computation of the dot product and ReLU in the LNS domain.

FIRST QUARTER 2021

IEEE CIRCUITS AND SYSTEMS MAGAZINE

33
IEEE Circuits and Systems Magazine - Q1 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q1 2021
