IEEE Circuits and Systems Magazine - Q3 2020 - 29

Using a stochastic integrator, DSC is well suited for implementing
accumulation-based iterative algorithms such as numerical
integration and gradient descent.

w
x

o = f(wx)
...

Hidden
Layers

...

Inputs

Outputs

(b)
wl+1
...

δl+1

V. Error Reduction and Assessment
One major issue in SC is the loss of accuracy [58]. Independent RNGs have been used to avoid correlation and
improve the accuracy of components such as stochastic
multipliers. This method inevitably increases the hardware overhead. However, in a stochastic integrator, the
sharing of RNGs for generating the two input sequences

0.5

Cross Entropy

0.4
0.3
0.2
0.1
0

0

5

10
Epoch
(a)

15

20

0

5

10
Epoch
(b)

15

20

98
96
94
92
90
88

Signed DSC
Bipolar DSC
Fixed-Point

...

o

wl

security-critical data on mobile devices if data are sensitive and cannot be uploaded to a cloud computer.

Testing Accuracy (%)

- onnecting layer l and l + 1, and d l + 1 is the local gradic
ent vector of the neurons in layer l + 1. Then, the gradient of a weight at layer l is computed by multiplying the
local gradient with the recorded output of the designated neuron, i.e., dF (w l) = - od l .
In [56], it is found that the local gradient can be decomposed into two parts d = d + - d -, where d + is contributed by the desired output of the NN and d - by the actual
output of the NN. So the gradient for weight w can be rewritten as dF (w) = - o ( d + - d - ). Similar to the implementation of (10), the DSC circuit in Fig. 22 can be used to implement the GD algorithm to perform the online training
(one sample at a time) of the fully connected NN with the
input signals d +, o and d -. However, a backpropagation,
i.e., d = f l(v) w l + 1 d l + 1 , is still computed by using a conventional method (e.g., a fixed-point arithmetic circuit) to
obtain the local gradients. Otherwise, the accuracy loss
would be too much for the algorithm to converge.
Fig. 25 shows the convergence of cross entropy (as
a cost function) and testing accuracy for the training of
a 784-128-128-10 NN with the MNIST handwritten digit dataset. As can be seen, the testing accuracy of the
NN trained by the DSC circuit using the sign-magnitude
representation is similar to the one produced by the
fixed-point implementation (around 97%). However, the
accuracy of the DSC implementation using the bipolar
representation is relatively low.
The proposed circuitry can be useful for online learning where real-time interaction with the environment is
required with an energy constraint [57]. It can also be
used to train a machine learning model using private or

(a)

(c)

Figure 24. (a) A fully connected NN, in which the neurons
are organized layer by layer. (b) The computation during the
forward propagation. (c) The local gradient for wl is computed
by using the backward propagation. The output o from previous neuron is required to compute the gradient.

THIRD QUARTER 2020 		

Figure 25. (a) Convergence of cross entropy and (b) testing
accuracy of MNIST hand-written digit recognition dataset.
Cross entropy is the cost function to be minimized during the
training of an NN. The weights of the NN are updated by the
DSC circuits. The DSC circuits using the sign-magnitude and
bipolar representations and the fixed-point implementation
are considered for comparison.

IEEE CIRCUITS AND SYSTEMS MAGAZINE	

29
IEEE Circuits and Systems Magazine - Q3 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2020
