IEEE Circuits and Systems Magazine - Q3 2020 - 28

In dynamic stochastic computing (DSC), varying signals are encoded
by random bit streams with variable probabilities, referred to as
dynamic stochastic sequences.
y i = F (w i, x i) = w i x i = / Mj =-01 w i x i - M + j + 1, (9)
j

	

where M is the length of the filter or the number of weights,
(0)
(1)
(M - 1)
w i is a vector of M weights, w i = [w i , w i , f, w i
]
and x i is the input vector for the digital signal sampled
at different time points, x i = [x i - M + 1, x i - M + 2, f, x i]T.
In the least-mean-square (LMS) algorithm, the cost function is the mean squared error between the filter output
and the guidance signal, {t i}. The weights are updated by
w i + 1 = w i + hx i (t i - y i), (10)

	

where h is the step size. The update is implemented by
the circuit in Fig. 22. The two multipliers are used to compute x i t i and x i y i , respectively. The integrator is then
used to accumulate hx i (t i - y i) over i = 0, 1, f

Trained
DSNG
Array

xi
yi

Parameters

×

+

ti

×

-

Figure 22. A DSC circuit training a weight in an adaptive filter. The stochastic multipliers are denoted by rectangles with
"×". Bipolar and sign-magnitude representations can be used
for the stochastic multipliers and integrator.

Magnitude (dB)

0

Target System
Adaptive Filter

-20
-40
-60
-80
0

0.5

1
1.5
2
2.5
Normalized Frequency

3

Figure 23. Frequency responses of the target and trained
filter using the sign-magnitude representation.
28 	

The dynamic stochastic circuit is used to perform
system identification for a high pass 103-tap FIR filter.
103 weights are trained by using randomly generated
samples with the LMS algorithm. 14-bit stochastic integrators are used. After around 340k iterations of training, the misalignment of the weights in the adaptive filter converges to about - 45 dB compared to the target
system. The frequency responses of the target and the
trained filter are shown in Fig. 23. As can be seen, the
adaptive filter has a similar frequency response to the
target system, especially in the passband, indicating an
accurate identification result.
D. A Gradient Descent (GD) Circuit
GD is a simple yet effective optimization algorithm. It
finds the local minima by iteratively taking steps along
the negatives of the gradients at current points. Onestep GD is given by
	

w i + 1 = w i - hdF (w i), (11)

where F (w i) is the function to be minimized at the ith
step, dF (w i) denotes the current gradient, and h is the
step size. The step size is an indicator of how fast the
model learns. A small step size leads to a slow convergence while a large one can lead to divergence.
Similar to the implementation of the Euler method, let a pair of differential DSS's encode the gradient
{- dF (w i)}, the stochastic integrator can be used to
compute (11) with a step size, h = 1/2 n, by using these
DSS's as inputs. DSC can then be used to perform the
optimization of the weights in a neural network (NN) by
minimizing the cost function.
The training of an NN usually involves two phases.
Take a fully connected NN as an example, shown in
Fig. 24. In the forward propagation, the sum of product
of the input vector x and the weight vector w is obtained
as v = wx. The output of the neuron, o, is then computed by o = f (v), where f is the activation function. During
this phase, the weights do not change. In the backward
propagation, the difference between the actual and the
desired final outputs, i.e., the classification results, is
calculated, and the local gradients are computed for
each neuron based on the difference [55]. Take the
neuron in Fig. 24(c) as an example, the local gradient is
obtained by d l = f l(v) w l + 1 d l + 1, where f l($) is the derivative of the activation function, w l + 1 is the weight vector

IEEE CIRCUITS AND SYSTEMS MAGAZINE 		

THIRD QUARTER 2020
IEEE Circuits and Systems Magazine - Q3 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2020
