IEEE Signal Processing Magazine - January 2018 - 128

16-bit fixed-point representation in stochastic rounding-based
er drawback of these binary nets is that existing binarization
CNN training, which significantly reduced memory usage and
schemes are based on simple matrix approximations and ignore
float- point operations with little loss in classification accuracy.
the effect of binarization on the accuracy loss. To address
The method proposed in [10] first pruned the unimportant conthis issue, the work in [17] proposed a proximal Newton algonections and retrained the sparsely connected networks. Then it
rithm with diagonal Hessian approximation that directly miniquantized the link weights using weight-sharing, and then applied
mizes the loss with respect to the binary weights. The work in
Huffman coding to the quantized weights as
[18] significantly reduced the time on floatwell as the codebook to further reduce the
point multiplication in the training stage by
Network pruning and
rate. As shown in Figure 1, it starts by learnstochastically binarizing weights and consharing has been used
ing the connectivity via normal network trainverting multiplications in the hidden state
ing, followed by pruning the small-weight
computation to sign changes.
both to reduce network
connections. Finally, the network is retrained
complexity and to address
to learn the final weights for the remaining
Pruning and sharing
the overfitting issue.
sparse connections. This work achieves the
Network pruning and sharing has been used
state-of-the-art performance among all paramboth to reduce network complexity and to
eter quantization-based methods. It was shown in [11] that Hesaddress the overfitting issue. An early approach to pruning was
sian weight could be used to measure the importance of network
biased weight decay [19]. The optimal brain damage [20] and
parameters and proposed to minimize Hessian-weighted quantizathe optimal brain surgeon [21] methods reduced the number
tion errors on average for clustering network parameters. A novel
of connections based on the Hessian of the loss function, and
quantization framework was introduced in [12], which reduced the
their works suggested that such pruning gave higher accuracy
precision of network weights to ternary values.
than magnitude-based pruning such as the weight decay methIn the extreme case of 1-bit representation of each weight, i.e.,
od. Those methods supported training from scratch.
binary weight neural networks, there are also many works that
A recent trend in this direction is to prune redundant, nondirectly train CNNs with binary weights; for instance, Binaryinformative weights in a pretrained CNN model. For example,
Connect [13], BinaryNet [14], and XNORNetworks [15]. The
Srinivas and Babu [22] explored the redundancy among neurons
main idea is to directly learn binary weights or activations durand proposed a data-free pruning method to remove redundant
ing the model training. The systematic study in [16] showed that
neurons. Han et al. [23] proposed to reduce the total number of
networks trained with backpropagation could be robust against
parameters and operations in the entire network. Chen et al. [24]
(robust against or resilient to) specific weight distortions, includproposed a HashedNets model that used a low-cost hash function
ing binary weights.
to group weights into hash buckets for parameter sharing. The
deep compression method in [10] removed the redundant connecDrawbacks
tions and quantized the weights and then used Huffman coding
However, the accuracy of such binary nets is significantly lowto encode the quantized weights. In [25], a simple regularization
ered when dealing with large CNNs such as GoogleNet. Anothmethod based on soft weight-sharing was proposed, which

Cluster the Weights
Original
Network

Train Connectivity
Generate Codebook

Encode Weights

Quantize the Weights
with Codebook

Encode Index

Compressed
Network

Prune Connections

Train Weights
Retrain Codebook

FIGURE 1. The three-stage compression method proposed in [10]: pruning, quantization, and encoding. The input is the original model, and the output is
the compression model.

128

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018