IEEE Signal Processing Magazine - January 2018 - 131

in [42], the transforms T ($) and T l ($) are not necessarily
where T (·, x, y) denoted the translation of the first operthe same as they operate on different objects. According to
and by (x, y) along its spatial dimensions, with proper zero
this theory, it is reasonable to apply the transform to layers
padding at borders to maintain the shape. The proposed
or filters U ($) to compress the whole network models. From
framework can be used to 1) improve the classification accuempirical observation, deep CNNs also benefit from using a
racy as a regularized version of maxout networks and 2)
large set of convolutional filters by applying a certain transto achieve parameter efficiency by flexibly varying their
form T ($) to a small set of base filters since it acts as a reguarchitectures to compress networks.
larizer for the model.
Table 3 briefly compares the performance of different
Following this trend, there are many recent works proposed
methods with transferred convolutional filters, using VGGto build a convolutional layer from a set of base filters [42]-
Net (16 layers) as the baseline model. The results are report[45]. What they have in common is that the transform T ($)
ed on the CIFAR-10 and CIFAR-100 data sets with top-five
lies in the family of functions that only operate in the spatial
error rates. It is observed that they can achieve reduction in
domain of the convolutional filters. For
parameters with little or no drop in clasexample, the work in [44] found that the
sification accuracy.
The basic idea of KD is to
lower convolution layers of CNNs learned
distill knowledge from a
redundant filters to extract both positive and
Drawbacks
negative phase information of an input sigThere are several issues that need to be
large teacher model into
nal, and defined T ($) to be the simple negaaddressed for approaches that apply transfer
a small one by learning
tion function
information to convolutional filters. First,
the class distributions
these methods can achieve competitive peroutput by the teacher
T ^W x h = W -x .
(4)
formance for wide/flat architectures (like
via softened softmax.
VGGNet) but not narrow/special ones (like
Here, W x is the basis convolutional filter
GoogleNet and ResNet). Second, the transand W -x is the filter consisting of the shifts whose activation is
fer assumptions sometimes are too strong to guide the algoopposite to that of W x and selected after max-pooling operarithm, making the results unstable on some data sets.
tion. By doing this, the work in [44] can easily achieve 2# comUsing a compact filter for convolution can directly reduce
pression rate on all the convolutional layers. It is also shown that
the computation cost. The key idea is to replace the loose and
the negation transform acts as a strong regularizer to improve
overparametric filters with compact blocks to improve the
the classification accuracy. The intuition is that the learning
speed, which significantly accelerate CNNs on several benchalgorithm with pair-wise positive-negative constraint can lead
marks. Decomposing 3 # 3 convolution into two 1 # 1 conto useful convolutional filters instead of redundant ones.
volutions was used in [47], which achieved state-of-the-art
In [45], it was observed that magnitudes of the responses
acceleration performance on object recognition. SqueezeNet
from convolutional kernels had a wide diversity of pattern rep[48] was proposed to replace 3 # 3 convolution with 1 # 1
resentations in the network, and it was not proper to discard
convolution, which created a compact neural network with
weaker signals with a single threshold. Thus, a multibias nonapproximately 50 fewer parameters and comparable accuracy
linearity activation function was proposed to generate more
when compared to AlexNet.
patterns in the feature space at low computational cost. The
transform T ($) was define as
KD
To the best of our knowledge, exploiting knowledge transfer to
T l U ^xh = W x + d,
(5)
compress model was first proposed by Caruana et al. [49]. They
trained a compressed model with pseudo-data labeled by an
where d were the multibias factors. The work in [46] considensemble of strong classifiers and reproduced the output of the
ered a combination of rotation by a multiple of 90° and horioriginal larger network. However, their work is limited to shalzontal/vertical flipping with
low models. The idea has been recently adopted in [50] as KD
to compress deep and wide networks into shallower ones, where
T l U ^xh = W T ,
(6)
i

where W T was the transformation matrix that rotated the original filters with angle i ! {90, 180, 270}. In [42], the transform
was generalized to any angle learned from data, and i was
directly obtained from data. Both [46] and [42] can achieve
good classification performance.
Reference [43] defined T ($) as the set of translation functions applied to 2-D filters
i

T l U ^xh = T ^·, x, yhx, y ! " -k, f, k ,, ^ x, yh ! (0, 0),

(7)

Table 3. Comparisons of different approaches based on transferred
convolutional filters on CIFAR-10 and CIFAR-100.
Model

CIFAR-100

CIFAR-10

Compression Rate

VGG-16

34.26%

9.85%

1

MBA [45]

33.66%

9.76%

2

CRELU [44]

34.57%

9.92%

2

CIRC [42]

35.15%

10.23%

4

DCNN [43]

33.57%

9.65%

1.62

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|

131
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
