IEEE Computational Intelligence Magazine - May 2023 - 64

FIGURE 6. An illustration of knowledge distillation. FKT refers to Feature-based Knowledge
Transfer, RKT refers to Relation-based Knowledge Transfer, and LKT refers to Logits-based
Knowledge Transfer.
also be extended to gradient computations.
DoReFa-Net [46] quantized gradients
to numbers with bit-width of less
than 8 before back-propagation. It allows
quantization of the network during training
or fine-tuning. Instead ofquantizing a
single weight, vector quantization
approaches focused on factorizing the
weight matrix by weight clustering and
sharing [47], [48].
Instead ofonly using pruning or quantization
for network compression, a combination
ofthese two have generally been
reported to lead to higher compression
rate.Han et al.presented athree-stage
pipeline which consists ofpruning, weight
sharing by vector quantization, and Huffman
encoding to compress a pre-trained
model, and it achieved state-of-the-art
performance at that time [49]. Both pruning
and quantization are effective ways to
reduce model size for efficient learning.
One limitation is that they can only deal
with models that share a common network
architecture, which means compressing
a cumbersome network to an
efficient one with less weights (or neurons)
and/or lower-precision. They cannot be
used across heterogeneous networks,
which can otherwise be addressed with
the knowledge distillation technique.
3) Knowledge Distillation
Knowledge distillation (KD) is a flexible
yet efficient approachtocompressAI
models. It aims to transfer the knowledge
learnt from a cumbersome network
to a compact one, which is also known
as 'Teacher-Student' learning framework
[50].Withthe knowledgefroma
teacher, a lightweight student network
can achieve competitive performance as
the cumbersome (teacher) network yet at
a higher computational-efficiency. The
successofKD reliesmainly ontwo key
factors: the knowledge and distilling
schemes.
The knowledge information can be
categorized into three different types: logits-based,
feature-based, and relationbased
knowledge [51],which aredepicted
in Fig. 6. The vanilla KD approach leverages
softened logits from a cumbersome
teacher as the knowledge to guide the
training process ofa student [52].Romero
et al. extended it by introducing the 'hint'
knowledge from intermediate layers of a
teacher network, which can be termed
feature-based knowledge [53].Instead of
directly minimizing the discrepancy offeature
maps between the teacher and the
student, other advanced feature-based
knowledge has also been explored. For
instance, Zagoruyko and Komodakis
regarded the teacher's attention maps
derived from feature maps as the knowledge
to be transferred [54]. Different from
the aforementioned two knowledge
types, relation-basedKDmethods emphasize
the transfer of intra-relationships
between data samples [55], [56], [57] or
correlations between feature maps from
multiple intermediate layers [58], [59].
Along with this knowledge information,
various distilling schemes have also
been proposed. Although most KD
64 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2023
methods follow the conventional singleteacher
single-student distilling scheme,
some other promising schemes have also
been explored in the literature. The
scheme of distilling informative knowledge
from an ensemble ofteachers instead
of a single teacher could better enhance
the student's generalization ability [60],
[61]. Apart from distilling knowledge
from large-scale networks, another
research direction intends to boost the
performance of a compact network by
transferring the knowledge between networks
which share identical architecture
(also termed as self-knowledge distillation)
[62]. Furlanello et al. proposed a
sequential distilling scheme called BornAgain
Networks (BANs), which takes the
model in earlier generation as the teacher
and trains a new initialized identical
model [62], [63]. To better align feature
representations between high-capacity
and low-capacity networks, adversarial
and contrastive distilling schemes are often
adopted. To demonstrate the feasibility of
transferring knowledge between two disparate
network architectures, Xu et al.
exploited adversarial distilling scheme to
transfer knowledge from a much bigger
LSTM-based teacher to a smaller CNNbased
student [64].
Generally, KD methods are more
flexible. The teacher and student networks
inKDcan be homogeneous or heterogeneous
architectures, which is
different from pruning and quantization.
However, in most cases, KD methods
require more training efforts, as they need
to train both the teacher and student networks.
In fact, most of the training efforts
have been given to the teacher network
whichisoften large insizeand complex.
To solve this issue, one way is to consider
adopting a trained network as the
teacher [65]. Building a shared community
for sharing trained AI models can significantly
reduce repeated model training
efforts, which can be an interesting direction
for sustainable AI.
Computation-efficient AI can lead to
compressed AI models which improve
environmental sustainability of AI in two
main aspects. Compressed models can be
deployed on resource-limited devices
(e.g.,embeddedsystems)withlessmemory
and computational power. This also

IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023