IEEE Computational Intelligence Magazine - August 2021 - 21

which trade-off the two objectives. The constraint on the
energy can be calculated based on the energy capacity and
battery life. Then, the solution that achieves the highest accuracy
will be selected as the optimal solution for the task.
Alternatively, one can select the knee point from the solution
set as the preferred solution.
VI. Conclusion
In this paper, an evolutionary multi-objective model compression
approach is proposed to accelerate and compress DNNs by
optimizing multiple objectives (e.g., accuracy, energy efficiency,
and model size) simultaneously. As the evaluation of each architecture
is extremely time-consuming during the evolution, a
two-stage pruning and optimization co-optimization strategy is
developed to speed up the architecture searching process.
Extensive experimental results demonstrate that the proposed
method can obtain a set of diverse networks in a single execution.
Furthermore, the proposed method outperforms the peer
methods in terms of energy efficiency and model size for model
compression of three popular DNNs.
Acknowledgement
This work is partly supported by the Agency for Science, Technology
and Research (A*STAR) under its AME Programmatic
Funding Scheme (No. A18A1b0045 and No. A1687b0033).
References
[1] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep Learning, vol. 1. Cambridge
MA: MIT Press, 2016, no. 2.
[2] K. Simonyan and A. Zisserman, " Very deep convolutional networks for large-scale
image recognition, " 2014. [Online]. Available: https://arxiv.org/abs/1804.09081
[3] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, " XLNet:
Generalized autoregressive pretraining for language understanding, " in Adv. Neural Inf.
Process. Syst., 2019, pp. 5753-5763.
[4] L. Zhen, P. Hu, X. Peng, R. S. M. Goh, and J. T. Zhou, " Deep multimodal transfer
learning for cross-modal retrieval, " IEEE Trans. Neural Netw. Learn. Syst., to be published.
doi: 10.1109/TNNLS.2020.3029181.
[5] H. Liu, K. Simonyan, and Y. Yang, " DARTS: Differentiable architecture search, " in
Proc. Int. Conf. Learn. Representations, 2018.
[6] Y. Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, and M. Zhang, " Surrogate-assisted evolutionary
deep learning using an end-to-end random forest-based performance predictor, " IEEE
Trans. Evol. Comput., vol. 24, no. 2, pp. 350-364, 2020. doi: 10.1109/TEVC.2019.2924461.
[7] Y. Sun, G. G. Yen, and Z. Yi, " Evolving unsupervized deep neural networks for learning
meaningful representations, " IEEE Trans. Evol. Comput., vol. 23, no. 1, pp. 89-103,
2018. doi: 10.1109/TEVC.2018.2808689.
[8] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Completely automated CNN architecture
design based on blocks, " IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 4, pp.
1242-1254, 2020. doi: 10.1109/TNNLS.2019.2919608.
[9] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Evolving deep convolutional neural networks
for image classification, " IEEE Trans. Evol. Comput., vol. 24, no. 2, pp. 394-407, 2020.
[10] S. Han, H. Mao, and W. J. Dally, " Deep compression: Compressing deep neural networks
with pruning, trained quantization and Huffman coding, " 2015. [Online]. Available:
https://arxiv.org/abs/1510.00149
[11] Y. Wang, C. Xu, J. Qiu, C. Xu, and D. Tao, " Towards evolutionary compression, "
in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2018, pp. 2476-2485.
[12] T.-J. Yang, Y.-H. Chen, and V. Sze, " Designing energy-efficient convolutional neural
networks using energy-aware pruning, " in Proc. IEEE Conf. Comput. Vision Pattern
Recognit., 2017, pp. 5687-5695.
[13] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, " HAQ: Hardware-aware automated quantization
with mixed precision, " in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2019.
[14] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, " Efficient processing of deep neural
networks: A tutorial and survey, " Proc. IEEE, vol. 105, no. 12, pp. 2295-2329, 2017. doi:
10.1109/JPROC.2017.2761740.
[15] Z. Wang, T. Luo, J. T. Zhou, and R. S. M. Goh, " EDCompress: Energy-aware model
compression with dataflow, " 2020. [Online]. Available: https://arxiv.org/abs/2006.04588
[16] Z. Du et al., " ShiDianNao: Shifting vision processing closer to the sensor, " in Proc.
Annu. Int. Symp. Computer Architecture, 2015, pp. 92-104.
[17] M. Song et al., " Towards efficient microarchitectural design for accelerating unsupervized
GAN-based deep learning, " in Proc. IEEE Int. Symp. High Performance Comput.
Architecture, 2018, pp. 66-77.
[18] N. P. Jouppi et al., " In-datacenter performance analysis of a tensor processing unit, "
in Proc. Annu. Int. Symp. Comput. Architecture, 2017, pp. 1-12.
[19] M. Alwani, H. Chen, M. Ferdman, and P. Milder, " Fused-layer CNN accelerators, "
in Proc. Annu. IEEE/ACM Int. Symp. Microarchitecture, 2016, pp. 1-12.
[20] J. Qiu et al., " Going deeper with embedded FPGA platform for convolutional neural
network, " in Proc. ACM/SIGDA Int. Symp. Field-Programmable Gate Arrays, 2016,
pp. 26-35.
[21] Y.-H. Chen et al., " Eyeriss: A spatial architecture for energy-efficient dataflow for
convolutional neural networks, " ACM SIGARCH Comput. Architecture News, vol. 44, no.
3, pp. 367-379, 2016. doi: 10.1145/3007787.3001177.
[22] H. Li, X. Fan, L. Jiao, W. Cao, X. Zhou, and L. Wang, " A high performance FPGAbased
accelerator for large-scale convolutional neural networks, " in Proc. Int. Conf. Field
Programmable Logic Appl., 2016, pp. 1-9.
[23] Y. Guo, A. Yao, and Y. Chen, " Dynamic network surgery for efficient DNNs, " in
Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 1379-1387.
[24] F. Manessi, A. Rozza, S. Bianco, P. Napoletano, and R. Schettini, " Automated
pruning for deep neural network compression, " in Proc. Int. Conf. Pattern Recognit., 2018,
pp. 657-664.
[25] C. Lemaire, A. Achkar, and P.-M. Jodoin, " Structured pruning of neural networks
with budget-aware regularization, " in Proc. IEEE Conf. Comput. Vision Pattern Recognit.,
2019, pp. 9108-9116.
[26] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, " Pruning filters for efficient
convnets, " 2016. [Online]. Available: https://arxiv.org/abs/1608.08710
[27] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, " AMC: AutoML for model
compression and acceleration on mobile devices, " in Proc. Eur. Conf. Comput. Vision, 2018,
pp. 784-800.
[28] Z. Liu, J. Xu, X. Peng, and R. Xiong, " Frequency-domain dynamic pruning for convolutional
neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 1043-1053.
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton, " ImageNet classification with deep
convolutional neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
[30] C. Szegedy et al., " Going deeper with convolutions, " in Proc. IEEE Conf. Comput.
Vision Pattern Recognit., 2015, pp. 1-9.
[31] K. Deb, " Multi-objective optimization, " in Search Methodologies. Springer-Verlag,
2014, pp. 403-449.
[32] J. D. Schaffer, " Multiple objective optimization with vector evaluated genetic algorithms, "
in Proc. Int. Conf. Genetic Algorithms, 1985, pp. 93-100.
[33] B. Li, J. Li, K. Tang, and X. Yao, " Many-objective evolutionary algorithms: A survey, "
ACM Comput. Surv., vol. 48, no. 1, 2015. doi: 10.1145/2792984.
[34] L. Zhen, M. Li, R. Cheng, D. Peng, and X. Yao, " Adjusting parallel coordinates for
investigating multi-objective search, " in Proc. Int. Conf. Simulated Evol. Learn., Shenzhen,
China, 2017, pp. 224-235.
[35] M. Li, L. Zhen, and X. Yao, " How to read many-objective solution sets in parallel
coordinates, " IEEE Comput. Intell. Mag., vol. 12, no. 4, pp. 88-100, 2017. doi: 10.1109/
MCI.2017.2742869.
[36] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley
Longman Publishing, 1989.
[37] N. Srinivas and K. Deb, " Muiltiobjective optimization using nondominated sorting
in genetic algorithms, " Evol. Comput., vol. 2, no. 3, pp. 221-248, 1994. doi: 10.1162/
evco.1994.2.3.221.
[38] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, " A fast and elitist multiobjective
genetic algorithm: NSGA-II, " IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182-197,
2002. doi: 10.1109/4235.996017.
[39] C. A. C. Coello, S. G. Brambila, J. F. Gamboa, M. G. C. Tapia, and R. H. Gómez,
" Evolutionary multiobjective optimization: Open research areas and some challenges lying
ahead, " Complex Intell. Syst., vol. 6, no. 2, pp. 221-236, 2020.
[40] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf,
" NSGA-Net: Neural architecture search using multi-objective genetic algorithm, " in
Proc. Genetic Evol. Comput. Conf., 2019, pp. 419-427.
[41] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, " Efficient neural architecture
search via parameters sharing, " in Proc. Int. Conf. Mach. Learn., 2018, pp. 4095-4104.
[42] A. Krizhevsky, V. Nair, and G. Hinton, " CIFAR-10 (Canadian Institute for Advanced
Research), " Tech. Rep., 2010. [Online]. Available: http://www.cs.toronto.edu/kriz/
cifar.html
[43] J. Blank and K. Deb, " Pymoo: Multi-objective optimization in python, " IEEE Access,
vol. 8, pp. 89,497-89,509, 2020.
[44] M. Zhu and S. Gupta, " To prune, or not to prune: Exploring the efficacy of pruning
for model compression, " 2017. [Online]. Available: https://arxiv.org/abs/1710.01878
[45] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, " A survey of model compression and
acceleration for deep neural networks, " 2017. [Online]. Available: https://arxiv.org/
abs/1710.09282
[46] A. G. Howard et al., " Mobilenets: Efficient convolutional neural networks for mobile
vision applications, " 2017. [Online]. Available: https://arxiv.org/abs/1704.04861
[47] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner et al., " Gradient-based learning applied
to document recognition, " Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, 1998. doi: 10.1109/
5.726791.
[48] Xilinx, " Vivado design suite user guide, " Technical Publication, 2018.
[49] E. G. Walters, " Array multipliers for high throughput in Xilinx FPGAs with 6-input
LUTs, " Computers, vol. 5, no. 4, p. 20, 2016. doi: 10.3390/computers5040020.
[50] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, " Play and Prune: Adaptive
filter pruning for deep model compression, " 2019. [Online]. Available: https://arxiv.org/
abs/1905.04446
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 21
https://www.arxiv.org/abs/1608.08710 https://www.arxiv.org/abs/1804.09081 http://www.cs.toronto.edu/kriz/cifar.html http://www.cs.toronto.edu/kriz/cifar.html https://www.arxiv.org/abs/1510.00149 https://www.arxiv.org/abs/1710.01878 https://arxiv.org/abs/1710.09282 https://arxiv.org/abs/1710.09282 https://www.arxiv.org/abs/1704.04861 https://www.arxiv.org/abs/2006.04588 https://arxiv.org/abs/1905.04446 https://arxiv.org/abs/1905.04446
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
