IEEE Computational Intelligence Magazine - August 2021 - 21
which trade-off the two objectives. The constraint on the
energy can be calculated based on the energy capacity and
battery life. Then, the solution that achieves the highest accuracy
will be selected as the optimal solution for the task.
Alternatively, one can select the knee point from the solution
set as the preferred solution.
VI. Conclusion
In this paper, an evolutionary multi-objective model compression
approach is proposed to accelerate and compress DNNs by
optimizing multiple objectives (e.g., accuracy, energy efficiency,
and model size) simultaneously. As the evaluation of each architecture
is extremely time-consuming during the evolution, a
two-stage pruning and optimization co-optimization strategy is
developed to speed up the architecture searching process.
Extensive experimental results demonstrate that the proposed
method can obtain a set of diverse networks in a single execution.
Furthermore, the proposed method outperforms the peer
methods in terms of energy efficiency and model size for model
compression of three popular DNNs.
Acknowledgement
This work is partly supported by the Agency for Science, Technology
and Research (A*STAR) under its AME Programmatic
Funding Scheme (No. A18A1b0045 and No. A1687b0033).
References
[1] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep Learning, vol. 1. Cambridge
MA: MIT Press, 2016, no. 2.
[2] K. Simonyan and A. Zisserman, " Very deep convolutional networks for large-scale
image recognition, " 2014. [Online]. Available: https://arxiv.org/abs/1804.09081
[3] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, " XLNet:
Generalized autoregressive pretraining for language understanding, " in Adv. Neural Inf.
Process. Syst., 2019, pp. 5753-5763.
[4] L. Zhen, P. Hu, X. Peng, R. S. M. Goh, and J. T. Zhou, " Deep multimodal transfer
learning for cross-modal retrieval, " IEEE Trans. Neural Netw. Learn. Syst., to be published.
doi: 10.1109/TNNLS.2020.3029181.
[5] H. Liu, K. Simonyan, and Y. Yang, " DARTS: Differentiable architecture search, " in
Proc. Int. Conf. Learn. Representations, 2018.
[6] Y. Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, and M. Zhang, " Surrogate-assisted evolutionary
deep learning using an end-to-end random forest-based performance predictor, " IEEE
Trans. Evol. Comput., vol. 24, no. 2, pp. 350-364, 2020. doi: 10.1109/TEVC.2019.2924461.
[7] Y. Sun, G. G. Yen, and Z. Yi, " Evolving unsupervized deep neural networks for learning
meaningful representations, " IEEE Trans. Evol. Comput., vol. 23, no. 1, pp. 89-103,
2018. doi: 10.1109/TEVC.2018.2808689.
[8] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Completely automated CNN architecture
design based on blocks, " IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 4, pp.
1242-1254, 2020. doi: 10.1109/TNNLS.2019.2919608.
[9] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Evolving deep convolutional neural networks
for image classification, " IEEE Trans. Evol. Comput., vol. 24, no. 2, pp. 394-407, 2020.
[10] S. Han, H. Mao, and W. J. Dally, " Deep compression: Compressing deep neural networks
with pruning, trained quantization and Huffman coding, " 2015. [Online]. Available:
https://arxiv.org/abs/1510.00149
[11] Y. Wang, C. Xu, J. Qiu, C. Xu, and D. Tao, " Towards evolutionary compression, "
in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2018, pp. 2476-2485.
[12] T.-J. Yang, Y.-H. Chen, and V. Sze, " Designing energy-efficient convolutional neural
networks using energy-aware pruning, " in Proc. IEEE Conf. Comput. Vision Pattern
Recognit., 2017, pp. 5687-5695.
[13] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, " HAQ: Hardware-aware automated quantization
with mixed precision, " in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2019.
[14] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, " Efficient processing of deep neural
networks: A tutorial and survey, " Proc. IEEE, vol. 105, no. 12, pp. 2295-2329, 2017. doi:
10.1109/JPROC.2017.2761740.
[15] Z. Wang, T. Luo, J. T. Zhou, and R. S. M. Goh, " EDCompress: Energy-aware model
compression with dataflow, " 2020. [Online]. Available: https://arxiv.org/abs/2006.04588
[16] Z. Du et al., " ShiDianNao: Shifting vision processing closer to the sensor, " in Proc.
Annu. Int. Symp. Computer Architecture, 2015, pp. 92-104.
[17] M. Song et al., " Towards efficient microarchitectural design for accelerating unsupervized
GAN-based deep learning, " in Proc. IEEE Int. Symp. High Performance Comput.
Architecture, 2018, pp. 66-77.
[18] N. P. Jouppi et al., " In-datacenter performance analysis of a tensor processing unit, "
in Proc. Annu. Int. Symp. Comput. Architecture, 2017, pp. 1-12.
[19] M. Alwani, H. Chen, M. Ferdman, and P. Milder, " Fused-layer CNN accelerators, "
in Proc. Annu. IEEE/ACM Int. Symp. Microarchitecture, 2016, pp. 1-12.
[20] J. Qiu et al., " Going deeper with embedded FPGA platform for convolutional neural
network, " in Proc. ACM/SIGDA Int. Symp. Field-Programmable Gate Arrays, 2016,
pp. 26-35.
[21] Y.-H. Chen et al., " Eyeriss: A spatial architecture for energy-efficient dataflow for
convolutional neural networks, " ACM SIGARCH Comput. Architecture News, vol. 44, no.
3, pp. 367-379, 2016. doi: 10.1145/3007787.3001177.
[22] H. Li, X. Fan, L. Jiao, W. Cao, X. Zhou, and L. Wang, " A high performance FPGAbased
accelerator for large-scale convolutional neural networks, " in Proc. Int. Conf. Field
Programmable Logic Appl., 2016, pp. 1-9.
[23] Y. Guo, A. Yao, and Y. Chen, " Dynamic network surgery for efficient DNNs, " in
Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 1379-1387.
[24] F. Manessi, A. Rozza, S. Bianco, P. Napoletano, and R. Schettini, " Automated
pruning for deep neural network compression, " in Proc. Int. Conf. Pattern Recognit., 2018,
pp. 657-664.
[25] C. Lemaire, A. Achkar, and P.-M. Jodoin, " Structured pruning of neural networks
with budget-aware regularization, " in Proc. IEEE Conf. Comput. Vision Pattern Recognit.,
2019, pp. 9108-9116.
[26] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, " Pruning filters for efficient
convnets, " 2016. [Online]. Available: https://arxiv.org/abs/1608.08710
[27] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, " AMC: AutoML for model
compression and acceleration on mobile devices, " in Proc. Eur. Conf. Comput. Vision, 2018,
pp. 784-800.
[28] Z. Liu, J. Xu, X. Peng, and R. Xiong, " Frequency-domain dynamic pruning for convolutional
neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 1043-1053.
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton, " ImageNet classification with deep
convolutional neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
[30] C. Szegedy et al., " Going deeper with convolutions, " in Proc. IEEE Conf. Comput.
Vision Pattern Recognit., 2015, pp. 1-9.
[31] K. Deb, " Multi-objective optimization, " in Search Methodologies. Springer-Verlag,
2014, pp. 403-449.
[32] J. D. Schaffer, " Multiple objective optimization with vector evaluated genetic algorithms, "
in Proc. Int. Conf. Genetic Algorithms, 1985, pp. 93-100.
[33] B. Li, J. Li, K. Tang, and X. Yao, " Many-objective evolutionary algorithms: A survey, "
ACM Comput. Surv., vol. 48, no. 1, 2015. doi: 10.1145/2792984.
[34] L. Zhen, M. Li, R. Cheng, D. Peng, and X. Yao, " Adjusting parallel coordinates for
investigating multi-objective search, " in Proc. Int. Conf. Simulated Evol. Learn., Shenzhen,
China, 2017, pp. 224-235.
[35] M. Li, L. Zhen, and X. Yao, " How to read many-objective solution sets in parallel
coordinates, " IEEE Comput. Intell. Mag., vol. 12, no. 4, pp. 88-100, 2017. doi: 10.1109/
MCI.2017.2742869.
[36] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley
Longman Publishing, 1989.
[37] N. Srinivas and K. Deb, " Muiltiobjective optimization using nondominated sorting
in genetic algorithms, " Evol. Comput., vol. 2, no. 3, pp. 221-248, 1994. doi: 10.1162/
evco.1994.2.3.221.
[38] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, " A fast and elitist multiobjective
genetic algorithm: NSGA-II, " IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182-197,
2002. doi: 10.1109/4235.996017.
[39] C. A. C. Coello, S. G. Brambila, J. F. Gamboa, M. G. C. Tapia, and R. H. Gómez,
" Evolutionary multiobjective optimization: Open research areas and some challenges lying
ahead, " Complex Intell. Syst., vol. 6, no. 2, pp. 221-236, 2020.
[40] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf,
" NSGA-Net: Neural architecture search using multi-objective genetic algorithm, " in
Proc. Genetic Evol. Comput. Conf., 2019, pp. 419-427.
[41] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, " Efficient neural architecture
search via parameters sharing, " in Proc. Int. Conf. Mach. Learn., 2018, pp. 4095-4104.
[42] A. Krizhevsky, V. Nair, and G. Hinton, " CIFAR-10 (Canadian Institute for Advanced
Research), " Tech. Rep., 2010. [Online]. Available: http://www.cs.toronto.edu/kriz/
cifar.html
[43] J. Blank and K. Deb, " Pymoo: Multi-objective optimization in python, " IEEE Access,
vol. 8, pp. 89,497-89,509, 2020.
[44] M. Zhu and S. Gupta, " To prune, or not to prune: Exploring the efficacy of pruning
for model compression, " 2017. [Online]. Available: https://arxiv.org/abs/1710.01878
[45] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, " A survey of model compression and
acceleration for deep neural networks, " 2017. [Online]. Available: https://arxiv.org/
abs/1710.09282
[46] A. G. Howard et al., " Mobilenets: Efficient convolutional neural networks for mobile
vision applications, " 2017. [Online]. Available: https://arxiv.org/abs/1704.04861
[47] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner et al., " Gradient-based learning applied
to document recognition, " Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, 1998. doi: 10.1109/
5.726791.
[48] Xilinx, " Vivado design suite user guide, " Technical Publication, 2018.
[49] E. G. Walters, " Array multipliers for high throughput in Xilinx FPGAs with 6-input
LUTs, " Computers, vol. 5, no. 4, p. 20, 2016. doi: 10.3390/computers5040020.
[50] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, " Play and Prune: Adaptive
filter pruning for deep model compression, " 2019. [Online]. Available: https://arxiv.org/
abs/1905.04446
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 21
https://www.arxiv.org/abs/1608.08710
https://www.arxiv.org/abs/1804.09081
http://www.cs.toronto.edu/kriz/cifar.html
http://www.cs.toronto.edu/kriz/cifar.html
https://www.arxiv.org/abs/1510.00149
https://www.arxiv.org/abs/1710.01878
https://arxiv.org/abs/1710.09282
https://arxiv.org/abs/1710.09282
https://www.arxiv.org/abs/1704.04861
https://www.arxiv.org/abs/2006.04588
https://arxiv.org/abs/1905.04446
https://arxiv.org/abs/1905.04446
IEEE Computational Intelligence Magazine - August 2021
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
Contents
IEEE Computational Intelligence Magazine - August 2021 - Cover1
IEEE Computational Intelligence Magazine - August 2021 - Cover2
IEEE Computational Intelligence Magazine - August 2021 - Contents
IEEE Computational Intelligence Magazine - August 2021 - 2
IEEE Computational Intelligence Magazine - August 2021 - 3
IEEE Computational Intelligence Magazine - August 2021 - 4
IEEE Computational Intelligence Magazine - August 2021 - 5
IEEE Computational Intelligence Magazine - August 2021 - 6
IEEE Computational Intelligence Magazine - August 2021 - 7
IEEE Computational Intelligence Magazine - August 2021 - 8
IEEE Computational Intelligence Magazine - August 2021 - 9
IEEE Computational Intelligence Magazine - August 2021 - 10
IEEE Computational Intelligence Magazine - August 2021 - 11
IEEE Computational Intelligence Magazine - August 2021 - 12
IEEE Computational Intelligence Magazine - August 2021 - 13
IEEE Computational Intelligence Magazine - August 2021 - 14
IEEE Computational Intelligence Magazine - August 2021 - 15
IEEE Computational Intelligence Magazine - August 2021 - 16
IEEE Computational Intelligence Magazine - August 2021 - 17
IEEE Computational Intelligence Magazine - August 2021 - 18
IEEE Computational Intelligence Magazine - August 2021 - 19
IEEE Computational Intelligence Magazine - August 2021 - 20
IEEE Computational Intelligence Magazine - August 2021 - 21
IEEE Computational Intelligence Magazine - August 2021 - 22
IEEE Computational Intelligence Magazine - August 2021 - 23
IEEE Computational Intelligence Magazine - August 2021 - 24
IEEE Computational Intelligence Magazine - August 2021 - 25
IEEE Computational Intelligence Magazine - August 2021 - 26
IEEE Computational Intelligence Magazine - August 2021 - 27
IEEE Computational Intelligence Magazine - August 2021 - 28
IEEE Computational Intelligence Magazine - August 2021 - 29
IEEE Computational Intelligence Magazine - August 2021 - 30
IEEE Computational Intelligence Magazine - August 2021 - 31
IEEE Computational Intelligence Magazine - August 2021 - 32
IEEE Computational Intelligence Magazine - August 2021 - 33
IEEE Computational Intelligence Magazine - August 2021 - 34
IEEE Computational Intelligence Magazine - August 2021 - 35
IEEE Computational Intelligence Magazine - August 2021 - 36
IEEE Computational Intelligence Magazine - August 2021 - 37
IEEE Computational Intelligence Magazine - August 2021 - 38
IEEE Computational Intelligence Magazine - August 2021 - 39
IEEE Computational Intelligence Magazine - August 2021 - 40
IEEE Computational Intelligence Magazine - August 2021 - 41
IEEE Computational Intelligence Magazine - August 2021 - 42
IEEE Computational Intelligence Magazine - August 2021 - 43
IEEE Computational Intelligence Magazine - August 2021 - 44
IEEE Computational Intelligence Magazine - August 2021 - 45
IEEE Computational Intelligence Magazine - August 2021 - 46
IEEE Computational Intelligence Magazine - August 2021 - 47
IEEE Computational Intelligence Magazine - August 2021 - 48
IEEE Computational Intelligence Magazine - August 2021 - 49
IEEE Computational Intelligence Magazine - August 2021 - 50
IEEE Computational Intelligence Magazine - August 2021 - 51
IEEE Computational Intelligence Magazine - August 2021 - 52
IEEE Computational Intelligence Magazine - August 2021 - 53
IEEE Computational Intelligence Magazine - August 2021 - 54
IEEE Computational Intelligence Magazine - August 2021 - 55
IEEE Computational Intelligence Magazine - August 2021 - 56
IEEE Computational Intelligence Magazine - August 2021 - 57
IEEE Computational Intelligence Magazine - August 2021 - 58
IEEE Computational Intelligence Magazine - August 2021 - 59
IEEE Computational Intelligence Magazine - August 2021 - 60
IEEE Computational Intelligence Magazine - August 2021 - 61
IEEE Computational Intelligence Magazine - August 2021 - 62
IEEE Computational Intelligence Magazine - August 2021 - 63
IEEE Computational Intelligence Magazine - August 2021 - 64
IEEE Computational Intelligence Magazine - August 2021 - 65
IEEE Computational Intelligence Magazine - August 2021 - 66
IEEE Computational Intelligence Magazine - August 2021 - 67
IEEE Computational Intelligence Magazine - August 2021 - 68
IEEE Computational Intelligence Magazine - August 2021 - 69
IEEE Computational Intelligence Magazine - August 2021 - 70
IEEE Computational Intelligence Magazine - August 2021 - 71
IEEE Computational Intelligence Magazine - August 2021 - 72
IEEE Computational Intelligence Magazine - August 2021 - 73
IEEE Computational Intelligence Magazine - August 2021 - 74
IEEE Computational Intelligence Magazine - August 2021 - 75
IEEE Computational Intelligence Magazine - August 2021 - 76
IEEE Computational Intelligence Magazine - August 2021 - 77
IEEE Computational Intelligence Magazine - August 2021 - 78
IEEE Computational Intelligence Magazine - August 2021 - 79
IEEE Computational Intelligence Magazine - August 2021 - 80
IEEE Computational Intelligence Magazine - August 2021 - 81
IEEE Computational Intelligence Magazine - August 2021 - 82
IEEE Computational Intelligence Magazine - August 2021 - 83
IEEE Computational Intelligence Magazine - August 2021 - 84
IEEE Computational Intelligence Magazine - August 2021 - 85
IEEE Computational Intelligence Magazine - August 2021 - 86
IEEE Computational Intelligence Magazine - August 2021 - 87
IEEE Computational Intelligence Magazine - August 2021 - 88
IEEE Computational Intelligence Magazine - August 2021 - 89
IEEE Computational Intelligence Magazine - August 2021 - 90
IEEE Computational Intelligence Magazine - August 2021 - 91
IEEE Computational Intelligence Magazine - August 2021 - 92
IEEE Computational Intelligence Magazine - August 2021 - 93
IEEE Computational Intelligence Magazine - August 2021 - 94
IEEE Computational Intelligence Magazine - August 2021 - 95
IEEE Computational Intelligence Magazine - August 2021 - 96
IEEE Computational Intelligence Magazine - August 2021 - Cover3
IEEE Computational Intelligence Magazine - August 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com