IEEE Computational Intelligence Magazine - August 2021 - 49

applicable to any search strategy that employs graph neural
networks as neural predictors.
For future works, combining the pre-trained neural predictors
to other neural predictor-based NAS to verify their generalization
ability is worth further investigation. Extending the integration of
pre-trained neural predictors with NPENAS to other tasks, such
as image segmentation, object detection and natural language
processing, is also promising research.
VI. Acknowledgment
This work was supported in part by the National Natural Science
Foundation of China under Grants Nos 61976167, U19B2030
and 11727813, the Key Research and Development Program in
the Shaanxi Province of China under Grant 2021GY-082, and
the Xi'an Science and Technology Program under Grant
201809170CX11JC12. The authors would like to thank Dr.
Karen von Deneen for her professional language editing.
References
[1] T. Elsken, J. H. Metzen, and F. Hutter, " Neural architecture search: A survey, " J. Mach. Learning Res.,
vol. 20, pp. 55:1-55:21, 2018.
[2] C. Liu et al., " Progressive neural architecture search, " in Proc. Computer Vision-ECCV 2018-
15th European Conf., Munich, Part I, Sept. 8-14, 2018, vol. 11205, pp. 19-35.
[3] C. White, W. Neiswanger, and Y. Savani, " Bananas: Bayesian optimization with neural
architectures for neural architecture search, " 2019. [Online]. Available: https://arxiv
.org/abs/1910.11858
[4] L. Wang, Y. Zhao, Y. Jinnai, Y. Tian, and R. Fonseca, " Neural architecture search using deep neural
networks and Monte Carlo tree search, " in Proc. 34th AAAI Conf. Artif. Intell. (AAAI 2020) 32nd Innovative
Appl. Artif. Intell. Conf. (IAAI 2020) 10th AAAI Symp. Educational Adv. Artif. Intell., EAAI 2020,
New York, Feb. 7-12, 2020, pp. 9983-9991.
[5] X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang, " A generic graph-based neural architecture
encoding scheme for predictor-based NAS, " in Proc. Comput. Vision-ECCV 2020-16th European
Conf., Glasgow, U.K., Aug. 23-28, 2020, pp. 189-204.
[6] C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, and J. Min Liang, " NPENAS: Neural predictor guided
evolution for neural architecture search, " 2020. [Online]. Available: https://arxiv.org/abs/2003.12857
[7] W. Wen, H. Liu, Y. Chen, H. H. Li, G. Bender, and P. Kindermans, " Neural predictor for neural
architecture search, " in Proc. Comput. Vision-ECCV 2020-16th European Conf., Glasgow, U.K.,
Aug. 23-28, 2020, pp. 660-676.
[8] S. Yan, Y. Zheng, W. Ao, X. Zeng, and M. Zhang, " Does unsupervised architecture representation
learning help neural architecture search? " in Proc. Adv. Neural Inf. Process. Syst. 33: Annu. Conf. Neural
Inf. Process. Syst. 2020, NeurIPS 2020, Dec. 6-12, 2020.
[9] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, " A simple framework for contrastive
learning
of
visual
representations, " 2020. [Online]. Available: https://arxiv
.org/abs/2002.05709
[10] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, " Momentum contrast for unsupervised visual
representation learning, " in Proc. IEEE/CVF Conf. Comput. Vision and Pattern Recogn., CVPR 2020,
Seattle, WA, June 13-19, 2020, pp. 9726-9735.
[11] J. Devlin, M. Chang, K. Lee, and K. Toutanova, " BERT: pre-training of deep bidirectional transformers
for language understanding, " in Proc. Conf. North American Chapter Assoc. Comput. Linguistics:
Human Language Technol. (NAACL-HLT 2019), Minneapolis, MN, June 2-7, 2019, pp. 4171-4186.
[12] Y. M. Asano, C. Rupprecht, and A. Vedaldi, " Self-labelling via simultaneous clustering and representation
learning, " in Proc. 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,
Apr. 26-30, 2020.
[13] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, " A comprehensive survey on graph neural
networks, " IEEE Trans. Neural Netw. Learning Syst., 2020.
[14] H. Cheng et al., " NASGEM: Neural architecture search via graph embedding method, " 2020.
[Online]. Available: https://arxiv.org/abs/2007.04452
[15] M. Noroozi and P. Favaro, " Unsupervised learning of visual representations by solving jigsaw
puzzles, " in Proc. Comput. Vision-ECCV 2016-14th European Conf., Amsterdam, The Netherlands,
Oct. 11-14, 2016, pp. 69-84.
[16] S. Gidaris, P. Singh, and N. Komodakis, " Unsupervised representation learning by predicting
image rotations, " in Proc. 6th Int. Conf. Learning Representations, ICLR 2018, Vancouver, BC, Canada,
April 30-May 3, 2018.
[17] C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, " NAS-bench-101: Towards
reproducible neural architecture search, " in Proc. 36th Int. Conf. Machine Learning, ICML 2019, June
9-15, 2019, vol. 97, pp. 7105-7114.
[18] J. You, J. Leskovec, K. He, and S. Xie, " Graph structure of neural networks, " in Proc. 37th Int. Conf.
Machine Learning, ICML 2020, July 12-18, 2020.
[19] X. Dong and Y. Yang, " NAS-Bench-201: Extending the scope of reproducible neural architecture
search, " in Proc. 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apr.
26-30, 2020.
[20] H. Liu, K. Simonyan, and Y. Yang, " DARTS: Differentiable architecture search, " in Proc. 7th Int.
Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6-9, 2019.
[21] B. Zoph and Q. V. Le, " Neural architecture search with reinforcement learning, " in Proc. 5th Int.
Conf. Learning Representations, ICLR 2017, Toulon, France, Apr. 24-26, 2017.
[22] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, " Efficient neural architecture search via
parameter sharing, " in Proc. 35th Int. Conf. Machine Learning, ICML 2018, Stockholmsmässan, Stockholm,
Sweden, July 10-15, 2018, vol. 80, pp. 4092-4101.
[23] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, " Learning transferable architectures for scalable image
recognition, " in Proc. IEEE Conf. Comput. Vision and Pattern Recogn., CVPR 2018, Salt Lake City,
UT, June 18-22, 2018, pp. 8697-8710.
[24] E. Real et al., " Large-scale evolution of image classifiers, " in Proc. 34th Int. Conf. Machine Learning,
ICML 2017, Sydney, NSW, Australia, Aug. 6-11, 2017, vol. 70, pp. 2902-2911.
[25] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, " Regularized evolution for image
classifier architecture search, " in Proc. 33rd AAAI Conf. Artif.
Intell., AAAI 2019, 31st
Innovative Appl. Artif. Intell. Conf., IAAI 2019, 9th AAAI Symp. Educational Adv. Artif. Intell., EAAI 2019,
Honolulu, HI, Jan. 27-Feb. 1, 2019, pp. 4780-4789.
[26] Y. Sun, B. Xue, M. Zhang, G. G. Yen, and J. Lv, " Automatically designing CNN architectures using
the genetic algorithm for image classification, " IEEE Trans. Cybernetics, 2020, pp. 1-15. doi: 10.1109/
TCYB.2020.2983860.
[27] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Completely automated CNN architecture
design based on blocks, " IEEE Trans. Neural Netw. Learning Syst., vol. 31, no. 4,
pp. 1242-1254, 2020. doi: 10.1109/TNNLS.2019.2919608.
[28] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, " Evolving deep convolutional neural networks for image
classification, " IEEE Trans. Evolutionary Comput., vol. 24, no. 2, pp. 394-407, 2020.
[29] H. Zhou, M. Yang, J. Wang, and W. Pan, " Bayesnas: A Bayesian approach for neural architecture
search, " in Proc. 36th Int. Conf. Machine Learning, ICML 2019, June 9-15, 2019, Long Beach, CA,
2019, vol. 97, pp. 7603-7613.
[30] S. Xie, H. Zheng, C. Liu, and L. Lin, " SNAS: Stochastic neural architecture search, " in Proc. 7th Int.
Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6-9, 2019.
[31] X. Chen, L. Xie, J. Wu, and Q. Tian, " Progressive differentiable architecture search: Bridging the
depth gap between search and evaluation, " pp. 1294-1303, 2019.
[32] Y. Xu et al., " PC-DARTS: Partial channel connections for memory-efficient architecture search, "
in 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apr. 26-30, 2020.
[33] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, " Neural architecture search
with Bayesian optimisation and optimal transport, " in Proc. Adv. Neural Information Process. Syst. 31:
Annu. Conf. Neural Information Process. Syst. 2018, NeurIPS 2018, Montréal, Canada, Dec. 3-8, 2018,
pp. 2016-2025.
[34] R. Luo, X. Tan, R. Wang, T. Qin, E. Chen, and T. Liu, " Semi-supervised neural architecture
search, " in Proc. Adv. Neural Information Processing Syst. 33: Annu. Conf. Neural Information Process. Syst.
2020, NeurIPS 2020, Dec. 6-12, 2020.
[35] L. Dudziak, T. C. P. Chau, M. S. Abdelfattah, R. Lee, H. Kim, and N. D. Lane, " BRP-NAS:
Prediction-based NAS using GCNS, " in Proc. Adv. Neural Inf. Process. Syst. 33: Annu. Conf. Neural Inf.
Process. Syst. 2020, NeurIPS 2020, Dec. 6-12, 2020.
[36] B. Baker, O. Gupta, R. Raskar, and N. Naik, " Accelerating neural architecture search using
performance prediction, " in Proc. 6th Int. Conf. Learning Representations, ICLR 2018, Vancouver, BC,
Canada, Apr. 30-May 3, 2018.
[37] D. P. Kingma and M. Welling, " Auto-encoding variational bayes, " in Proc. 2nd Int. Conf. Learning
Representations, ICLR 2014, Banff, AB, Canada, Apr. 14-16, 2014.
[38] Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli, " Graph matching networks for learning the similarity
of graph structured objects, " in Proc. 36th Int. Conf. Machine Learning, ICML 2019, Long Beach,
CA, June 9-15, 2019, pp. 3835-3845.
[39] Y. Bai, H. Ding, S. Bian, T. Chen, Y. Sun, and W. Wang, " SIMGNN: A neural network approach to
fast graph similarity computation, " in Proc. 12th ACM Int. Conf. Web Search and Data Mining, WSDM
2019, Melbourne, VIC, Australia, Feb. 11-15, 2019, pp. 384-392.
[40] Y. Bai et al., " Unsupervised inductive graph-level representation learning via graph-graph proximity, "
in Proc. 28th Int. Joint Conf. Artif. Intell., IJCAI 2019, Macao, China, Aug. 10-16, 2019, pp.
1988-1994.
[41] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, " How powerful are graph neural networks? " in Proc. 7th
Int. Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6-9, 2019.
[42] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, " Unsupervised learning
of visual features by contrasting cluster assignments, " 2020. [Online]. Available: https://arxiv.org/
abs/2006.09882
[43] A. Paszke et al., " Pytorch: An imperative style, high-performance deep learning library, " in Proc.
Adv. Neural Information Processing Syst. 32: Annu. Conf. Neural Information Process. Syst. 2019, NeurIPS
2019, Dec. 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 8024-8035.
[44] M. Fey and J. E. Lenssen, " Fast graph representation learning with PyTorch Geometric, " in Proc.
Int. Conf. Learning Representations Workshop Representation Learning Graphs and Manifolds, 2019.
[45] C. Wei, " Code for self-supervised representation learning for evolutionary neural architecture
search, " 2020. [Online]. Available: https://github.com/auroua/SSNENAS
[46] A. Krizhevsky and G. E. Hinton, " Learning multiple layers of features from tiny images, " Tech.
Rep., 2009.
[47] A. Krizhevsky, I. Sutskever, and G. E. Hinton, " Imagenet classification with deep convolutional
neural networks, " in Proc. Adv. Neural Information Process. Syst. 25: 26th Annu. Conf. Neural Information
Process. Syst. 2012. Proc. Meeting, Lake Tahoe, NV, Dec. 3-6, 2012, pp. 1106-1114.
[48] D. P. Kingma and J. Ba, " Adam: A method for stochastic optimization, " CoRR, vol. abs/1412.6980,
2014.
[49] I. Loshchilov and F. Hutter, " SGDR: Stochastic gradient descent with warm restarts, " in Proc. 5th
Int. Conf. Learning Representations, ICLR 2017, Toulon, France, Apr. 24-26, 2017, 2017.
[50] L. Li and A. Talwalkar, " Random search and reproducibility for neural architecture search, " in
Proc. 35th Conf. Uncertainty Artif. Intell., UAI 2019, Tel Aviv, Israel, July 22-25, 2019, p. 129.
[51] R. Luo, F. Tian, T. Qin, and T.-Y. Liu, " Neural architecture optimization, " in Proc. Adv. Neural Inf.
Process. Syst. 31: Annu. Conf. Neural Inf. Process. Syst. 2018, NeurIPS 2018, Dec. 3-8, 2018, Montréal,
Canada, 2018, pp. 7827-7838.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 49
https://arxiv.org/abs/1910.11858 https://arxiv.org/abs/1910.11858 https://www.arxiv.org/abs/2003.12857 https://arxiv.org/abs/2002.05709 https://arxiv.org/abs/2002.05709 https://arxiv.org/abs/2006.09882 https://arxiv.org/abs/2006.09882 https://www.arxiv.org/abs/2007.04452 https://www.github.com/auroua/SSNENAS
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
