0.6 0.5 0.4 0.3 0.2 0.1 0.0 Training Epoch: 200 7.2 7.4 7 6.6 6.8 MLP NP_NAS SS-CCL SS-RL SUPERVISED SemiNAS 20 50 100 Search Budget FIGURE 7 Performance comparison of different neural predictors. FIGURE 8 Performance comparison of NAS algorithms on NASBench-101. TABLE III Performance comparison of NAS algorithms on NASBench-101. METHODS RS [50] REA [25] BANANAS-PE [3] BANANAS-AE [3] BANANAS-PAPE [3] NPENAS-NP [6] NPENAS-NP-FIXED [6] ARCH2VEC-RL [8] ARCH2VEC-BO [8] NPENAS-SSRL NPENAS-SSRL-FIXED NPENAS-SSCCL NPENAS-SSCCL-FIXED SEARCH BUDGET 150 150 150 150 150 150 90† 400 400 150 90† 150 90† TEST ERR (%) AVG 6.42 ± 0.20 6.32 ± 0.23 5.90 ± 0.15 5.85 ± 0.14 5.86 ± 0.14 5.83 ± 0.11 5.90 ± 0.16 5.90 5.95 5.86 ± 0.14 5.88 ± 0.15 5.83 ± 0.11 5.85 ± 0.13 200 6.4 6.2 6 5.8 10 30 70 90 110 130 150 Number of Samples 50 RS REA BANANAS-PE BANANAS-PAPE NPENAS-NP NPENAS-SSRL NPENAS-SSCCL ARCHITECTURE EMBEDDING - DISCRETE SUPERVISED SUPERVISED SUPERVISED SUPERVISED SUPERVISED UNSUPERVISED UNSUPERVISED SELF- SUPERVISED SELF- SUPERVISED SELF- SUPERVISED SELF- SUPERVISED †The neural predictor is trained with 90 evaluated neural architectures, while other algorithms use 150 neural architectures for evaluation. SEARCH METHOD RANDOM SEARCH EVOLUTION BAYESIAN OPTIMIZATION BAYESIAN OPTIMIZATION BAYESIAN OPTIMIZATION EVOLUTION EVOLUTION REINFORCE BAYESIAN OPTIMIZATION EVOLUTION EVOLUTION EVOLUTION EVOLUTION TABLE IV Impact of fixed search budget on NASBench-101. METHODS NPENAS-SSRL NPENAS-SSRL NPENAS-SSRL NPENAS-SSRL NPENAS-SSRL NPENAS-SSCCL NPENAS-SSCCL NPENAS-SSCCL NPENAS-SSCCL NPENAS-SSCCL SEARCH BUDGET† 20 50 80 110 150 20 50 80 110 150 5.94 ± 0.18 5.87 ± 0.14 5.86 ± 0.11 5.86 ± 0.14 5.99 ± 0.21 5.87 ± 0.15 5.83 ± 0.12 5.83 ± 0.12 5.83 ± 0.12 †The neural predictor is trained with the given number of search budgets. TEST ERR (%) AVG 6.04 ± 0.25 is greater than 70, the performance of the extra sampled neural architectures is predicted by the neural predictor. Upon completion of the search, the best performing neural architecture is selected for evaluation on the CIFAR-10 dataset, and the architecture is evaluated five times with different seeds. All the other evaluation settings are the same as DARTS [32]. These experiments are executed on two Nvidia RTX 2080Ti GPUs. 2) NAS Results on NASBench-101 The performance of different NAS algorithms on the NASBench-101 benchmark is illustrated in Figure 8, and the quantitative comparison is also provided in Table III. Except for RS and REA, the other algorithms achieve comparable performance on NASBench-101 (Figure 8), while NPENAS-SSCCL and NPENAS-NP have the best performance overall, as shown in 46 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021 Kendall Tau Correlation Testing Error of Best Neural Net