IEEE Computational Intelligence Magazine - May 2021 - 23
MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
23
u
0
2
x
(a)
3
4
v=5
5
10-5
10-7
10-5
10-7
10-9
200 k
0
10-3
10-3
50 k
100 k
150 k
Number of Evaluation
10-1
10-1
10-9
101
101
v=2
(c)
0
tNES
10-9
200 k
0
SGD (ADAM)
50 k
100 k
150 k
Number of Evaluation
10-7
10-5
10-3
10-1
101
0
1
0
0
0.2
0.2
0.6
0.8
1
0.4
v=2
v=5
v=8
v = 10
0.4
0.6
0.8
1
u
2
v=8
x
3
4
xNES
50 k
100 k
150 k
Number of Evaluation
1
Ground Truth
NN Solution
DE Residual
SGD (ADAM)
101
10-7
10-5
10-3
10-1
(b)
0
10-9
200 k
0
5
0.2
0.4
0.6
0.8
1
0
x
3
v = 10
2
4
50 k
100 k
150 k
Number of Evaluation
1
Ground Truth
NN Solution
DE Residual
NES
200 k
5
FIGURE 5 (a) Ground truth (analytical) solution for 1D steady state convection-diffusion equation, with v = 2, 5, 8, 10 in (12). The underlying optimization problem is getting more challenging
when the velocity v increases. For example, at v = 8, the SGD solution shown in (b) is not meaningful because it could not fulfill the boundary conditions, thus it induces high residual at the two
boundaries of x. The worst of xNES is seen to perform notably better than SGD. In (c), the convergence trends from 30 independent SGD, xNES and tNES optimization runs are plotted. The bold
lines indicate the mean convergence path, and the shaded areas indicate the interpercentile range from their 10th-90th percentiles. In all problems, the difference in performance (at 200k evaluations) between three optimization algorithms is concluded by the Friedman tests at 5% significance level. Both xNES and tNES perform better than SGD at 200k evaluations; furthermore, tNES
significantly outperforms xNES when considered under a limited computational budget of 50k evaluations (concluded by Mann-Whitney rank test, at 5% significance level).
Loss
u
IEEE Computational Intelligence Magazine - May 2021
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
Contents
IEEE Computational Intelligence Magazine - May 2021 - Cover1
IEEE Computational Intelligence Magazine - May 2021 - Cover2
IEEE Computational Intelligence Magazine - May 2021 - Contents
IEEE Computational Intelligence Magazine - May 2021 - 2
IEEE Computational Intelligence Magazine - May 2021 - 3
IEEE Computational Intelligence Magazine - May 2021 - 4
IEEE Computational Intelligence Magazine - May 2021 - 5
IEEE Computational Intelligence Magazine - May 2021 - 6
IEEE Computational Intelligence Magazine - May 2021 - 7
IEEE Computational Intelligence Magazine - May 2021 - 8
IEEE Computational Intelligence Magazine - May 2021 - 9
IEEE Computational Intelligence Magazine - May 2021 - 10
IEEE Computational Intelligence Magazine - May 2021 - 11
IEEE Computational Intelligence Magazine - May 2021 - 12
IEEE Computational Intelligence Magazine - May 2021 - 13
IEEE Computational Intelligence Magazine - May 2021 - 14
IEEE Computational Intelligence Magazine - May 2021 - 15
IEEE Computational Intelligence Magazine - May 2021 - 16
IEEE Computational Intelligence Magazine - May 2021 - 17
IEEE Computational Intelligence Magazine - May 2021 - 18
IEEE Computational Intelligence Magazine - May 2021 - 19
IEEE Computational Intelligence Magazine - May 2021 - 20
IEEE Computational Intelligence Magazine - May 2021 - 21
IEEE Computational Intelligence Magazine - May 2021 - 22
IEEE Computational Intelligence Magazine - May 2021 - 23
IEEE Computational Intelligence Magazine - May 2021 - 24
IEEE Computational Intelligence Magazine - May 2021 - 25
IEEE Computational Intelligence Magazine - May 2021 - 26
IEEE Computational Intelligence Magazine - May 2021 - 27
IEEE Computational Intelligence Magazine - May 2021 - 28
IEEE Computational Intelligence Magazine - May 2021 - 29
IEEE Computational Intelligence Magazine - May 2021 - 30
IEEE Computational Intelligence Magazine - May 2021 - 31
IEEE Computational Intelligence Magazine - May 2021 - 32
IEEE Computational Intelligence Magazine - May 2021 - 33
IEEE Computational Intelligence Magazine - May 2021 - 34
IEEE Computational Intelligence Magazine - May 2021 - 35
IEEE Computational Intelligence Magazine - May 2021 - 36
IEEE Computational Intelligence Magazine - May 2021 - 37
IEEE Computational Intelligence Magazine - May 2021 - 38
IEEE Computational Intelligence Magazine - May 2021 - 39
IEEE Computational Intelligence Magazine - May 2021 - 40
IEEE Computational Intelligence Magazine - May 2021 - 41
IEEE Computational Intelligence Magazine - May 2021 - 42
IEEE Computational Intelligence Magazine - May 2021 - 43
IEEE Computational Intelligence Magazine - May 2021 - 44
IEEE Computational Intelligence Magazine - May 2021 - 45
IEEE Computational Intelligence Magazine - May 2021 - 46
IEEE Computational Intelligence Magazine - May 2021 - 47
IEEE Computational Intelligence Magazine - May 2021 - 48
IEEE Computational Intelligence Magazine - May 2021 - 49
IEEE Computational Intelligence Magazine - May 2021 - 50
IEEE Computational Intelligence Magazine - May 2021 - 51
IEEE Computational Intelligence Magazine - May 2021 - 52
IEEE Computational Intelligence Magazine - May 2021 - 53
IEEE Computational Intelligence Magazine - May 2021 - 54
IEEE Computational Intelligence Magazine - May 2021 - 55
IEEE Computational Intelligence Magazine - May 2021 - 56
IEEE Computational Intelligence Magazine - May 2021 - 57
IEEE Computational Intelligence Magazine - May 2021 - 58
IEEE Computational Intelligence Magazine - May 2021 - 59
IEEE Computational Intelligence Magazine - May 2021 - 60
IEEE Computational Intelligence Magazine - May 2021 - 61
IEEE Computational Intelligence Magazine - May 2021 - 62
IEEE Computational Intelligence Magazine - May 2021 - 63
IEEE Computational Intelligence Magazine - May 2021 - 64
IEEE Computational Intelligence Magazine - May 2021 - 65
IEEE Computational Intelligence Magazine - May 2021 - 66
IEEE Computational Intelligence Magazine - May 2021 - 67
IEEE Computational Intelligence Magazine - May 2021 - 68
IEEE Computational Intelligence Magazine - May 2021 - 69
IEEE Computational Intelligence Magazine - May 2021 - 70
IEEE Computational Intelligence Magazine - May 2021 - 71
IEEE Computational Intelligence Magazine - May 2021 - 72
IEEE Computational Intelligence Magazine - May 2021 - 73
IEEE Computational Intelligence Magazine - May 2021 - 74
IEEE Computational Intelligence Magazine - May 2021 - 75
IEEE Computational Intelligence Magazine - May 2021 - 76
IEEE Computational Intelligence Magazine - May 2021 - 77
IEEE Computational Intelligence Magazine - May 2021 - 78
IEEE Computational Intelligence Magazine - May 2021 - 79
IEEE Computational Intelligence Magazine - May 2021 - 80
IEEE Computational Intelligence Magazine - May 2021 - 81
IEEE Computational Intelligence Magazine - May 2021 - 82
IEEE Computational Intelligence Magazine - May 2021 - 83
IEEE Computational Intelligence Magazine - May 2021 - 84
IEEE Computational Intelligence Magazine - May 2021 - 85
IEEE Computational Intelligence Magazine - May 2021 - 86
IEEE Computational Intelligence Magazine - May 2021 - 87
IEEE Computational Intelligence Magazine - May 2021 - 88
IEEE Computational Intelligence Magazine - May 2021 - 89
IEEE Computational Intelligence Magazine - May 2021 - 90
IEEE Computational Intelligence Magazine - May 2021 - 91
IEEE Computational Intelligence Magazine - May 2021 - 92
IEEE Computational Intelligence Magazine - May 2021 - 93
IEEE Computational Intelligence Magazine - May 2021 - 94
IEEE Computational Intelligence Magazine - May 2021 - 95
IEEE Computational Intelligence Magazine - May 2021 - 96
IEEE Computational Intelligence Magazine - May 2021 - 97
IEEE Computational Intelligence Magazine - May 2021 - 98
IEEE Computational Intelligence Magazine - May 2021 - 99
IEEE Computational Intelligence Magazine - May 2021 - 100
IEEE Computational Intelligence Magazine - May 2021 - Cover3
IEEE Computational Intelligence Magazine - May 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com