IEEE Computational Intelligence Magazine - August 2019 - 15

A. Architecture

The network architecture chosen is similar to the ones
used in Atari 2600 game AI research and early experiments in the StarCraft II Learning Environment (SC2LE)
[31]. Lower dimensional input was used instead of raw
pixels, as coarser grid representations worked well in previous experiments on μRTS and StarCraft combat scenarios. The 64 × 64 input feature planes (described at the
end of the current subsection) were first processed by
three convolutional layers. The extracted map representation was then passed through a fully connected layer then
through a recurrent (LSTM) layer with 128 units both.
The recurrent layer provides 'memory' functionality and
helps in partial observability settings and with concepts

such as unit velocity which cannot be inferred from a single observation/frame. The Q-values were represented
using value and advantage functions as recommended by
the dueling architecture [32]. This architecture is shown
in Figure 5.
Due to the limited project time and resources available,
the Independent Q-Learning (IQL) [33] paradigm was
chosen. In IQL, the current agent is considered to be
learning while all other agents are treated as part of the
environment. Thus, the multi-agent lear ning task is
decomposed into simultaneous single-agent problems. An
alternative to IQL is centralized learning, where the joint
Q-function is learned directly. This can lead to better
agent coordination and results, but at the cost of increased

FIGURE 4 Example battle scenario in TotalWar: Warhammer.

FC 11 Soft
Input: 64 × 64,
18 Planes

CONV 8 × 8,
32 Filters, Stride 4,
Pad 2, LReLu

CONV 4 × 4,
32 Filters, Stride 2,
Pad 1, LReLu

CONV 4 × 4,
64 Filters, Stride 2,
Pad 1, LReLu

FC 128,
Drop 0.2

LSTM 128

FC 1

FIGURE 5 Actor-Critic Network Architecture for TW battle scenarios, assuming maximum of E = 6 enemies (number of input planes = 12 + E,
number of possible actions = 5 + E).

AUGUST 2019 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

15



IEEE Computational Intelligence Magazine - August 2019

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2019

Contents
IEEE Computational Intelligence Magazine - August 2019 - Cover1
IEEE Computational Intelligence Magazine - August 2019 - Cover2
IEEE Computational Intelligence Magazine - August 2019 - Contents
IEEE Computational Intelligence Magazine - August 2019 - 2
IEEE Computational Intelligence Magazine - August 2019 - 3
IEEE Computational Intelligence Magazine - August 2019 - 4
IEEE Computational Intelligence Magazine - August 2019 - 5
IEEE Computational Intelligence Magazine - August 2019 - 6
IEEE Computational Intelligence Magazine - August 2019 - 7
IEEE Computational Intelligence Magazine - August 2019 - 8
IEEE Computational Intelligence Magazine - August 2019 - 9
IEEE Computational Intelligence Magazine - August 2019 - 10
IEEE Computational Intelligence Magazine - August 2019 - 11
IEEE Computational Intelligence Magazine - August 2019 - 12
IEEE Computational Intelligence Magazine - August 2019 - 13
IEEE Computational Intelligence Magazine - August 2019 - 14
IEEE Computational Intelligence Magazine - August 2019 - 15
IEEE Computational Intelligence Magazine - August 2019 - 16
IEEE Computational Intelligence Magazine - August 2019 - 17
IEEE Computational Intelligence Magazine - August 2019 - 18
IEEE Computational Intelligence Magazine - August 2019 - 19
IEEE Computational Intelligence Magazine - August 2019 - 20
IEEE Computational Intelligence Magazine - August 2019 - 21
IEEE Computational Intelligence Magazine - August 2019 - 22
IEEE Computational Intelligence Magazine - August 2019 - 23
IEEE Computational Intelligence Magazine - August 2019 - 24
IEEE Computational Intelligence Magazine - August 2019 - 25
IEEE Computational Intelligence Magazine - August 2019 - 26
IEEE Computational Intelligence Magazine - August 2019 - 27
IEEE Computational Intelligence Magazine - August 2019 - 28
IEEE Computational Intelligence Magazine - August 2019 - 29
IEEE Computational Intelligence Magazine - August 2019 - 30
IEEE Computational Intelligence Magazine - August 2019 - 31
IEEE Computational Intelligence Magazine - August 2019 - 32
IEEE Computational Intelligence Magazine - August 2019 - 33
IEEE Computational Intelligence Magazine - August 2019 - 34
IEEE Computational Intelligence Magazine - August 2019 - 35
IEEE Computational Intelligence Magazine - August 2019 - 36
IEEE Computational Intelligence Magazine - August 2019 - 37
IEEE Computational Intelligence Magazine - August 2019 - 38
IEEE Computational Intelligence Magazine - August 2019 - 39
IEEE Computational Intelligence Magazine - August 2019 - 40
IEEE Computational Intelligence Magazine - August 2019 - 41
IEEE Computational Intelligence Magazine - August 2019 - 42
IEEE Computational Intelligence Magazine - August 2019 - 43
IEEE Computational Intelligence Magazine - August 2019 - 44
IEEE Computational Intelligence Magazine - August 2019 - 45
IEEE Computational Intelligence Magazine - August 2019 - 46
IEEE Computational Intelligence Magazine - August 2019 - 47
IEEE Computational Intelligence Magazine - August 2019 - 48
IEEE Computational Intelligence Magazine - August 2019 - 49
IEEE Computational Intelligence Magazine - August 2019 - 50
IEEE Computational Intelligence Magazine - August 2019 - 51
IEEE Computational Intelligence Magazine - August 2019 - 52
IEEE Computational Intelligence Magazine - August 2019 - 53
IEEE Computational Intelligence Magazine - August 2019 - 54
IEEE Computational Intelligence Magazine - August 2019 - 55
IEEE Computational Intelligence Magazine - August 2019 - 56
IEEE Computational Intelligence Magazine - August 2019 - 57
IEEE Computational Intelligence Magazine - August 2019 - 58
IEEE Computational Intelligence Magazine - August 2019 - 59
IEEE Computational Intelligence Magazine - August 2019 - 60
IEEE Computational Intelligence Magazine - August 2019 - 61
IEEE Computational Intelligence Magazine - August 2019 - 62
IEEE Computational Intelligence Magazine - August 2019 - 63
IEEE Computational Intelligence Magazine - August 2019 - 64
IEEE Computational Intelligence Magazine - August 2019 - 65
IEEE Computational Intelligence Magazine - August 2019 - 66
IEEE Computational Intelligence Magazine - August 2019 - 67
IEEE Computational Intelligence Magazine - August 2019 - 68
IEEE Computational Intelligence Magazine - August 2019 - 69
IEEE Computational Intelligence Magazine - August 2019 - 70
IEEE Computational Intelligence Magazine - August 2019 - 71
IEEE Computational Intelligence Magazine - August 2019 - 72
IEEE Computational Intelligence Magazine - August 2019 - 73
IEEE Computational Intelligence Magazine - August 2019 - 74
IEEE Computational Intelligence Magazine - August 2019 - 75
IEEE Computational Intelligence Magazine - August 2019 - 76
IEEE Computational Intelligence Magazine - August 2019 - Cover3
IEEE Computational Intelligence Magazine - August 2019 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com