IEEE Circuits and Systems Magazine - Q2 2023 - 25

Table 6.
Comparison of different model configurations on
one-billion [39]. Tensorized Transformer is the
model with multi-linear attention. Core-1 denotes
a model with a single-block term tensor, whereas
core-2 denotes a model with two block term
tensors.
Model
RNN-1024+9 Gram [72]
LSTM-2018-512 [73]
GCNN-14 bottleneck [74]
LSTM-8192-1024+CNN
Input [73]
High-Budget MoE [75]
LSTM+Mos [76]
Transformer+adaptive
input [77]
Transformer-XL Base [78]
Transformer-XL Large [78]
Tensorized Transformer
core-1 [39]
Tensorized Transformer
core-2 [39]
Params
20B
0.83B
-
1.04B
5B
113M
0.46B
0.46B
0.8B
0.16B
0.16B
Test PPL
51.3
43.7
31.9
30.0
28.0
37.10
23.7
23.5
21.8
20.5
19.5
tasks [36]. Three datasets were chosen: one of small
size (PTB), one of medium size (WikiText-103) and one
of large size (One-billion). PTB contains 929,900 training
tokens, 73,900 validation words, and 82,900 test words
[71]. WikiText-103 has 267,735 distinct tokens. The dataset
is a long-term dependency word-level language modeling
benchmark. It contains 103 million training tokens
from 28 thousand articles, with an average length of
3.6 thousand tokens per article. The One-Billion Word
benchmark is a large dataset with 829 250 940,, tokens
over a vocabulary of 793,471 words. Models were evaluated
based on the average per-word log-probability,
Perplexity (PPL). The lower the PPL, the more accurate
the model. The standard multi-head attention layers in
Transformer were replaced with Multi-linear attention.
A comparison of different model configurations on different
datasets is shown in Tables 6 and 7. Notice that
the tensorized transformer with Multi-linear attention
achieves lower PPL with much fewer parameters than
other models in the three datasets.
Neural Machine Translation involves translating
text or speech from one language to another. The
baseline is a vanilla Transformer trained on WMT
2016 English-German dataset [83]. For comparison,
Table 7.
Comparison of different model configurations on PTB and WiKiTEXT-103 [39]. Tensorized transformer is the model
with multi-linear attention. Core-1 denotes a model with a single-block term tensor, whereas core-2 denotes a
model with two block term tensors.
PTB
Model
Params
LSTM+augmented loss
[79]
Variational RHN [80]
4-layer QRNN [81]
AWD-LSTM-MoS [76]
Transformer+adaptive
input [77]
Transformer-XL-Base
[78]
Transformer-XL-Large
[78]
Transformer-XL+TT [38]
Sparse Transformer [82]
Tensorized Transformer
core-1 [39]
Tensorized Transformer
core-2 [39]
SECOND QUARTER 2023
24M
23M
-
22M
24M
24M
-
18M
14M
12M
12M
Val
PPL
75.7
67.9
-
58.08
59.1
56.72
-
57.9
74.0
60.5
54.25
Test
PPL
48.7
65.4
-
55.97
57
54.52
-
55.4
73.1
57.9
49.8
WikiText-103
Params
-
-
151M
-
247M
151M
257M
130M
174M
85.3M
85.3M
Val
PPL
-
-
-
29.0
19.8
23.1
-
23.61
38.98
22.7
19.7
Test PPL
48.7
45.2
33.0
29.2
20.5
24.0
18.3
25.70
40.23
20.9
18.9
IEEE CIRCUITS AND SYSTEMS MAGAZINE
25
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
