IEEE Computational Intelligence Magazine - November 2019 - 29

Based on this, the update of the Q-value in AI
router i can be rewritten as follows:
Q i (o t, a t) ! D i (s, a) + mQ i (o t + 1, a t + 1).

(8)

Based on the difference reward single, the
centralized network mind can continuously
revise the strategy of each router. The underlying
distributed intelligence can be trusted to adapt
correctly to changes in the network state, thereby reducing the need for reaction, recomputation and updating of the centralized AI platform.

Based on the difference reward single, the
centralized network mind can continuously revise
the strategy of each router. The underlying
distributed intelligence can be trusted to adapt
correctly to changes in the network state, thereby
reducing the need for reaction, recomputation and
updating of the centralized AI platform.

5.2. Simulation

In this section, we present simulation results to demonstrate the
feasibility and performance of our architecture and algorithm.
In our experiment, we focused on the congestion control
problem for the distributed routing paradigm, which is difficult
for traditional routing algorithms to address.
Our experimental environment was developed based on
[26]; we simulated a network with 4 nodes and 6 unidirectional links and generated 400 data packets to be routed
through the network. For simplicity, all these packets started
at the same source node and were sent to the same destination node. Each packet was routed in a distributed manner by
the AI routers.
As shown in Fig. 9, we compared our algorithm with a
deterministic routing strategy and a single-agent RL algorithm. For the deterministic routing strategy, all data packets
were routed along the same path. This strategy cannot respond
to the network state in a timely manner; thus, it will lead to
serious congestion problems and achieve an extremely low
global utility. In contrast, RL can dynamically adapt to the
congestion state of the network. However, due to the nonstationary environment of the MAS, the learning process for single-agent RL suffers from severe non-convergence, also
resulting in a relatively low global score, as shown in Fig. 9. By
contrast, in our architecture, a difference reward is introduced
to modify the reward signal to enhance the collective behavior
of the AI routers, thereby improving the global utility of the
whole system.

(GPU) for image processing. Similarly, to meet the requirements of the AI-driven networking age, there is an urgent need
for a specific AI networking processor [27].
Current networks generate millions of different types of
flows every millisecond. Running AI algorithms on such
massive volumes of data is extremely challenging. The computing power of current routers is far from being able to satisfy the requirements for AI&ML deployment. Recently, as
highly parallel, multicore, multithreaded processors, GPU and
Tensor Processing Unit (TPU) chips have become a cornerstone of the AI age. Some studies have already shown that a
GPU can offer improved packet processing capabilities [28].
However, due to the need for high-speed processing of massive amounts of data (more than 10 Gb/s) and the stringent
response delay requirements (less than 1 ms) for future networks, there is still a large gap between universal AI processing chips and their actual deployment prospects in the
networking field.
6.2. Advanced Software Systems

Currently, the handling of network data is posing challenges
typical of big data; recent years have seen a 3-fold increase in
total IP traffic and a >60% increase in the number of devices
deployed and the amount of telemetry data streamed in near

5.5

4.5

6.1. New Hardware Architectures

Every innovation with regard to upper-level services is based
on significant advances in the performance of the underlying
hardware, such as the Central Processing Unit (CPU) for general-purpose computations, the Digital Signal Processor (DSP)
for a communication system, and the Graphics Processing Unit

Global Utility

6. Challenges and Open Issues

AI&ML-driven networking control is a promising paradigm
for future networks, but many challenges still remain, and
much more work needs to be done. In this section, we will
discuss the major challenges and open issues regarding
AI&ML-driven networking.

Deterministic
Different
Local

5.0

4.0
3.5
3.0
2.5
2.0
1.5
0

10,000

20,000
Iterations

30,000

40,000

FIGURE 9 The global utility of the whole system.

NOVEMBER 2019 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - November 2019

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2019