IEEE Circuits and Systems Magazine - Q3 2020 - 14

this end, some hard restrictions have been established
in order to select a communication interface for sending
a given message (Table III). With these restrictions and
considering the characteristics of each RAT (Table II), we
have developed an automatic labelling scheme as follows.
LoRaWAN interface is preferred for small packets with low
urgency, given its highly reduced power consumption. Nevertheless, it is not the best option for transmitting urgent
packets because of its limited throughput and the long latency introduced. In turn, NB-IoT interface is more suitable
for transporting urgent messages, given its lower latency
and higher data-rate. However, its more limited sensitivity
causes that its usage is not recommended with a coverage
level below 10%. The same applies to the battery level,
when the remaining energy is below 10%, using the NB-IoT
interface is discarded due to its higher power consumption. When one of these situations happen, short urgent
packets are sent via the LoRaWAN interface if it is available
(L = true). Of course, considering the mentioned restrictions, some combinations of the input features may lead to
the decision of dropping messages.
D. TinyML integration and evaluation
Given the notoriety and mass adoption of Arduino Uno,
this board has been chosen as the testbench for evaluating the performance of TinyML frameworks. We consider
it as a good candidate for conducting this benchmarking
due to its processing and memory constraints, as it is
equipped with an 8 bit processor at 16 MHz, flash memory of 32 KB, and 2 kB of Static RAM (SRAM). According
to the device classification defined in [8], this platform
belongs to the most constrained type of MCUs (Class 0).
Taking a look at Table I, the only publicly available
frameworks that can produce code for Arduino Uno
are MicroMLGen [46], sklearnporter [47], m2cgen [48],
and emlearn [51]. The workflow followed by these tools
is similar in all of them: firstly, a selected algorithm is
trained using a Python-based ML library; thereafter, the
model generated by this library is ported to C code by
the corresponding TinyML translation tool; finally, the
produced code is integrated within an Arduino sketch
by importing it as an external library.
Regarding m2cgen, as discussed above, although this
framework is highly versatile as it supports many algorithms and output languages, it does not generate opti-

mized code in terms of memory allocation. We proved this
fact by porting to C code a number of models produced by
Scikit-learn, namely, RF, decision trees, and SVM. In every
case, m2cgen generated non-compilable code for Arduino
Uno due to unallowed memory and object manipulations.
MicroMLGen is specialized in translating Scikitlearn's SVM models to Arduino-like platforms. It is able
to handle different SVM kernels such as linear, polynomial, and Radial Basis Function (rbf). SVMs are highly
sensitive to the number of training samples as they determine the number of generated support vectors which
have a direct impact on the required memory. Table IV
summarizes the attained results in terms of accuracy
and F1-score provided by Scikit-learn as well as program
memory footprint and classification time in Arduino
Uno. Regarding the latter, we present the average running time for 100 random input vectors as well as the
confidence interval ^a = 0.05 h . The alpha SVM parameter has been fixed to 0.0001. It can be seen how the classifier accuracy is improved by increasing the number of
training samples, no matter the kernel employed. The linear kernel is the fastest one and shows better accuracy
than the rbf kernel, which presents the greatest memory
footprint. Observe that both of them surpass the maximum MCU's program memory when employing 1000 and
1250 training e
- xamples due to the large number of generated support vectors. Finally, the polynomial kernel is
the most accurate one and also presents the most moderate memory footprint. However, it is the slowest kernel
although always under the threshold of 200 ms.
sklearnporter has been evaluated when dealing with
SVMs as well. However, the produced C code from the
Scikit-learn's models is not optimized for constrained
devices as mentioned previously, hence it provoked an
unstable behaviour of the Arduino unit. Given the inconsistent performance obtained, we could not collect reliable outcomes to present here.
As discussed above, the family of NN-based algorithms is the one that has received greater attention
from the TinyML community. In this case, we have
evaluate the performance of the MLP implementation
provided by the emlearn framework. We have employed
Scikit-learn for producing trained networks with different number of hidden layers and neurons per layer, using the sigmoid activation function. As the number of

Table III.
Feature restrictions for generating the dataset.
RAT

Battery
Level (B) (%)

Coverage (C)
(%)

LoRaWAN
Availability (L)

Packet Size (S)
(bytes)

Urgency (U)
(%)

LoRaWAN

True

<240

<50

NB-IoT

>10

IEEE CIRCUITS AND SYSTEMS MAGAZINE

THIRD QUARTER 2020

IEEE Circuits and Systems Magazine - Q3 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2020