IEEE Computational Intelligence Magazine - August 2022 - 24

average pooling layer with a kernel of size 2; and a fully connected
layer of size 192 × 10.
The Python code of the privacy-preserving LeNet-1
is available in the same repository cited in Section I.
{ (·)H
B. Performance and Accuracy of the
Privacy-Preserving LeNet-1
To demonstrate the effectiveness and efficiency of the designed
privacy-preserving version of LeNet-1, we designed an experimental
campaign aimed at measuring and comparing the accuracy,
memory occupation (in MB) and computation time (in s)
of the original LeNet-1 (·)F
, the simplified LeNet-1 ()F $l
(without the second tanh activation), the approximated version
{ (·) of ()F $l
, and the outcome (·){H
of the methodology. The
experiments were conducted on a machine with an Intel
i7-4770K 64-bit CPU and 16 GB of RAM. The experimental
results, computed on the testing images from the MNIST and
Fashion-MNIST datasets, are detailed in Table V. Figure 5
graphically compares the accuracy, execution time, and memory
occupation on the Fashion-MNIST dataset. The following
three main comments can be made.
First, the simplified version ()F $l
of the original LeNet-1
F (·) provided accuracies of 98.22% on MNIST and 86.23% on
Fashion-MNIST. The drop in accuracy between (·)F and ()F $l
TABLE V Experimental results on the MNIST and FashionMNIST
datasets. Memory occupation (MB) and computation
time (s) refer to the processing of a single image. No
parallelization of the code is considered.
Symbol Model
LeNet-1
F()$
F ()$
l
{ ()$
{H ()$
LeNet-1 (single
tanh)
Approximated
F ()$
l
{ ()$ on
encrypted data
Accuracy (%)
86.88%
86.23%
85.29%
F(.)
F′(.)
ϕ(.)
ϕΘ(.)
98.18%
85.29% 780MB
138s
Accuracy
MNIST
Accuracy
Fashion
MNIST
Memory
Occup.
98.76% 86.88% 7.6MB
98.22% 86.23% 6.5MB
98.18%
85.29% 6.5MB
Comp.
Time
0.001s
0.0009s
0.0009s
was approximately 0.6% in both datasets. It was therefore crucial
to verify whether the drop in accuracy induced by the use
of ()F $l
(instead of (·)F ) was acceptable.
Second, as expected, the model approximation step of the procedure,
leading from ()F $l
to (·){ caused a loss of accuracy. This
loss was negligible in the case of the MNIST dataset (0.04%),
whereas it was more relevant in the case of the Fashion-MNIST
dataset (0.94%). This study speculated that the removal of nonlinearities
from LeNet-1 has a higher impact when more complex
tasks are considered (e.g., the recognition of fashion products
instead of simple digits). In addition, the accuracy of (·){ was
equal to that of (·){H
meaning that the privacy-preserving model
applied on encrypted images provided the same accuracy as that
obtained on plain images. These results confirmed that, thanks to
a correct choice of H and an accurate approximation of nonlinear
layers, the discrepancy between the plain and encrypted results
was negligible for the considered scenario.
Third, as expected, encrypted processing introduced a crucial
overhead in terms of memory usage and computation time
for the classification of a single image. Currently, HE libraries
do not support parallelization; hence, GPU support for HE is
only partially available with few examples present in the literature
(e.g., [26]).
This ends the second main contribution of this study, which
was the introduction of a methodology for designing privacypreserving
CNNs. The next section will discuss the main challenges
to be addressed in this research field.
V. Homomorphic Encryption and
Deep Learning: The Challenges
Designing privacy-preserving deep learning solutions is a novel
research area with several open research challenges to address.
This section, without aiming to be exhaustive, discussed three
main challenges in this research area that will be relevant over
the next few years:
❏ Automatic parameter configuration: The optimal parameter
configuration H is currently selected using a trial-anderror
approach. The challenge here lies in the study of
optimization algorithms, theoretical solutions, and meta-learning
algorithms (e.g., AutoML) to provide the optimal configuration
of H given a processing pipeline (·)F and a reference
dataset describing the problem to be addressed.
Time (s)
0.001 0.01 0.1 110 100 1,000
FIGURE 5 Time, accuracy, and memory occupation of the different
models for the Fashion-MNIST dataset. The dimensions of the circles
are representative of the memory occupation (in MB).
❏ Privacy-preserving recurrent neural networks and
transformers: Currently, the privacy-preserving deep
learning solutions in the literature have mostly focused on
CNNs, whose transformation in HE-compliant models is
easier than other deep learning solutions. One major challenge
in this field is the design of privacy-preserving recurrent
neural networks and transformers that are able to deal
with data sequences. The major issue to be addressed here is
the ability to manage the NB in processing pipelines where
data are sequentially processed over time.
❏ Training privacy-preserving models: The literature
regarding privacy-preserving machine and deep learning models
with HE focuses on the inference of privacy-preserving
24 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2022
IEEE Computational Intelligence Magazine - August 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2022
