Signal Processing - November 2017 - 51

networks. The transfer of these deep networks to critical applications that possibly consist in classifying high-stake information is seriously challenged by the low robustness of deep
networks. For example, in the context of self-driving vehicles,
it is fundamental to accurately recognize cars, traffic signs,
and pedestrians, when these are affected by clutter, occlusions,
or even adversarial attacks. In medical imaging [4], it is also
important to achieve high classification rates on potentially
perturbed test data. The analysis of state-of-the-art deep classifiers' robustness to perturbation at test time is therefore an
important step for validating the models' reliability to unexpected (possibly adversarial) nuisances that might occur when
deployed in uncontrolled environments. In addition, a better
understanding of the capabilities of deep networks in coping
with data perturbation actually allows us to develop important
insights that can contribute to developing yet better systems.
The fundamental challenges raised by the robustness of deep
networks to perturbations have led to a large number of important works in recent years. These works study empirically and
theoretically the robustness of deep networks to different types
of perturbations, such as adversarial perturbations, additive random noise, structured transformations, or even universal perturbations. The robustness is usually measured as the sensitivity of
the discrete classification function (i.e., the function that assigns
a label to each image) to such perturbations. While robustness
analysis is not a new problem, we provide an overview of the
recent works that propose to assess the vulnerability of deep
network architectures. In addition to quantifying the robustness
of deep networks to various forms of perturbations, the analysis of robustness has further contributed to developing important
insights on the geometry of the complex decision boundary of
such classifiers, which remain hardly understood due to the very
high dimensionality of the problems that they address. In fact,
the robustness properties of a classifier are strongly tied to the
geometry of the decision boundaries. For example, the high instability of deep neural networks to adversarial perturbations shows
that data points reside extremely close to the classifier's decision
boundary. The study of robustness is, therefore, not only interesting from the practical perspective of the system's reliability but
has a more fundamental component that allows "understanding"
of the geometric properties of classification regions and derives
insights toward the improvement of current a- rchitectures.
This overview article has multiple goals. First, it provides
an accessible review of the recent works in the analysis of
the robustness of deep neural network classifiers to different
forms of perturbations, with a particular emphasis on image
analysis and visual understanding applications. Second, it
presents connections between the robustness of deep networks
and the geometry of the decision boundaries of such classifiers. Third, the article discusses ways to improve the robustness in deep networks architectures and finally highlights
some of the important open problems.

Robustness of classifiers
In most classification settings, the proportion of misclassified
samples in the test set is the main performance metric used

to evaluate classifiers. The empirical test error provides an
estimate of the classifier's risk, defined as the probability of
misclassification, when considering samples from the data distribution. Formally, let us define n to be a distribution defined
over images. The risk of a classifier f is equal to
R ( f ) = xP~ ( f (x) ! y (x)), (1)

where x and y (x) correspond, respectively, to the image
and its associated label. While the risk captures the error of
f on the data distribution n, it does not capture the robustness to small arbitrary perturbations of data points. In visual
classification tasks, it is desirable to learn classifiers that
achieve robustness to small perturbations of the input; i.e.,
the application of a small perturbation to images (e.g., additive perturbations on the pixel values or geometric transformation of the image) should not alter the estimated label of
the classifier.
Before going into more detail about robustness, we first
define some notations. Let X denote the ambient space where
images live. We denote by R the set of admissible perturbations. For example, when considering geometric perturbations,
R is set to be the group of geometric (e.g., affine) transformations under study. Alternatively, if we are to measure the
robustness to arbitrary additive perturbations, we set R = X.
For r ! R, we define Tr: X " X to be the perturbation operator by r; i.e., for a data point x ! X, Tr (x) denotes the image
x perturbed by r. Armed with these notations, we define the
minimal perturbation changing the label of the classifier, at x,
as follows:
r ) (x) = argmin r

r!R

subject to f (Tr (x)) ! f (x), (2)

where · R is a metric on R. For notation simplicity, we omit
the dependence of r ) (x) on f, R, d, and operator T. Moreover,
when the image x is clear from the context, we will use r ) to refer
to r ) (x) . See Figure 2 for an illustration. The pointwise robustness of f at x is then measured by r ) (x) R . Note that larger
values of · R indicate a higher robustness at x. While this
definition of robustness considers the smallest perturbation r ) (x)
(with respect to the metric · Rh that causes the classifier f to

(a)

(b)

(c)

FIGURE 1. An example of an adversarial perturbations in state-of-the-art
neural networks. (a) The original image that is classified as a "whale," (b)
the perturbed image classified as a "turtle," and (c) the corresponding
adversarial perturbation that has been added to the original image to fool
a state-of-the-art image classifier [5].

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017