IEEE Signal Processing - July 2018 - 74

tion is the denoised result. What have we done here? We imposed
suggest it is compressible due to low entropy, or even mention the
a model on our unknown, forcing the result to be likely under the
possibility of embedding it into a low-dimensional manifold. No
believed structure, thus enabling the denoising operation. The
matter what the assumption is, the bottom line is the same-the
same thought process underlies the solution of almost any task in
data we operate on is structured. This is true for images of varisignal and image processing and machine learning, either explicous kinds: video sequences; audio signals; three-dimensional
itly or implicitly.
(3-D) objects given as meshes or point clouds; financial time
As yet another example for a model in image processing and
series; data on graphs, as is the case in social or traffic networks;
its impact, consider the JPEG compression algorithm [15]. We
text files such as e-mails and other documents; and more. In fact,
are well aware of the impressive ability of this
we could go as far as stating that if a given
method to compress images by factor of ten to
data is unstructured (e.g., being independent
This article's main goal
20 with hardly any noticeable artifacts. What
and identically distributed random noise), it
is to present a novel
are the origins of this success? The answer
would be of no interest to us, since processtheory for explaining deep is twofold: the inner structure that exists in
ing it would be virtually futile.
images, and the model that JPEG harnesses to
So, coming back to our earlier question,
(convolutional) neural
the reason we can process data is the aforenetworks and their origins, exploit it. Images are redundant, as we already
mentioned structure, which facilitates this
all through the language of claimed, allowing for the core possibility of
such compression to take place. However, the
ability in all its manifestations. Indeed, the
sparse representations.
structure alone cannot suffice to get the actual
fields of signal and image processing and
compression, as a model is needed to capture
machine learning are mostly about identithis redundancy. In the case of JPEG, the model exposes this
fying the structure that exists in a given information source,
structure through the belief that small image patches (of size 8 × 8
and then exploiting it to achieve the processing goals. This
pixels) taken from natural images tend to concentrate their energy
brings us to discuss models and the central role they play in
in the lower-frequency part of the spectrum once operated upon
data processing.
by the discrete cosine transform (DCT). Thus, few transform coefficients can be kept while the rest can be discarded, leading to the
Identifying structure via models
desired compression result. One should nevertheless wonder, will
An appealing approach for identifying structure in a given inforthis algorithm perform just as well on other signal sources? The
mation source is imposing a (parametric) model on it, explicitly
answer is not necessarily positive, suggesting that every informastating a series of mathematical properties that the data is
tion source should be fitted with a proper model.
believed to satisfy. Such constraints lead to a dimensionality
reduction that is so characteristic of models and their modus operandi. We should note, however, that models are not the only
The evolution of models
avenue for identifying structure-the alternative being a nonA careful survey of the literature in image processing reveals an
parametric approach that simply describes the data distribution
evolution of models that have been proposed and adopted over
by accumulating many of its instances. We will not dwell on this
the years. We will not provide here an exhaustive list of all of
option in this article, as our focus is on models and their role in
these models, but we do mention a few central ideas such as
data processing.
Markov random fields for describing the relation between neighConsider the following example, brought to clarify our discusboring pixels [16], Laplacian smoothness of various sorts [17],
sion. Assume that we are given a measurement vector y ! R n,
total variation [18] as an edge-preserving regularization, wavelets' sparsity [19], [20], and Gaussian mixture models (GMMs)
and all that is known to us is that it is built from an ideal signal
[21], [22]. With the introduction of better models, performance
of some sort, x ! R n, contaminated by white additive Gaussimproved in a wide front of applications in image processing.
ian noise of zero mean and unit variance, i.e., y = x + e, where
Consider, for example, the classic problem of image denoise ~ N (0, I) . Could we propose a method to clean the signal y
ing, on which thousands of papers have been written. Our ability
from the noise? The answer is negative! Characterizing the noise
to remove noise from images has advanced immensely over the
alone cannot suffice for handling the denoising task, as there are
years. Indeed, the progress made has reached the point where this
infinitely many possible ways to separate y into a signal and a
problem is regarded by many in our field as nearly solved [23],
noise vector, where the estimated noise matches its desired statis[24]. Performance in denoising has improved steadily over time,
tical properties.
and this improvement was enabled mostly by the introduction of
Now, suppose that we are given additional information on the
better and more effective models for natural images. The same
unknown x, believed to belong to the family of piece-wise conprogress applies to image deblurring, inpainting, superresolution,
stant (PWC) signals, with the tendency to have as few jumps as
compression, and many other tasks.
possible. In other words, we are given a model for the underlying
In our initial description of the role of models, we stated that
structure. Could we leverage this extra information to denoise y?
these are expressing what the data is believed to satisfy, eluding
The answer is positive-we can seek the simplest PWC signal that
to the fact that models cannot be proven to be correct, just as a
is closest to y in such a way that the error matches the expected
formula in physics cannot be claimed to describe our world pernoise energy. This can be formulated mathematically in some
fectly. Rather, models can be compared and contrasted, or simply
way or another, leading to an optimization problem whose solu74

IEEE Signal Processing Magazine

July 2018

Table of Contents for the Digital Edition of IEEE Signal Processing - July 2018