Signal Processing - November 2017 - 131

digital camera technology, digital television, streaming video services, and social media applications, is driving a critical need for
improved picture-quality monitoring. The pipelines from picture
content generation to consumption are fraught with numerous
sources of distortions, including blur, noise, and artifacts arising
from such processes as compression, scaling, format conversion,
color modification, and so on. Multiple interacting distortions
are often present, which greatly complicates the problem. Picture-quality models that can accurately predict human-quality
judgments can be used to greatly improve consumer satisfaction
via automatic monitoring of the qualities of massively distributed pictures and videos and to perceptually benchmark picture
processing algorithms such as compression engines, denoising algorithms, and superresolution systems that substantially
affect viewed picture quality. While many successful picturequality models have been devised, the problem is hardly solved,
and there remains significant scope for improvement [3]. Deeplearning engines offer a potentially powerful framework for
achieving sought-after gains in performance; however, as we
will explain, progress has been limited by a lack of adequate
amounts of distorted picture data and ground-truth subjective
quality scores, which are much harder to acquire than other
kinds of labeled image data. Furthermore, typical data-augmentation strategies such as those used for machine vision are of
little use on this problem.

Perceptual picture-quality prediction
Picture-quality models are generally classified according to
whether a pristine reference image is available for comparison.
Full-reference and reduced-reference models assume that a
reference is available; otherwise, the model is no-reference, or
blind. Reference models are generally deployed when a process
is applied to an original image, such as compression or enhancement. No-reference models are applied when the quality of an
original image is suspect, as in a source inspection process, or
when analyzing the output of a digital camera. Generally, noreference prediction is a more difficult problem.
Both reference and no-reference picture-quality models rely
heavily on principles of computational visual neuroscience and/
or on highly regular models of natural picture statistics [1]. Heretofore, the most successful no-reference models have relied on
powerful but shallow regression engines to achieve results that
approach the prediction accuracy of reference-quality predictors.

Deep learning and CNNs
Deep learning has had a transformative impact on such difficult
problems as speech recognition and image classification, achieving improvements in performance that are significantly superior to those obtained using conventional model-based methods
optimized using shallower networks. In particular, most of the
top-ranked image recognition and classification systems have
been optimized using CNNs. One of the principal advantages
of deep-learning models are the remarkable generalization capabilities that they can acquire when they are trained on large-scale
labeled data sets. Models learned using conventional machinelearning methods are heavily dependent on the determination

and discrimination capability of sophisticated training features.
By contrast, deep-learning models employ multiple levels of linear and nonlinear transformations to generate highly general data
representations, thereby greatly decreasing dependence on the
selection of features, which are often reduced simply to raw pixel
values [2], [4]. In particular, deep CNNs optimized for image
recognition and classification have greatly outperformed conventional methods. Open-source frameworks such as TensorFlow
[5] have also greatly increased the accessibility of deep-learning
models, and their application to diverse image processing and
analysis problems has greatly expanded.
Unlike traditional NNs, CNNs can be adapted to effectively
process high-dimensional, raw image data such as red, green, and
blue (RGB) pixel values. Two key ideas underlie a convolutional
layer: local connectivity and shared weights. Each output neuron of a convolutional layer is computed only on a locally connected subset of the input, called a local receptive field (drawing
from vision science terminology). However, by stacking multiple
convolutional layers, the effective receptive fields may enlarge
to capture global picture characteristics. Usually, the parameters
in a layer (i.e., filter weights) are shared across the entire visual
field to limit their number. A common conception is that CNNs
resemble processing by neurons in visual cortex. This idea largely
arises from the observation that, in deep convolutional networks
deploying many layers of adaptation on images, early layers of
processing often resemble the profiles of low-level cortical neurons in visual area V1, i.e., directionally tuned Gabor filters [6],
or neurons in visual area V2 implicated in assembling low-level
representations of image structure [7]. At early layers of network
abstraction, these perceptual attributes make them appealing
tools for adaption to the picture-quality prediction problem.
An example of a CNN structure similar to those studied here
is shown in Figure 1, which also illustrates the kernels learned
and the feature maps obtained when the model is trained for the
picture-quality prediction task. Generally, a CNN model consists of several convolutional layers followed by fully connected
layers. Some convolutional layers may be followed by pooling
layers, which reduce the sizes of the feature maps. The fully connected layers are essentially traditional NNs, where all of the
neurons in a previous layer are connected to every neuron in a
current layer.
Motivated by the great success of CNNs on numerous image
analysis applications, we comprehensively review and analyze
the use of deep CNNs on the picture-quality prediction problem.

Overview of the problem
Machine learning has played an important role in the development of modern picture-quality models. Although these models
have been largely driven by features drawn from meaningful
quantitative perceptual models, mapping them against the wide
variety of generally nonlinear, often commingled, and poorly
understood distortions that occur in practice is a formidable problem. Sophisticated, yet shallow mapping engines such as support
vector regressors (SVR), have produced good prediction results
(against human-quality opinions), yet there remains substantial
room for improvement [3], which greatly motivates the study of

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

131

Table of Contents for the Digital Edition of Signal Processing - November 2017