Signal Processing - July 2017 - 83

focus was on distinguishing noisy images from clean ones in
terms of a different quality measure rather than artistic/photographic aesthetics.
In this article, we contribute a thorough overview of the
field of image aesthetic assessment. Meanwhile, we also
cover the basics of deep-learning methodologies. Specifically, as different data sets exist and evaluation criteria vary
in the image aesthetics literature, we do not aim to directly
compare the system performance of all of the reviewed works;
instead, we point out in the survey their main contributions
and novelties in model designs, and give potential insights for
future directions in this field of study. In addition, following
the recent emergence of deep-learning techniques and the
effectiveness of the data-driven approach in learning better
image representation, we systematically evaluate different
techniques that could facilitate the learning of a robust deep
classifier for aesthetic scoring. Our study covers topics such as
data preparation, fine-tuning strategies, and multicolumn deep
architectures, which we believe to be useful for researchers
working in this domain.
In particular, we summarize useful insights on how to alleviate the potential problem of data distribution bias in a binary
classification setting and show the effectiveness of rejecting
false-positive predictions using our proposed convolutional
neural network (CNN) baselines, as revealed by the balanced
accuracy metric. We also review the most commonly used
publicly available image aesthetic assessment data sets for
this problem and draw connections between image aesthetic
assessment and image aesthetic manipulation, including image
enhancement, computational photography, and automatic
image cropping.

This is typically followed by a nonlinear function, such
as sigmoid
z=

1
1 + exp (- y)

(2)

or the rectified linear unit z = max (0, y), which acts as the
activation function and produces the net activation output z.
To learn the weights W in a data-driven manner, we need
to have the feedback information that reports the current
performance of the network. Essentially, we are trying to tune
the knobs W to achieve a learning objective. For example,
given an objective t for the input x, we want to minimize the
squared error between the net output z and t by defining a loss
function L:
L = 1 z - t 2.
2

(3)

To propagate this feedback information to the weights,
we define the backward operation for each layer using gradient backpropagation [33]. We hope to get the direction DW
to update the weights W to better suit the training objective
(i.e., to minimize L ): W ! W - hDW, where h is the learning rate. In our example, DW can be easily derived based on
the chain rule:
DW = 2L
2W
2y
2
= L 2z
2z 2y 2W
exp (- y)
= (z - t ) ·
·x.
(exp (- y) + 1) 2

(4)

Background
The deep neural network
The deep neural network belongs to the family of deep-learning methods that are tasked to learn feature representation in a
data-driven approach. While shallow models (e.g., SVM and
boosting) showed success in earlier studies concerning relatively smaller amounts of data, they require highly engineered feature designs in solving machine-learning problems. Common
architectures in deep neural networks consist of a stack of
parameterized individual modules that we call layers, such as
the convolution layer and the fully connected layer. The architecture design of stacking layers on top of layers is inspired by
the hierarchy in the human visual cortex ventral pathway, offering different levels of abstraction for the learned representation
in each layer. Information propagation among layers in feedforward deep neural networks typically follows a sequential
pattern. A forward operation F (·) is defined respectively in
each layer to propagate the input x it receives and produces an
output y to the next layer. For example, the forward operation
in a fully connected layer with learnable weights W can be
written as
y = F (x) = Wx = / w ij ·x i.

(1)

In practice, researchers resort to batch stochastic gradient
descent or more advanced learning procedures that compute
more stable gradients, as averaged from a batch of training
examples {(x i, t i) | x i ! X} to train deeper and deeper neural networks with continually increasing numbers of layers.
We refer readers to [27] for an in-depth overview of additional
deep-learning methodologies.

Image-quality metrics
Image-quality metrics are defined in an attempt to quantitatively measure the objective quality of an image. This is typically used in image restoration applications (superresolution
[34], deblurring [35], and deartifacting [36]), where we have
a default high-quality reference image for comparison.
However, these quality metrics are not designed to measure
the subjective nature of human-perceived aesthetic quality
(see examples in Figure 2). Directly applying these objective quality metrics to our domain of image aesthetic
assessment may produce misleading results, as can be seen
from the measured values in Figure 2(b). Interest in developing more robust metrics has increased in the research
community, as a means to assess the more subjective quality of image aesthetics.

IEEE SIGNAL PROCESSING MAGAZINE

July 2017

Table of Contents for the Digital Edition of Signal Processing - July 2017