Signal Processing - July 2017 - 100

high-quality landscape images differentiate from those
low-quality ones in a more apparent way. Similar observations are found, e.g., in the humorous and rural categories.
The observation explains why it could be inherently difficult for the baseline model to judge whether images
from some specific categories are aesthetically pleasing or
not, revealing yet another challenge in the assessment of
image aesthetics.

0.8
0.75
0.7

From generic aesthetics
to user-specific taste
fc7

fc6

c 53

c 52

c 51

c 43

c 42

c 41

c 33

0.6

c 32

0.65

c 31

Balanced Accuracy

0.85

Layer Feature

Rural
Landscape

Black and White
Humorous

FIGURE 16. A layer-by-layer analysis showing the difficulties of
understanding aesthetics across different categories. From the learned
feature hierarchy and the classification results, we observe that image
aesthetics in the landscape and rural categories can be judged reasonably
by the proposed baselines, yet the more ambiguous humorous and
black-and-white images are inherently difficult for the model to handle
(see also Figure 17).

Individual users may hold different opinions on the aesthetic
quality of any single image. One may consider that all of the
images in Figure 13 are of high quality to some extent, even
though the average scores by the data set annotators say otherwise. Coping with individual aesthetic bias is a challenging
problem. We may follow the idea behind transfer learning [83]
and directly model the aesthetic preference of individual users
by transferring the learned aesthetic features to fitting personal
taste. In particular, we consider that the DAN-1 baseline network has already captured a sense of generic aesthetics in the
aforementioned learning process; so to adapt to personal aesthetic preferences, one can include additional data sources for
positive training samples that are user specific, such as the
user's personal photographic album or the collection of photos
that the user "liked" on social media. As such, our proposed
baseline can be further fine-tuned with personal-taste data for
individual users and become a personalized aesthetic classifier.

this dth layer are of size w # h # K, where w # h is the size
of each of the K output maps. Denote M k as the kth output
map. Specifically, a point M ki, j in output map M k is computed
from a local patch region L of the input image I using the
forward propagation. By aligning all such points into a vector
Image aesthetic manipulation
A task closely related to image aesthetic assessment is image
v L = 6M 1i, j, M 2i, j, ..., M ki, j, ..., M Ki, j@, we obtained the feature representation of the local patch region L. A dictionary codebook
aesthetics manipulation, the aim of which is to improve the aeswas created using GMM from all of the " v L ,L ! I train, and an FV
thetic quality of an image. A full review of the techniques of
representation is subsequently computed using this codebook
image aesthetics manipulation in the literature is beyond the
to describe an input image. The obtained convolutional
scope of this article. Still, we make an attempt to connect image
Fisher representation is used for training SVM classifiers.
aesthetic assessment to a broader topic surrounding image aesWe compared features from layer
thetics by focusing on one of the major aesconv3_1 to fc7 of the DAN-1 baseline and
thetic enhancement operations, i.e.,
DAN-2 trained with random automatic image cropping.
reported selected results that we find interminibatches attains the
esting in Figure 16. We obtained the folhighest overall accuracy
lowing results:
Aesthetics-based image cropping
1) Model depth is important: More abstract
Image cropping improves the aesthetic
on the AVA standard
aesthetic representation can be learned
testing partition compared composition of an image by removing
in deeper layers. The performance of
undesired regions, increasing its aesthetic
to the previous state-ofaesthetic assessment can generally be
value. A majority of cropping schemes in
the-art methods.
benefited from model depth. This obserthe literature can be divided into three
vation aligns with that in general
main approaches. Attention/saliency-based
object recognition tasks.
approaches [99]-[101] typically extract the primary subject
2) Different categories demand different model depths: The
region in the scene of interest according to attention scores or
aesthetic classification accuracy on images belonging to
saliency maps as the image crops. Aesthetics-based approachthe black and white category are generally lower than the
es [102]-[104] assess the attractiveness of some proposed
accuracy on images in the landscape category across all of
candidate crop windows with low-level image features and
the layer features. Sample classification results are shown
rules of photographic composition. However, simple handin confusion matrix ordering (see Figure 17). High-quality
crafted features are not robust for modeling the huge aesthetic
black-and-white images show subtle details that should be
space. The state-of-the-art method is the change-based
considered when assessing their aesthetic level, whereas
approach proposed by Yan et al. [105], [106], which aims to
100

IEEE SIGNAL PROCESSING MAGAZINE

July 2017

Table of Contents for the Digital Edition of Signal Processing - July 2017