Signal Processing - July 2017 - 88

then encoded using the BOV words or FV. Spatial pyramid is
also adopted, and the per-region encoded FVs are concatenated as the final image representation. These methods ([47]-
[49]) represent an attempt to implicitly model photographic
rules by encoding them in generic content-based features,
which is competitive with or even outperforms simple handcrafted features.

Task-specific features
Task-specific features is a term that refers to features in image
aesthetic assessment that are optimized for a specific category
of photos, which can be efficient when the use-case or task
scenario is fixed or known beforehand. Explicit information
(such as human facial characteristics, geometry tag, scene
information, or intrinsic character component properties) is
exploited based on the different task nature.
Li et al. [70] propose a regression model that targets only
consumer photos with faces. Face-related social features (such
as facial expression features, facial pose features, and relative
facial position features) and perceptual features (facial distribution symmetry, facial composition, and pose consistency) are
specifically designed for measuring the quality of images with
faces, and it is shown in [70] that for this task they complement
conventional handcrafted features (brightness contrast, color
correlation, clarity contrast, and background color simplicity).
Support vector regression is used to produce aesthetic scores
for images.
Lienhard et al. [71] study particular facial features for
evaluating the aesthetic quality of headshot images. To
design features for face/headshots, the input image is divided into subregions (the eyes, mouth, global face, and entire
image regions). Low-level features (sharpness, illumination,
contrast, dark channel, and hue and saturation in the HSV
color space) are computed from each region. These pixellevel features assume the human way of perceiving a facial
image and hence can reasonably model the headshot images.
SVM with Gaussian kernel is used as the classifier.
Su et al. [72] propose bag of aesthetics-preserving features
for scenic/landscape photographs. Specifically, an image is
decomposed into n # n spatial grids; then low-level features
in HSV-color space as well as local binary patterns, HOG,
and saliency features are extracted from each patch. The final
feature is generated by a predefined patch-wise operation to
exploit the landscape composition geometry. AdaBoost is used
as the classifier. These features aim at modeling only landscape images and may be limited in their representation power
in general image aesthetic assessment.
Yin et al. [73] build a scene-dependent aesthetic model
by incorporating the geographic location information with
GIST descriptors and spatial layout of saliency features for
scene aesthetic classification (such as bridges, mountains,
and beaches). SVM is used as the classifier. The geographic
location information is used to link a target scene image
to relevant photos taken within the same geocontext; then
these relevant photos are used as the training partition to the
SVM. The authors' proposed model requires input images
88

with geographic tags and is also limited to scenic photos.
For scene images without geo-context information, SVM
trained with images from the same scene category is used.
Sun et al. [74] design a set of low-level features for aesthetic
evaluation of Chinese calligraphy. They target the handwritten
Chinese character on a plain white background; hence, conventional color information is not useful in this task. Global shape
features, extracted based on standard calligraphic rules, are
introduced to represent a character. In particular, the authors
consider alignment and stability, distribution of white space,
stroke gaps, and a set of component layout features while modeling the aesthetics of handwritten characters. A backpropagation neural network is trained as the regressor to produce an
aesthetic score for each given input.

Deep-learning approaches
The powerful feature representation learned from a large
amount of data has shown an ever-improving performance in
the tasks of recognition, localization, retrieval, and tracking,
surpassing the capability of conventional handcrafted features
[75]. Since the work by Krizhevsky et al. [75], where CNNs
are adopted for image classification, a great degree of interest
has arisen in learning robust image representations through
deep-learning approaches. Recent works in the literature of
image aesthetic assessment using deep-learning approaches
to learn image representations can be broken down into two
major schemes: 1) adopting generic deep features learned
from other tasks and training a new classifier for image aesthetic assessment and 2) learning aesthetic deep features and
training a classifier directly from image aesthetics data.

Generic deep features
A straightforward approach to employing deep-learning
aims is to adopt generic deep features learned from other
tasks and train a new classifier on the aesthetic classification task. Dong et al. [50] propose adopting the generic
features from the penultimate layer output of AlexNet
[75] with spatial pyramid pooling. Specifically, the
4,096 (fc7) # 6 (SpatialPyramid) = 24, 576 - d i m e n s i o n a l
feature is extracted as the generic representation for images; then an SVM classifier is trained for binary aesthetic
classification. Lv et al. [51] also adopt the normalized
4,096-dimension fc7 output of AlexNet [75] for feature
representation. They propose to learn the relative ordering
relationship of images of different aesthetic quality. They
use SVM rank [76] to train a ranking model for image
pairs of {I HighQuality, I LowQuality}.

Learned aesthetic deep features
Features learned with single-column CNNs
Peng et al. [52] propose to train CNNs of AlexNet-like architecture for eight different abstract tasks (emotion classification, artist classification, artistic style classification, aesthetic
classification, fashion style classification, architectural style
classification, memorability prediction, and interestingness

IEEE SIGNAL PROCESSING MAGAZINE

July 2017

Table of Contents for the Digital Edition of Signal Processing - July 2017