Signal Processing - July 2017 - 85

Data sets
The assessment of image aesthetic quality assumes a standard
training set and testing set containing both high-quality and
low-quality image examples, as previously mentioned. Judging the ground-truth aesthetic quality of a given image is,
however, a subjective task. As such, it is inherently challenging to obtain a large amount of such annotated data. Most of
the earlier papers [21], [38], [39] on image aesthetic assessment collect a small amount of private image data. These data
sets typically contain from a few hundred to a few thousand
images, with binary labels or aesthetic scoring for each image.
Yet such data sets where the model performance is evaluated
are not publicly available. Much research effort has later been
made to contribute publicly available image aesthetic data sets
of larger scale for more standardized evaluation of model performance. In the following, we introduce those data sets that
are most frequently used in performance benchmarking for
image aesthetic assessment.
The Photo.net data set and the DPChallenge data set are
introduced in [28] and [60], respectively. These two data sets
can be considered the earliest attempts to construct large-scale
image databases for image aesthetic assessment. The Photo.net
data set contains 20,278 images, with at least ten score ratings
per image. The ratings range from zero to seven, with seven
assigned to the most aesthetically pleasing photos. Typically,
images uploaded to Photo.net are rated as somewhat pleasing,
with the peak of the global mean score skewing to the right
in the distribution [28]. The more challenging DPChallenge
data set contains diverse ratings. The DPChallenge data set
contains 16,509 images in total, and was later replaced by the
Aesthetic Visual Analysis (AVA) data set, where a significantly
larger number of images derived from DPChallenge.com are
collected and annotated.
The Chinese University of Hong Kong-PhotoQuality
(CUHK-PQ) data set is introduced in [45] and [61]. It contains
17,690 images collected from DPChallenge.com and amateur
photographers. All of the images are given binary aesthetic
labels and grouped into seven scene categories, i.e., animals,

plants, static, architecture, landscape, humans, and night. The
standard training and testing set from this data set are random partitions of a 50-50 split or a fivefold cross-validation
partition, where the overall ratio of the total number of posi: .
tive examples and that of the negative examples is around 13
Sample images are shown in Figure 3.
The AVA data set [49] contains ~250,000 images in total.
These images are obtained from DPChallenge.com and labeled by aesthetic scores. Specifically, each image receives
78 ~ 549 votes of scores ranging from one to ten. The average score of an image is commonly taken to be its groundtruth label. As such, it contains more challenging examples, as
images that lie within the center score range could be aesthetically ambiguous [Figure 4(a)]. For the task of binary aesthetic
quality classification, images with an average score higher
than a threshold of 5 + v are treated as positive examples, and
images with an average score lower than 5 - v are treated as
negative ones. Additionally, the AVA data set contains 14 style
attributes and more than 60 category attributes for a subset of
images. There are two typical training and testing splits from
this data set, i.e., 1) a large-scale standardized partition with
~230,000 training images and ~20,000 testing images using
a hard threshold of v = 0, and 2) an easier partition modeling
that of CUHK-PQ by taking those images whose score ranking is at the top 10% and the bottom 10%, resulting in ~25,000
images for training and ~25,000 images for testing. The ratio
of the total number of positive examples to that of the negative
examples is around 12: 5.
Apart from these two standard benchmarks, more recent
research also introduces new data sets that take into consideration the data-balancing issue. The Image Aesthetic Data Set
(IAD) introduced in [55] contains 1.5 million images derived
from DPChallenge and Photo.net. Similar to AVA, images in
the IAD data set are scored by annotators. Positive examples
are selected from those images with a mean score larger than
a threshold. All IAD images are used for model training, and
the model performance is evaluated on AVA in [55]. The ratio
of the number of positive examples to that of the negative

~13,000
~4,500
CUHK-PQ Data Set
Number of
Positive Images
Number of
Negative Images
(a)

(b)

FIGURE 3. Some sample images in the CUHK-PQ data set [45]. (a) Distinctive differences can be visually observed between the high-quality (grouped in
the green-framed box) and low-quality images (grouped in the red-framed box). (b) The number of images in the CUHK-PQ data set.
IEEE SIGNAL PROCESSING MAGAZINE

July 2017

http://www.DPChallenge.com http://www.Photo.net http://www.Photo.net http://www.Photo.net http://www.DPChallenge.com http://www.Photo.net http://www.DPChallenge.com

Table of Contents for the Digital Edition of Signal Processing - July 2017