Signal Processing - July 2017 - 95

We further visualize some categories in the learned aesthetic
embedding space in Figure 10. It is interesting to observe that
the embedding learned with triplet loss demonstrates much
better aesthetic grouping in comparison to that without the use
of triplet loss.

Table 7. Triplets pretraining and multitask learning.
Methods

Balanced
Accuracy

Overall
Accuracy

DAN-1

72.82

74.06

DAN-1 (triplet pretrained)

73.29

75.32

DAN-1 (multitask-aesthetic and category)

73.39

75.36

DAN-1 (triplet pretrained + multitask)

73.59

74.42

Multitask learning with image category prediction
Can aesthetic prediction be facilitated provided that a model
understand to which category the image belongs? Following
the work in [94], where auxiliary information is used to
regularize the learning of the main task, we investigate the
potential benefits of using image categories as an auxiliary
label in training the aesthetic quality classifier.
Specifically, given an image labeled with main task label
y, where y = 0 for low-quality images and y = 1 for highquality ones, we provide an auxiliary label c ! C denoting
one of the image categories, such as animals, landscape,
portraits, and so forth. In total, we include 30 image categories. To learn a classifier for the auxiliary class, a new fully
connected layer is attached to the fc7 of the vanilla VGG-16
structure to predict a softmax probability for each category
class. The modified one-column CNN baseline architecture is shown in Figure 9(b). The loss function in (8) is now
changed to

Using a one-column CNN baseline (DAN-1) with VGG-16 as the base network.
Balanced minibatch formation is used.

To enforce such a relationship in an aesthetic embedding,
one needs to generate minibatches of triplets for deep feature
learning, i.e., an anchor x, a positive instance x +ve of the same
class, and a negative instance x -ve of a different class. Furthermore, we found it useful to constrain each image triplet
to be selected from the same image category. In addition, we
observed better performance by introducing triplet loss in the
pretraining stage and continuing with conventional supervised
learning on the triplet-pretrained model. Table 7 shows that the
DAN model pretrained with triplets gives better performance.

25
20
15
10
5
0
-5
-10
-15
-20

Low Quality

(a)

20

15

10

5

0

-10
-10

-5

-15
-15

20

15

10

5

0

-5

20

15

10

5

0

-5

-10

-15

25
20
15
10
5
0
-5
-10
-15
-20
-25
-20

0
5
10
15
20
25

-5

-25
-20
-15
-10

-5
-10
-15
-20

-20

20

15

5

Portraiture

25
20
15
10
5
0
-5
-10
-15
-20

15
10
5
0

-20

Landscape

10

0

-5

-10

-20

0
5
10
15
20
25

-5

-25
-20
-15
-10

-5
-10
-15
-20

25
20
15
10
5
0
-5
-10
-15
-20
-25
-15

15
10
5
0

High Quality

(b)

(c)

FIGURE 10. Aesthetic embeddings of AVA images (testing partition) learned by triplet loss, visualized using t-SNE [84]: (a) ordinary supervised learning without
triplet pretraining and multitask learning, (b) triplet pretrained, and (c) combined triplet pretraining and multitask learning. t-SNE: t-distributed stochastic
neighbor embedding.
IEEE SIGNAL PROCESSING MAGAZINE

|

July 2017

|

95
Table of Contents for the Digital Edition of Signal Processing - July 2017
