Signal Processing - November 2017 - 136

Table 1. A comparison of implementations of CNN-based picture-quality prediction models.
Models

Type

Layer Depth

Preprocessing

Feature Aggregation or Score Pooling

[23]

NR

2 Conv and 2 FC

Local normalization

Mean pooling (during testing)

[24]

NR

14 Conv (4 NiN blocks)

Local normalization

Mean pooling (during testing)

[25]

NR

10 Conv and 2 FC

Raw RGB image

Mean or weighted average pooling

[26]

NR

2 Conv and 6 FC

Local normalization

Mean and standard deviation aggregation

[27]

NR

8 Conv and 3 FC

Low-frequency subtraction

Mean or weighted average aggregation

[28]

FR

(2 Conv, 1 FC)×2 and 2 FC

Local normalization

(Not mentioned)

[29]

FR

13 Conv and 3 FC

Raw RGB image

Mean aggregation and pooling

[30]

FR

(2 Conv)×2, 6 Conv and 2 FC

Low-frequency subtraction

Weighted average aggregation

Training Targets
Models

Type

First Stage

Comments
Second Stage

(Comparison strategy for FR models)

[23]

NR

Subjective scores

N/A

Patchwise training

[24]

NR

Semantic label

Subjective scores

Fine-tuning of pretrained CNN on ImageNet

[25]

NR

Subjective scores

N/A

Weighted average patch aggregation

[26]

NR

Proxy scores

Subjective scores

Uses proxy patch labels

[27]

NR

Objective error map

Subjective scores

Uses proxy patch labels

[28]

FR

Subjective scores

N/A

Concatenation of feature vectors

[29]

FR

Semantic label

N/A

SSIM between feature maps of each layer

[30]

FR

Subjective scores

N/A

Concatenation of feature maps

FR: full-reference, NR: no-reference, Conv: convolutional layers, and FC: fully connected layers.

deep picture-quality models is also an issue. While simpler models often use perceptually relevant bandpass processing and local
divisive normalization [23], similar processes may be learned by
the network at the early stages. However, it should be possible to
impose perceptual weighting or pooling strategies on the network
to account for aspects of visual sensitivity, which could accelerate
the process of training on subjective scores.
In CNN-based schemes, the process of feature aggregation or
score pooling determines the form of a loss function. Examples
of aggregation and pooling strategies are shown in Figure 4. The
patch-based algorithms described in [23] and [24] did not use
aggregation or pooling during training. Instead, each image patch
was independently regressed onto the global subjective-quality
score. The loss function used is
	

N

L = 1 / f ^ p i h - S , (1)
N i

where p i refers to the ith patch obtained, N is the number of
patches, S is the ground-truth score, and f ($) is an NN process.
The models were trained via a patchwise optimization, and, during testing, the outputs of multiple patches composing an input
image were averaged to obtain a final predicted subjective score.
Conversely, imagewise approaches use aggregation or pooling
during training. For example, weighted average pooling methods
[25] may be used, where the loss function looks like
	
136

Ll = pool ^ f ^ p 1h, ...f ^ p N hh - S , (2)

where pool ($) refers to an unspecified pooling method [Figure 4(a)]. In [26] and [27] [Figure 4(b) and (c)], simple feature
aggregation was used. A more complicated model, such as a
multilayer perception or recurrent NN [4], could also be used
for aggregation [Figure 4(d)]. Here, the loss function becomes
L m = g ^aggr ^ f ^ p 1 h, ...f ^ p N hhh - S , (3)

	

where aggr (·) refers to a feature aggregation process and g (·)
is a regression NN. The forms (2) and (3) have the advantage
that the model can be trained under the same conditions
as the actual testing conditions, where the imagewise scores
are predicted.

Description of picture-quality databases
The choice and consideration of a database for training is
important for deep-learning-based models, since their performance depends highly on the size of the training set. In
most picture-quality databases, the distorted images are
afflicted by only a single type of synthetically introduced distortion, such as JPEG compression, simulated sensor noise,
or simulated blur, as exemplified in Figure 5(a). Since they
have played important roles in the development of perceptual
picture-quality studies, we briefly describe several popular
legacy databases in the following.
The LIVE IQA database [12], which was the first successful public-domain picture-quality database and is still the most
widely used, contains 29 reference images and 982 images, each

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|
Table of Contents for the Digital Edition of Signal Processing - November 2017
