IEEE Signal Processing - July 2018 - 134

y = [y 1, y 2, f, y N ] T ! R N # 1, where N
is the number of images in the database
and supposing that the DMOS for the
images in the database is denoted as
z = [z 1, z 2, f, z N ] T ! R N # 1, then the
two residuals between the DMOS and
the two nonlinearly mapped scores
are d x = z - x and d y = z - y.
Sheikh et al. [7] assumed that d x =
[d x1, d x2, f, d xN ] T ! R N # 1 a nd d y =
[d y1, d y2, f, d yN ] T ! R N # 1 are the samples drawn from Gaussian distributions
with zero means. Therefore, testing
whether d x and d y are from the same
Gaussian distribution becomes testing
whether their population variances v 2x
and v 2y are the same.
The F-test is adopted in [7] to test the
equality of variances. The null hypothesis H 0 is v 2x v 2y = 1, and the alternative
hypothesis H 1 is v 2x v 2y ! 1. Given the
two sets of samples d x and d y, the F-test
statistic is calculated as
F=

s 2x
,
s 2y

(1)

where s 2x = (1/ (N -1)) R iN= 1 (d xi - drx) 2 and
s 2y = (1/ (N -1)) R iN= 1 (d yi - dry)2 a r e t h e
sample variances, and drx = (1/N ) R iN= 1 d xi
and dr y = (1/N ) R iN= 1 d yi are the sample
means. The F-test statistic has an F distribution with N - 1 and N - 1 degrees
of freedom. Therefore, the test conclusion
is drawn by comparing the value of F in
(1) with the critical values of an F distribution with N - 1 and N - 1 degrees of
freedom. Given the significance level a
(which usually takes values of 1, 5, or
10%), the null hypothesis is rejected if
F 2 F1 - a/2, N - 1, N - 1 or F 1 Fa/2, N - 1, N - 1,
where F1 - a/2, N - 1, N - 1 and Fa/2, N - 1, N - 1
are the critical values. Then, we conclude that d x and d y are from different Gaussian distributions. Otherwise, if
Fa/2, N - 1, N - 1 # F # F1 - a/2, N - 1, N - 1, we
do not reject the null hypothesis, and the
conclusion is that d x and d y are from the
same Gaussian distribution.

Why is the F-test unsuitable for
comparing IQA metrics?
The F-test assumes that d x and d y are
two independent samples from Gaussian populations. However, d x and d y
may be correlated because the samples
134

are paired; one sample d xi in d x is
uniquely paired with one sample d yi in
d y because they are calculated on the
same ith image. Such samples are called
paired samples in statistics. If the two
IQA metrics are both well designed,
their scores both decrease as the degree
of degradation increases in the same
image. Such correlations between scores
may also render the residuals d x and d y
as correlated. Empirical evidence of the
correlation between residuals is provided later in experimental results. Therefore, the conclusion drawn from the
F-test of whether d x and d y are from
the same distribution can be unreliable.

The Pitman test as a solution
In statistics, a hypothesis test for paired
samples is usually different from that for
independent samples. For example, the
t-test is used to test the equality of means
for independent samples, whereas the
paired t-test is used for paired samples.
For evaluating the equality of variances,
the Pitman test is designed for correlated
samples [9], [11], [12].
Here, we introduce the Pitman test to
examine the equality of variances for the
residuals of two IQA metrics. The null
hypothesis H 0 is v 2x = v 2y, and the alternative hypothesis H 1 is v 2x ! v 2y . The
Pitman test statistic is calculated as
t=

` 1 - s 2x s 2y j N - 2
4 (1 - r 2) ` s 2x s 2y j

,

(2)

where
N

r=

/ ^d xi - drxh^d yi - dryh

i=1
N

N

/ ^d xi - drxh2 / ^d yi - dryh2

i=1

i=1

(3)

is the Pearson correlation coefficient
between the two sets of samples d x and
d y . It is clear that in (2) the correlation
r is considered in the test statistic. The
Pitman test statistic exhibits a Student's
t distribution with N - 2 degrees of
freedom.
Similar to that in the F-test, the test
conclusion is drawn by comparing the
value of t in (2) with the critical values
of a t distribution with N - 2 degrees
of freedom.
IEEE Signal Processing Magazine

|

July 2018

|

The F-test versus the Pitman test
In Figure 1, we illustrate the use of the
F-test and the Pitman test in comparing
IQA metrics. Two IQA metrics M x and
M y are applied to the same IQA database, providing two residuals d x and d y,
respectively. In the comparison of d x and
d y by using the F-test, two assumptions
are applied: 1) independence between d x
and d y and 2) normality of d x and d y, as
shown in Figure 1(a). By contrast, when
the Pitman test is used, the only assumption is the normality of d x and d y, as
shown in Figure 1(b). Given that d x and
d y are paired and correlated, the Pitman
test is more appropriate to test the equality of variances than the F-test.

Experimental results
In the following experiments, we aim to
test whether d x and d y are from Gaussian distributions with the same variances
on the LIVE database. We show that different conclusions can be drawn from the
F-test and the Pitman test.
Following the experiments in [7],
all experiments are performed on five
types of degradations (JPEG2000, JPEG,
Gaussian noise, Gaussian blur, and fastfading wireless) separately and then on
the overall database.
We compare the scores of the following seven IQA metrics: FSIM [5], GSM
[6], most apparent distortion (MAD) [13],
MS-SSIM [3], NQM [1], peak signal-tonoise ratio (PSNR), and SSIM [2]. All
scores and their nonlinearly mapped
scores are obtained from http://sse.tongji
.edu.cn/linzhang/IQA/IQA.htm. The significance levels of the F-test and the Pitman test are both set to 5%.

Are two conclusions different?
The results show that, for all types of
degradations and the overall database,
the F-test does not always produce the
same conclusion as that by the Pitman
test regarding whether two IQA metrics are statistically significantly different. Here, we show two examples on
the overall database and the Gaussian
noises in Table 1.
A total of 21 pairs of IQA metrics are
compared in the experiments. For the
overall database, we obtain 19 same conclusions and two different conclusions
http://sse.tongji.edu.cn/linzhang/IQA/IQA.htm http://sse.tongji.edu.cn/linzhang/IQA/IQA.htm
Table of Contents for the Digital Edition of IEEE Signal Processing - July 2018
