IEEE Consumer Electronics Magazine - September/October 2020 - 97

The obtained results let us state that the
most promising and generic approach is the Â2
ESRGAN reconstruction. In order to validate this
result, we have further considered a subjective
evaluation, involving ESRGAN Â 2 reconstruction and BPG-only compression, described in the
following section.

Evaluation and Quality Assessment
Assessing the quality of compressed images
is a highly complex issue that has been extensively studied in the literature. To the very best
of our knowledge, no universal solution exists
and this still remains an open issue of research.
From the origins, subjective evaluation protocols have been introduced in order to cope with
the limitations of objective metrics like PSNR
and structure SIMilarities scores, which are
the common metrics employed to evaluate the
quality of processed images. This issue is
highlighted by Chikkerur et al.,17 where authors
compare different objective measures. They
notably show that PSNR and SSIM measures are
poorly correlated with visual perception. In our
work, we have also encountered this difficulty.
For example, our ESRGAN Â 2 results show a
clear increase of the content/image perceptual
quality compared to its BPG counterpart, for the
majority of the Kodak image data set. However,
if the perceptual quality seems to be significantly
better, the PSNR and SSIM scores remain lower
in each image of the testing set (e.g., respectively
-3.3 dB and -0.06 in average over the considered test set).
Indeed, the loss functions used in our model
are perceptual losses that are not optimized in
regards to pixel-wise metrics. In fact, with a neural network as deep as our model, obtaining
a good score with an objective metric would be
an easy task. In order to check this hypothesis,
we have trained the same network, under the
same experimental conditions and on the same
data set by replacing the perceptual loss with an
MSE loss function. In average, the corresponding
PSNR (SSIM), scores are of 29.12 dB (0.81) and
26.67 dB (0.74) for the MSE loss and perceptual
loss, respectively. However, even if the MSErelated objective scores are superior, the visual
quality of the reconstructed images is significantly lower, as illustrated in Figure 6.

September/October 2020

Figure 6. ESRGANÂ2 reconstructed images with
MSE and perceptual losses. (a) MSE loss PSNR ¼
25.3 dB SSIM ¼ 0.58. (b) Perceptual loss PSNR ¼
23.7 dB SSIM ¼ 0.61. (c) MSE loss PSNR ¼ 30.91 dB
SSIM ¼ 0.86. (d) Perceptual loss PSNR ¼ 28.72 dB
SSIM ¼ 0.80.

These results confirm the conclusion presented by Ledig et al.,13 where authors highlight
that recent state-of-the-art SR techniques generally present low PSNR and SSIM scores, despite a
superior perceptual quality.
This aspect is also detailed in the ESRGAN16
article where authors show the impact of the
balance between perceptual and MSE-loss.

Subjective Evaluation of ESRGAN Â2-Based
Results
To validate our results and assess the quality of the reconstructed images, we have considered a subjective evaluation protocol. This
type of evaluation is much more time-consuming than an objective evaluation. Nevertheless,
it appears that with a sufficient number of evaluators,17 subjective evaluation appears to be
the most reliable image quality assessment
method to accurately measure the perceptual
quality.
However, in order to avoid bias and errors
related to the subjectivity of the assessment, a
strict evaluation protocol needs to be set up. In
our case, we have followed the standardized ITUR BT.500-13 recommendation.18 The evaluation

IEEE Consumer Electronics Magazine - September/October 2020

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - September/October 2020