Signal Processing - November 2017 - 140

Table 3. The SRCC and PLCC comparison on five public-domain subjective picture-quality databases.
LIVE IQA

CSIQ

TID2013

LIVE MD

LIVE Challenge

Type

Methods

SRCC

PLCC

SRCC

PLCC

SRCC

PLCC

SRCC

PLCC

SRCC

PLCC

PSNR

0.876

0.872

0.806

0.800

0.636

0.706

0.725

0.815

N/A

SSIM [15]

0.948

0.945

0.876

0.861

0.775

0.691

0.845

0.882

N/A

FSIMc [35]

0.963

0.960

0.931

0.919

0.851

0.877

0.863

0.818

N/A

DeepQA [30]

0.981

0.982

0.961

0.965

0.939

0.947

0.938

0.942

N/A

BRISQUE [9]

0.939

0.942

0.756

0.797

0.572

0.651

0.897

0.921

0.607

0.585

CORNIA [11]

0.942

0.943

0.714

0.781

0.549

0.613

0.900

0.915

0.618

0.662

FRIQUEE-ALL [21]

0.948

0.962

0.839

0.863

0.669

0.704

0.925

0.940

0.720

BIECON [26]

0.958

0.960

0.815

0.823

0.717

0.762

0.909

0.933

0.595

0.613

DIQA [27]

0.970

0.972

0.844

0.880

0.843

0.868

0.920

0.933

0.687

0.701

AlexNet + SVR

0.901

0.908

0.712

0.736

0.263

0.365

0.760

0.803

0.769

0.790

ResNet50 + SVR

0.925

0.935

0.654

0.700

0.435

0.495

0.797

0.833

0.806

0.825

AlexNet + fine-tuning

0.947

0.952

0.817

0.840

0.615

0.668

0.899

0.914

0.748

0.779

ResNet50 + fine-tuning

0.950

0.954

0.876

0.905

0.712

0.756

0.909

0.920

0.819

0.849

Imagewise CNN

0.963

0.964

0.812

0.791

0.800

0.802

0.914

0.929

0.663

0.705

FR: full reference, NR: no reference. Italics indicate CNN-based methods. Boldface entries indicate the top three performers on each database for each performance metric.

A possible explanation for these results is that the pretrained
deep models adapted easily to the authentic distortions in LIVE
Challenge as a consequence of having learned image recognition
tasks on real-world pictures. Applying them to databases with
synthetic distortions, however, like LIVE IQA and TID2013,
likely failed to exploit what was learned regarding authentic distortions; hence, significant retraining would be needed to deal
with the synthetic distortions. This may help explain the excellent
generalization power of pretrained models when applied to other
real world image tasks: their ability to handle authentic distortions, by representing them to improve task performance.

Envisioning the future
The sizes of the training sets used is critical to the success of
deep NN models. Current public-domain databases have insufficient size as compared to widely used image recognition databases. However, constructing large-scale perceptual-quality
databases is a much more difficult problem than image recognition databases. Creating databases for picture-quality assessment requires time-consuming and expensive subjective studies,
which must be conducted under controlled laboratory conditions. Even if the number of reference images is small, the
required number of subjective tests quickly becomes excessive.
Conducting subjective tests using online crowdsourcing is one
possible solution (like the LIVE Challenge database), yet even
online tests are (probably) prohibitively difficult to scale up
to the necessary size, especially while ensuring the aggregate
quality of the collected human data. Another possibility would
be if a large social media company were to engage their customers to provide picture-quality scores, similar to the Netflix
DVD ratings by e-mail of a decade ago. Generally, understanding how to successfully create reliable, very large-scale, and
authentic picture-quality databases remains an open question.
140

Authors
Jongyoo Kim (jongky@yonsei.ac.kr) received his B.S. and M.S.
degrees in electrical and electronic engineering from Yonsei
University, Seoul, South Korea, in 2011 and 2013, respectively. He
is currently working toward his Ph.D. degree in the Department of
Electrical and Electronic Engineering, Yonsei University, South
Korea. His research interests include two-dimensional (2-D)/threedimensional (3-D) image and video processing based on the
human visual system, quality assessment of 2-D/3-D image and
video, 3-D computer vision, and deep learning. He was a recipient
of the Global Ph.D. Fellowship by the National Research
Foundation of Korea from 2011 to 2016.
Hui Zeng (cshzeng@comp.polyu.edu.hk) received his M.S.
degree from the School of Information and Communication
Engineering, Dalian University of Technology, China, in 2016.
He is currently pursing his Ph.D. degree in the Department of
Computing, The Hong Kong Polytechnic University, under the
supervision of Prof. Lei Zhang. His research interests include
computer vision, image and video processing, and deep learning.
Deepti Ghadiyaram (deepti@cs.utexas.edu) received her Ph.D.
degree from the Department of Computer Science at the University
of Texas (UT) at Austin. Her research interests include image and
video processing, computer vision, and machine learning. Her Ph.D.
work focused on perceptual image and video quality assessment,
particularly on building quality-prediction models for pictures and
videos captured in the wild and understanding a viewer's time-varying quality of experience while streaming videos. She was a recipient of the UT Austin's Microelectronics and Computer
Development Fellowship from 2013 to 2014 and the Graduate
Student Fellowship from the Department of Computer Science from
2013 to 2016. She joined Facebook Research in September 2017.
Sanghoon Lee (slee@yonsei.ac.kr) received the B.S. degree
from Yonsei University, Seoul, South Korea, in 1989, the M.S.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017