Signal Processing - September 2017 - 176

Conv

Conv + BNorm + ReLU

Conv + ReLU
Noisy Image

Residual Image

FIGURE 4. A residual learning-based image denoising framework. Conv: convolution, ReLU: rectified linear unit, and BNorm: batch normalization.

pairs. The general model of discrimina-
tive learning methods can be written as
min H loss ^ xt , xh,
s.t. xt = F ^ y, H; Hh,

(14)

where F ($) is the inference or map-
ping function with parameter set H and
loss ($) is the loss function to measure
the similarity between output image xt
and ground-truth image x. The recent-
ly developed deep learning-based IR
methods [3], [7]-[10] are typical dis-
criminative learning methods, where
F ($) is a deep CNN.

Deep CNNs for IR
One of the first CNN-based IR meth-
ods is the superresolution CNN
(SRCNN) method [3] for single image
superresolution (SISR). It is actually
modestly deep because it has only two
hidden layers. A truly deep CNN for
SISR was developed in [7]. The so-
called very deep superresolution
(VDSR) method first initializes the
low-resolution (LR) image to a high-
resolution (HR) image (e.g., by bicubic
interpolation) and then learns a CNN
to predict the residual between the ini-
tialized HR image and the ground-truth
image. VDSR shows highly competi-
tive peak signal-to-noise ratio (PSNR)
results, and it demonstrates that a single
CNN can perform SISR with multiple
scaling factors.
VDSR enlarges the LR input to
the same size of the HR image before
176

it goes through a CNN. This not only
restricts the area of receptive fields but
also increases the cost of convolution
operations in CNN. A more efficient
solution is to directly predict the miss-
ing HR pixels from the LR image, as
proposed in [9]. The so-called efficient
subpixel CNN (ESPCNN) learns to pre-
dict the feature maps of each subimage,
and aggregates the subimages into the
final HR image.
Most existing SISR methods use
the mean squared error (MSE) as the
loss function to optimize the network
parameters H. Minimizing MSE tends
to produce the mean image of possible
observations. As a result, the output HR
image can be oversmoothed without a
good perceptual quality. Inspired by the
great success of the recently developed
generative adversarial nets (GANs)
[5]-a perceptual loss function, which
consists of a content loss (i.e., MSE) and
a GAN-based adversarial loss-is pro-
posed in [8] to generate the HR images.
Though the PSNR index is not very
high, the so-called superresolution with
GAN method produces perceptually
very pleasant SISR results.
CNNs have also been successfully
used in image denoising. A residual
learning-based image denoising meth-
od, called DnCNN, is proposed in [10],
whose framework is shown in Fig-
ure 4. The network is learned to pre-
dict the noise (i.e., residual) corrupted
in the image, and the denoised image
can be obtained by subtracting the pre-
dicted noise from the noisy image. It
IEEE SIGNAL PROCESSING MAGAZINE

September 2017

is shown in [10] that batch normaliza-
tion is very useful for Gaussian noise
removal since it enforces the output in
each layer to be Gaussian-like distrib-
uted. The DnCNN method can also
handle more general noise, such as the
compression errors and the interpo-
lation errors, using a single network.
Figure 5 shows an example. The input
image is partly noise corrupted, partly
interpolated, and partly compressed.
One can see that the trained DnCNN
can handle the three tasks simultane-
ously with very good performance.

Learn a deep denoiser prior for
general IR tasks
Though CNN has achieved very com-
petitive performance compared with
the model-based optimization method
in IR, it has limitations, especially on
the capacity of generality. In most of
the CNN-based SISR methods, the
downsampling kernel is assumed to be
the bicubic kernel. When applying the
trained CNN to an LR image produced
by a different kernel, the result will
become much less satisfactory. Similar
things will happen for image deblurring
problems, where the blur kernels can be
very different but it is hardly possible
to train a CNN for each instantiation
of the blur kernel. Table 2 summarizes
the pros and cons of model-based opti-
mization methods and deep CNN-based
methods for IR.
Compared with the deep CNN-based
methods, model-based optimization
methods have better generality. Given

Table of Contents for the Digital Edition of Signal Processing - September 2017