IEEE Signal Processing Magazine - January 2018 - 108

EPLL P (x) = / log P (y i),

Table 2. PSNRs for different image inpainting methods
on Berkeley data set.
Method

PSNR

KSVD [5]

24.13

FoE [28]

24.79

GMM [40]

25.71

DRBN (CA)

29.42

DRBN (AugCA)

33.46

(a)

(b)

(c)

(d)

FIGURE 4. An example of the image inpainting experiment, (a) original image,
(b) corrupted image, (c) restored image using the GMM, and (d) restored
image using the DRBN. (Images used with permission from [19].)

Applications
To demonstrate the effectiveness of the DRBN models, we will
apply the DRBN to different computer vision tasks to demonstrate its capability for both data representation and feature
learning for classification.

Data representation
First, we quantitatively compare the DRBN with other generative models in terms of peak signal-to-noise ratio (PSNR) in
several image restoration and denoising tasks. Then, we evaluate the generative representation power of different models in
terms of generating synthetic data.
The first task is image restoration, which is to restore an
image from its corrupted version. It is shown that a higher likelihood of patches leads to better denoising on whole images
[40]. Therefore, we train a DRBN model on the 8 × 8 patches
from the Berkeley data set [19], which contains 200 training
and 100 testing images. One million training and 50,000 testing patches are randomly sampled from 200 training and 100
test images, respectively. A DRBN with two latent layers is
used, with each layer containing 50 latent variables respectively. The total training epochs are 100. With a learned DRBN as
a prior model, we use the expected patch log likelihood (EPLL)
[40] framework to perform image restoration. The EPLL of an
image x is defined as
108

(26)

i

where {y i} represents all the overlapping patches in the image.
+
Given a corrupted image x, the cost we use to reconstruct the
image with patch prior P is
fP (x | x+ ) = m | | x - x+ | | 2 - EPLL P (x) .
2

(27)

It is difficult to directly optimize the complicated cost
function. We employ the half-quadratic slitting method [7]
with m set to 10 6, following the settings in [40]. To perform
image inpainting, we superimposed some sentences on the
clean image as the noise. During optimization, both the CA
and AugCA methods were used to perform MAP inference for
each patch. Typically the CA and AugCA converge after three
iterations, and initialization affects the final configuration of
latent variables but not posterior likelihood. We compared
DRBN to three state-of-the-art approaches with generic priors:
field of experts (FoE) [28], KSVDG [5], and the GMM [40]
with full covariance matrix. The quantitative results on the 100
test images are given in Table 2. The DRBN outperforms all
other approaches in terms of PSNR values.
An example is given in Figure 4, where (a) represents an
original image, (b) a corrupted image, (c) a restored image
using the GMM as a prior model, and (d) a restored image
using the DRBN as a prior model. The PSNR values for
(c) and (d) are 26.31 and 29.80, respectively. It can be seen
that using the GMM prior, some parts of the letters in the
corrupted image remain in the restored image, especially
if the background color is white. The DRBN model completely removes the letters, and the PSNR values show significant improvement.
The second task for image representation is face restoration, where we use DRBN to restore a given cropped image.
For this task, we put two kinds of noise on the face images
from the Multi-PIE data set: random noise and block occlusion, where 40% of the pixels are corrupted by random noise
with a standard deviation of 0.4, following the same procedure
as [35]. For the latter, 12 × 12 blocks are superimposed on a
random part of the 32 × 32 faces with Gaussian random noise,
following the same procedure as [35]. The same MAP inference methods-CA and AugCA-are applied. Also, results
from the inference network (IN) [21] is compared with CA and
AugCA. The baseline methods include the robust Boltzmann
machine (RoBM) [35], GRBM [12], and robust PCA (RPCA)
[38]. PSNR is employed to quantitatively evaluate different
methods. The result is given in Table 3. The AugCA method
outperforms all other methods in terms of PSNR values. In
the cross-subject setting, the RBN generalizes well to unseen
subjects. Some examples of the reconstructed faces are given
in Figure 5.
Next we demonstrate the DRBN's image reconstruction ability. In the image restoration task presented in Figure 5, images
are restored using trained DRBN models from given corrupted
images. Unlike the image reconstruction task, given an image x,
we first perform MAP inference to obtain the most likely latent

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
