IEEE Signal Processing Magazine - January 2018 - 25

Convolution
Layer

Input Image

n Feature Maps

Output Image

FIGURE 3. A three-layer CNN with successive convolutional layers, where the spatial dimensions of the feature maps match those of the input and output
images. Following each convolution there is a nonlinearity operation, not shown here.

SR. In this case, to predict an HR patch at time t, multiple
motion-compensated frames from previous and future time
instances are fed through separate CNNs. The individual feature maps produced by each CNNs are later fused by another
CNN (different depths for implementing the fusion process
were investigated) to produce the final prediction of the HR
frame. In the CS literature, Kulkarni et al. [18] use a CNN
with six convolutional layers to obtain an intermediate reconstruction of the input CS measurements, which is then further
refined with a nondeep-learning-based denoising method.
Hradis et al. [19] surpassed state-of-the-art analytical methods
for their blind deconvolution task on text images, which they
solved by training a deep CNN with up to 15 layers.

Achieving greater depth: CNNs with
residual blocks and skip connections
In their work on blind deconvolution using CNNs, Hradis et al.
[19] show that training deeper networks produced results of
significantly better quality compared with the results obtained
from shallow networks. One might expect that constructing
models of greater depth provides the model with more representation power. In addition, increasing the depth of the net-

Input Image

work increases the overall receptive field of the model, which
provides more contextual information at each layer of the network and improves the performance of the image recovery
task. Until a couple of years ago, it was particularly challenging to train DNNs with more than a few layers, mostly due
to image databases not being large enough, unstable training,
and limited computational power. Many works had to resort
to using greedy layer-wise pretraining (a greedy approach is
used, e.g., in [14] for training a five-layer CNN for denoising).
Recently, however, access to very large data sets and powerful
computational systems, along with the introduction of effective activation functions (e.g., the rectified linear unit [20]),
parameter initialization strategies (e.g., [21]) and more efficient
architectural design choices (e.g., batch normalization [22]),
have provided new possibilities for training deeper networks.
The use of residual blocks [23], has also played a significant
role in training very deep models. Instead of learning a new
mapping function from one layer to the next, residual blocks
learn a residual between two or more layers by adding a skip
connection from the input of the residual block to its output.
Figure 4 illustrates this concept by showing a generic architecture of a deep residual CNN. Because learning residuals

Residual
Block

Output Image

Residual Block

FIGURE 4. An example of a deep residual CNN. Each residual block, consisting here of three convolutions, learns a residual between its input and
its output.
IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018