IEEE Signal Processing Magazine - January 2018 - 62

This bird is completely black.
Beak
Belly
Right Leg
This bird is bright blue.
Head

A Man in an Orange Jacket, Black Pants, and a Black Cap Wearing Sunglasses Skiing

FIGURE 7. Examples of image synthesis using the GAWWN. In the GAWWN, images are conditioned
on both text descriptions and image location specified as either a keypoint or bounding box. (Figure
reproduced from [44] with permission.)

using a spatiotemporal GAN architecture [40]. In some cases,
models trained on synthetic data do not generalize well when
applied to real data [3]. Bousmalis et al. [3] propose to address
this problem by adapting synthetic samples from a source
domain to match a target domain using adversarial training.
Additionally, Liu et al. [41] propose using multiple GANs-
one per domain-with tied weights to synthesize pairs of corresponding images samples from different domains. Because
the quality of generated samples is hard to quantitatively
judge across models, classification tasks are likely to remain
an important quantitative tool for performance assessment
of GANs, even as new and diverse applications in computer
vision are explored.

Image synthesis
Much of the recent GAN research focuses on improving the
quality and utility of the image-generation capabilities. The
LAPGAN model introduced a cascade of convolutional networks within a Laplacian pyramid framework to generate
images in a coarse-to-fine fashion [13]. A similar approach is
used by Huang et al. [42] with GANs operating on intermediate
representations rather than lower-resolution images.
LAPGAN also extended the conditional version of the GAN
model where both G and D networks receive additional label
information as input; this technique has proved useful and is
now a common practice to improve image quality. This idea
of GAN conditioning was later extended to incorporate natural
language. For example, Reed et al. [43] used a GAN architecture to synthesize images from text descriptions, which one
62

might describe as reverse captioning.
For example, given a text caption of
a bird such as "white with some black
on its head and wings and a long,
orange beak," the trained GAN can
generate several plausible images that
match the description.
In addition to conditioning on text
descriptions, the generative adversarial
what-where network (GAWWN) conditions on image location [44]. The
GAWWN system supported an interactive interface in which large images
could be built up incrementally with
textual descriptions of parts and usersupplied bounding boxes (Figure 7).
Conditional GANs not only allow us
to synthesize novel samples with specific attributes, they also allow us to develop tools for intuitively editing images;
e.g., changing the hairstyle of a person
in an image, making them wear glasses, or editing the image so they appear
younger [35]. Additional applications of
GANs to image editing include work by
Zhu and Brock et al. [2], [45].

Image-to-image translation
Conditional adversarial networks are well suited for translating an input image into an output image, which is a recurring
theme in computer graphics, image processing, and computer
vision. The pix2pix model offers a general-purpose solution to this family of problems [46]. In addition to learning
the mapping from input image to output image, the pix2pix
model also constructs a loss function to train this mapping.
This model has demonstrated effective results for different
problems of computer vision that had previously required separate machinery, including semantic segmentation, generating
maps from aerial photos, and colorization of black and white
images. Wang et al. present a similar idea, using GANs to first
synthesize surface-normal maps (similar to depth maps) and
then map these images to natural scenes.
CycleGAN [4] extends this work by introducing a cycle
consistency loss that attempts to preserve the original image
after a cycle of translation and reverse translation. In this formulation, matching pairs of images are no longer needed for
training. This makes data preparation much simpler, and opens
the technique to a larger family of applications. For example,
artistic style transfer [47] renders natural images in the style of
artists, such as Picasso or Monet, by simply being trained on an
unpaired collection of paintings and natural images (Figure 8).

Superresolution
Superresolution allows a high-resolution image to be generated
from a lower-resolution image, with the trained model inferring photo-realistic details while upsampling. The SRGAN

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018