Signal Processing - November 2017 - 123

in the source and similar hash codes for images belonging to
the same category; and 3) unsupervised entropy loss that guides
the unlabeled target data to align its hash codes according to the
source hash codes. The DAH network solves two important problems: classification with weak supervision or insufficient labels
(through domain adaptation) and determining hash codes in an
unsupervised setting (hash codes for target data).
An extension to DAN is achieved with the residual transfer
network (RTN) in [75], which implements a residual layer as the
final layer of the network in addition to a softmax loss. In the
RTN, feature adaptation is achieved with MMD loss, and the
source and target classifier adaptation is implemented through
the residual layer [76]. The source classifier fS (x) is tightly
coupled with the target classifier fT (x), varying with only a
slight perturbation Df ( fT (x)), which is learned by the network,
with fS (x) = fT (x) + Df ( fT (x)). In addition, the source classifier is constrained by the softmax loss over the source data, and
the target classifier is constrained with unlabeled entropy loss
over the target data.
Compacting deep neural networks and reducing the number of parameters are essential for creating smaller, more manageable networks. These procedures usually replace the larger
convolutional layer kernels with kernels of size 1 × 1 and 3 × 3.
Although such procedures produce networks that maintain the
classification accuracies, Wu et al. [77] note that the adaptability
of these networks is adversely affected, resulting in low accuracies for domain adaptation. Wu et al. propose a set of layers called
Conv-M, which consist of multiscale convolution and deconvolution with kernels larger than 3 × 3. The proposed compact network also uses MMD to align the source and target features at
multiple layers and produces state-of-the-art results on the standard Office and Office-Caltech data sets. The network is also
guided with a reconstruction loss that reconstructs images using
the encoded feature representations. The domain reconstruction
and classification network developed by Ghifary et al. [78] is
also guided by a reconstruction loss that decodes the feature
encoding along with a standard classification loss.
While the MMD is a standard nonparametric measure used
to align the features of the domains, Koniusz et al. [4] propose
a technique to align the higher-order statistics of the features.
The scatter statistics of samples belonging to a class (withinclass) are aligned across the two domains. These include
the means, scale/shear, and orientation measures of samples
belonging to a single class. The procedure also maintains good
separation for between-class scatters to enhance classification
accuracies. Unlike the popular unsupervised setting, this deeplearning technique is trained using a few labeled data from the
target domain.
In all of these deep domain adaptation approaches, the
weights are shared between the source and the target network
to ensure domain invariant features. The authors in [79] argue
that merely ensuring domain invariant features may be detrimental to the discriminative power of the features. Their model
is a twin network (one for the source and another for the target)
with a loss function over the weights for every source target
layer pair. The loss term ensures the weights of the source and

the target are closely related (but not identical). The source
network is trained with a softmax loss over the source data,
and both the networks also minimize the MMD loss to extract
domain invariant features.

Insights
Adopted deep methods are well suited for medium-sized data
sets (thousands of images like Office). These data sets are large
enough to fine-tune a deep network but not too large to fully
train a deep network from scratch. A pretrained deep network
like Alexnet [43] is often used as a base network and fine-tuned
for domain adaptation [63], [65]. One technique to adopt a shallow method is determining a closed-form solution that can then
be modeled as an objective function for a deep network [64].

Adversarial learning methods
In recent years, one of the most significant advances to deep
learning has been the introduction of generative adversarial
networks (GANs) by Goodfellow et al. [80]. GANs are networks that generate data (text, images, audio, etc.) such that
the data follow a predetermined distribution P (X ). A vanilla
GAN implementation has two deep networks, generator g ($)
and discriminator f ($), competing against each other. The generator network takes in a noise vector z ! R d sampled from a
uniform or normal distribution and generates an output g (z).
The discriminator takes in x ! P (X ) and g (z) and tries to
discriminate between the two. The generator network tries to
fool the discriminator network by generating data that appear
to belong to P (X ) , and the discriminator tries to distinguish
between real images and fake images. The equilibrium is a
saddle point in the network parameter space. The core concept
of the GAN is applied to achieve domain adaptation. Whereas,
in a standard GAN, a noise vector z ! R d is converted into
a fake image, in a domain adaptive setting, a source image
is converted into a fake target image. The pixel-GAN in [33]
is a straightforward extension of the GAN for unsupervised
domain adaptation. In this model, along with a noise vector
input z, the generator inputs the source image and is trained to
convert it into a target image. The discriminator, on the other
hand, is trained to distinguish between real target images and
generated target images (fake target images generated from
the source). In addition, a separate network is trained to classify the generated target images. Along similar lines, Taigman
et al. [81] develop an image translation network that converts
an image from one domain into an image in another domain
using adversarial networks. There have been many recent works
applying adversarial training for domain adaptation. A few of
the most recent procedures based on the core GAN idea but
with subtle variations are [82]-[85].
In the domain adversarial neural network (DaNN) in [86],
the authors train a deep neural network in a domain-adversarial manner for image classification-based domain adaptation.
The bottom layers of the network act as feature extractors. The
features from the bottom layers are fed into two branches of the
network. The first branch is a softmax classifier trained with the
labeled source data. The second branch is a domain classifier

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

123

Table of Contents for the Digital Edition of Signal Processing - November 2017