IEEE Signal Processing Magazine - January 2018 - 118

and it can identify visual objects using both labeled image data
as well as semantic information gleaned from unannotated text.
ConSE [43] constructed the image embedding approach by
mapping images into the semantic embedding space via convex
combination of the class label embedding vectors. Both DeViSE
and ConSE are evaluated on large-scale data sets-ImageNet
[the ImageNet Large-Scale Visual Recognition Challenge
(ILSVRC)] 2012 1K and ImageNet 2011 21K.
To combine the visual and textual branches in the deep
embedding, different loss functions can be considered, including margin-based losses [46] or Euclidean distance loss [53].
Zhang et al. [76] employed the visual space as the embedding
space and proposed an end-to-end deep-learning architecture
for zero-shot recognition. Their networks have two branches: a
visual encoding branch, which uses CNNs to encode the input
image as a feature vector, and the semantic embedding branch,
which encodes the input semantic representation vector of each
class to which the corresponding image belongs.

recognition can be addressed by label propagation from unseen
prototype instances to unseen testing instances. Changpinyo
et al. [78] synthesized classifiers in the embedding space for
zero-shot recognition. For multilabel zero-shot learning,
the recognition models have to consider the co-occurrence/
correlations of different semantic labels [41], [45], [79].
Latent SVM structures have also been used as the recognition models [12], [80]. Wang et al. [80] treated the object
attributes as latent variables and learned the correlations of
attributes through an undirected graphical model. Hwang
et al. [12] utilized a kernelized multitask feature-learning
framework to learn the sharing features between objects and
their attributes. Additionally, Long et al. [81] employed the
attributes to synthesize unseen visual features at the training
stage and, thus, zero-shot recognition can be solved by the
conventional supervised classification models.

Problems in zero-shot recognition
There are two intrinsic problems in zero-shot recognition-
projection domain shift and hubness.

Recognition models in the embedding space
Once the embedding model is learned, the testing instances can
be projected into this embedding space. The recognition can be
carried out by using different recognition models. The most
commonly used one is the nearest neighbor classifier, which
classifies the testing instances by assigning the class label in
terms of the nearest distances of the class prototypes against
the projections of testing instances in the embedding space. Fu
et al. [7] proposed a semilatent zero-shot learning algorithm to
update the class prototypes by one-step self-training.
Manifold information can be used in the recognition models
in the embedding space. Fu et al. [77] proposed a hypergraph
structure in their multiview embedding space; and zero-shot

Projection domain shift problems
The projection domain shift problem in zero-shot recognition was first identified by Fu et al. [77]. This problem can be
explained as follows: since the source and target data sets have different classes, the underlying data distribution of these classes may
also differ. The projection functions learned on the source data set,
from visual space to the embedding space, without any adaptation
to the target data set, will cause an unknown shift/bias. Figure 1
from [77] gives a more intuitive illustration of this problem. It plots
the 85-dimensional (85-D) attribute space representation spanned
by feature projections that are learned from source data, and class

Zebra

Prototype

The Same "hasTail" Attribute
Different Visual Appearance
Pig
Prototype

Pig
Prototype
(a)

(c)

(b)

FIGURE 1. Illustrating the projection domain shift problem. (a) The visual space, (b) attribute space, and (c) multiview embedding space. Zero-shot prototypes are
annotated as red stars and predicted semantic attribute projections are shown in blue. Both pig and zebra share the same "hasTail" attribute yet with a very different
visual appearance of a tail. (Figure used with permission from [77].)

118

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
