IEEE Signal Processing Magazine - January 2018 - 114

materials, etc. Another commonly used methodology (e.g., in
human action recognition (Liu et al. [6]), and in attribute and
object-based modeling (Wang et al. [11]) is to take the attribute labels as latent variables on the training data set, e.g., in
the form of a structured latent support vector machine (SVM)
model where the objective is to minimize prediction loss. The
attribute description of an instance or a category is useful as a
semantically meaningful intermediate representation bridging
a gap between low-level features and high-level class concepts
(Palatucci et al. [2]).
The attribute-learning approaches have emerged as a promising paradigm for bridging the semantic gap and addressing data sparsity through transferring attribute knowledge
in image and video understanding tasks. A key advantage of
attribute learning is that it provides an intuitive mechanism
for multitask learning (Hwang et al. [12]) and transfer learning (Hwang et al. [12]). Particularly, attribute learning enables
learning with few or zero instances of each class via attribute
sharing, i.e., zero-shot and one-shot learning. The challenge
of zero-shot recognition is to recognize unseen visual object
categories without any training exemplars of the unseen class.
This requires the knowledge transfer of semantic information
from auxiliary (seen) classes with example images, to unseen
target classes.
Later works (Parikh et al. [13]) extended the unary/binary
attributes to compound attributes, which makes them extremely useful for information retrieval (e.g., by allowing complex
queries such as "Asian women with short hair, big eyes, and
high cheekbones") and identification (e.g., finding an actor
whose name you forgot, or an image that you have misplaced
in a large collection).
In a broader sense, the attribute can be taken as one special type of subjective visual property [14], which indicates the
task of estimating continuous values representing visual properties observed in an image/video. These properties are also
examples of attributes, including image/video interestingness
[15], and human-face age estimation [16]. Image interestingness was studied in Gygli et al. [15], which showed that three
cues contribute the most to interestingness: aesthetics, unusualness/novelty, and general preferences; the last of which refers
to the fact that people, in general, find certain types of scenes
more interesting than others, e.g., outdoor-natural versus indoor-manmade. Jiang et al. [17] evaluated different features for
Table 1. Different types of semantic representations for zero-shot
recognition.

114

Different Types of Attributes

Papers

User-defined attributes

[4], [5], [11], [18]-[21], [23], [24],
[30]

Relative attributes

[13], [14], [25]-[29]

Data-driven attributes

[5]-[7], [10], [31]-[33]

Video attributes

[34]-[38]

Concept ontology

[39]-[42]

Semantic word embedding

[8], [43]-[48]

video interestingness prediction from crowdsourced pairwise
comparisons. The ACM International Conference on Multimedia Retrieval 2017 published a special issue ("Multimodal
Understanding of Subjective Properties") on the applications
of multimedia analysis for subjective property understanding,
detection and retrieval (see http://www.icmr2017.ro/call-forspecial-sessions-s1.php). These subjective visual properties
can be used as an intermediate representation for zero-shot
recognition as well as other visual recognition tasks, e.g., people can be recognized by the description of how pale their skin
complexion is and/or how chubby their face looks [13]. Next,
we will briefly review different types of attributes.

User-defined attributes
User-defined attributes are defined by human experts [4] or
by concept ontology [5]. Different tasks may also necessitate
and contain distinctive attributes, such as facial and clothes
attributes [11], [18]-[20], attributes of biological traits (e.g., age
and gender) [21], product attributes (e.g., size, color, price), and
three-dimensional shape attributes [22]. Such attributes transcend the specific learning tasks and are, typically, prelearned
independently across different categories, thus allowing transference of knowledge [23]. Essentially, these attributes can
either serve as the intermediate representations for knowledge
transfer in zero-shot, one-shot, and multitask learning, or be
directly employed for advanced applications, such as clothes
recommendations [11].
Ferrari et al. [24] studied some elementary properties such
as color and/or geometric pattern. From human annotations,
they proposed a generative model for learning simple color and
texture attributes. The attribute can be viewed as either unary
(e.g., red color, round texture), or binary (e.g., black/white
stripes). The unary attributes are simple attributes, whose
characteristic properties are captured by individual image segments (appearance for red, shape for round). In contrast, the
binary attributes are more complex attributes, whose basic element is a pair of segments (e.g., black/white stripes).

Relative attributes
The aforementioned attributes use a single value to represent
the strength of an attribute being possessed by one instance/
class; they can indicate properties (e.g., spotted) or annotations
of images or objects. In contrast, relative information, in the
form of relative attributes, can be used as a more informative
way to express richer semantic meaning and thus better represent visual information. The relative attributes can be directly
used for zero-shot recognition [13].
Relative attributes (Parikh et al. [13]) were first proposed
to learn a ranking function capable of predicting the relative
semantic strength of a given attribute. The annotators give
pairwise comparisons on images, and a ranking function is
then learned to estimate relative attribute values for unseen
images as ranking scores. These relative attributes are learned
as a form of richer representation, corresponding to the strength
of visual properties, and used in a number of tasks including
visual recognition with sparse data, interactive image search

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|
http://www.icmr2017.ro/call-for-special-sessions-s1.php http://www.icmr2017.ro/call-for-special-sessions-s1.php
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
