IEEE Signal Processing Magazine - January 2018 - 79

Table 5. Data sets used for training All-in-One Face.
Data Set

Facial Analysis Task

Number of
Training Samples

CASIA [13]

Identification, gender

490,356

MORPH [108]

Age, gender

55,608

IMDB+WIKI [109]

Age, gender

224,840

Adience [97]

Age

19,370

CelebA [80]

Smile, gender

182,637

AFLW [110]

Detection, pose, fiducials

20,342

Total

■

993,153

■

are fused together into a single All-in-One Face. Table 5 lists
the different data sets used for training the All-in-One Face. Taskspecific loss functions are used to train the complete network
end to end. Some sample outputs for All-in-One Face are shown
in Figure 9.
MTL-based DCNNs have also been used for determining
multiple facial attributes. Dehghan et al. proposed Deep Age,
Gender, and Emotion Recognition (DAGER) [111] for recognizing age, gender, and emotion using DCNNs. Similar to Allin-One Face [2], it uses different training sources labeled for
different tasks for training the DCNN. He et al. [112] jointly
learn a deep architecture for face detection and facial attribute analysis. Unlike other MTL-based methods, they use full
image as an input to the network instead of a face region. A
faster R-CNN based approach is used to obtain the detections
along with the attributes for the faces. Table 6 summarizes the
facial analysis tasks performed by some of the recent MTLbased methods.

viewpoints, occlusions, etc. Other factors such as blur and
low resolution challenge the face detection task.
Fiducial detection: Most data sets contain only a few thousand images. A large-scale annotated and unconstrained data
set will make the face alignment system more robust to the
challenges, including extreme pose, low illumination, and
small, blurry face images. Researchers have hypothesized
that deeper layers of CNNs can encode more abstract information such as identity, pose, and attributes; however, it has
not yet been thoroughly studied which layers exactly correspond to local features for fiducial detection.
Face identification/verification: For face identification/verification, the performance can be improved by learning a
discriminative distance measure. However, due to memory
constraints limited by graphics cards, how to choose informative pairs or triplets and train the network end to end
using online methods (e.g., stochastic gradient descent) on
large-scale data sets is still an open problem. Another challenging problem to address is to incorporate full motion
video processing in deep networks for enabling video-based
face analytics.

Conclusions
In this article, we presented an overview of recent developments in designing an automatic face recognition system. We
also discussed various multitask CNN-based approaches for
face analytics. The examples shown in the article demonstrate
that subject-independent tasks benefit from MTL, and network
initialization from the face recognition task. They also show
how these tasks help to improve face recognition performance.
Given the sufficient amount of annotated data and GPUs,
DCNNs have been shown to yield impressive performance
improvements. Still many issues remain to be addressed to
make the DCNN-based systems robust and practical, such as
reducing reliance on large training data sets, handling data bias
and degradation in training data, incorporating domain knowledge, reducing the training time when the network goes deeper
and wider, and building the theoretical foundations to understand the characteristics and behaviors of DCNN models. We
refer the interested readers to [12] for more details about other
challenges for DCNN models.

Open issues
We briefly discussed several design considerations for each
component of an automated face identification/verification
system, including the following:
■ Face detection: In contrast to a generic object detection task,
face detection is more challenging due to the wide range of
variations in the appearance of faces. The variability is
caused mainly by changes in illumination, facial expression,

Table 6. A list of various MTL-based facial analysis algorithms along with the types of face tasks they can perform.
Method

Face Detection

Fiducials

Head Pose

Zhu et al. [55]

✓

JointCascade [105]

✓

Zhang et al. [32]

✓

TCDCN [107]
HyperFace [10]

✓

He et al. [112]

✓

DAGER [111]
All-in-One Face [2]

✓

Gender

✓

Age

Expression

Other Attributes

✓

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Face Recognition

✓

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018