IEEE Signal Processing Magazine - January 2018 - 71

from the keypoints can help the network determine the structure of the face. Zhang et al. [32], Chen et al. [22], Li et al. [21],
and HyperFace [10] train the face detection network along with
the keypoint estimation task, and All-in-One Face [2] extends
the MTL approach to additional tasks such as face verification,
gender, smile, and age estimation. Figure 3(b) provides a comparative performance evaluation of different algorithms for the
task of facial keypoint estimation on the AFW [55] data set.

Face identification and verification
In this section, we review the relevant works on face identification and verification. In Figure 5, we illustrate a general
pipeline for training and testing a face identification/verification system using DCNNs. There are two major components
for face identification/verification: 1) robust face representation and 2) a discriminative classification model (face identification) or similarity measure (face verification). Since we
focus on deep-learning-based approaches, we refer the interested readers to [56], which summarizes traditional approaches
including local binary pattern (LBP) and the Fisher vector, as
well as metric learning such as the one-shot similarity (OSS)
kernel, Mahalanobis metric learning, cosine metric learning,
large-margin nearest neighbor, attribute-based classifier, and
joint Bayesian (JB).

Robust feature learning for faces using deep learning
Learning invariant and discriminative feature representations
is a critical step in a face recognition system. Deep-learning
methods have shown that compact and discriminative representations can be learned using a DCNN trained with very
large data sets. We first review some important face recognition works using deep learning followed by a brief summary of
recent methods for feature representation learning.

Instead of relying on handcrafted features such as the popular face descriptor LBP, Huang et al. [57] proposed to learn the
face representation using convolutional deep belief networks
based on local convolutional restricted Boltzmann machines.
Their method first learns useful representation in an unsupervised manner, from an unlabeled natural image data set containing images of natural scenes, and then transfers the learned
representation to a face identification/verification task through
classification models [e.g., support vector machine (SVM)]
and metric-learning approaches (e.g., OSS). The approach
achieved satisfactory results on the LFW data set without using
large-scale annotated face data sets.
One of the earlier large-scale applications of 3-D-modelbased alignment along with DCNNs for face recognition was
proposed by Taigman et al. in their DeepFace [58] approach.
They derive a face representation using a nine-layer deep neural network, which involves more than 120 million parameters
and uses several locally connected layers without weight sharing,
rather than the standard convolutional layers. A proprietary face
data set that consists of 4 million facial images belonging to more
than 4,000 identities was used to train their system.
Since collecting a large-scale annotated data set is costly,
Sun et al. proposed DeepID frameworks [59]-[61] with JB for
face verification, which utilized an ensemble of shallower and
smaller deep convolutional networks (i.e., each DCNN consists of four convolutional layers and uses 39 # 31 # 1 patches
as the input) than DeepFace and were trained using 202,599
images of 10,177 subjects. The large number of distinct identities of the data set and an ensemble of DCNNs for different
local and global face patches make DeepID learn discriminative and informative face representation. The approach was
the first to achieve results that surpass human performance for
face verification on the LFW data set.

Training
Pos

Neg
Metric Learning

DCNN Model

Testing

W, b

Compute
Similarity

DCNN Model

FIGURE 5. A general pipeline for training and testing a face verification system using DCNNs.
IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018