IEEE Signal Processing - July 2018 - 34

objects or people and a slow-changing background scene; see
the example in Figure 2(a). Assume that all images are arranged
as one-dimensional vectors. Then, the tth image forms the tth
column, m t, of the data matrix, M. If the image size is denoted
by n and the total number of images in the video by d, then
M is an n # d matrix. Assuming that the background scene
changes slowly over time, the tth background image forms the
tth column, , t, of the low-rank matrix L. Let rL denote its
rank. If the background never changes, then L will be rank-1.
The low-rank assumption implies that the background changes
depend on a much smaller number, rL, of factors than either
the number of images d or the image size n. Let Tt denote the
support (set of indices of the nonzero pixels) of the foreground
frame t. To get the M = L + S formulation, we define st as a
sparse vector with support Tt and with nonzero entries equal
to the difference between foreground and background intensities on this support.
Being able to correctly solve the video-layering problem (separate a video into foreground and background layers) enables
better solutions to many other video analytics and computer
vision applications. For example, the foreground layer directly provides a video surveillance or object-tracking solution,
while the background layer and its subspace estimate are useful
in video editing or animation applications. Also, an easy lowbandwidth video conferencing solution would be to transmit
only the layer of interest (usually the foreground). As another
example, automatic video retrieval, used to look for videos of
moving waters or other natural scenes, will become significant-

ly easier if the retrieval algorithm is applied to only the background layer, whereas if the goal is to find a certain videos of
a certain dog or cat breed, the algorithm can be applied to only
the foreground layer. Finally, denoising and enhancement of
very noisy videos becomes easier if the denoiser is applied to
each layer separately (or to only the layer of interest). We show
an example of this in Figure 3.

Dynamic and functional magnetic resonance imaging
The S+LR model is a good one for many dynamic MRI
sequences. The changing region of interest (ROI) forms the
sparse outlier for this problem while everything else that is
slowly changing forms the low-rank component [22]. We show
a cardiac sequence in Figure 2(b). The beating heart valves
form the ROI in this case. This model can be used both to accurately recover a dynamic MRI sequence from undersampled
data (solve the "compressive" MRI problem) and to correctly
separate out the ROI. Similarly, in functional MRI-based brain
activity imaging, only a sparse brain region is activated in
response to given stimulus. This is the changing ROI (sparse
outlier) for this problem. There is always some background
brain activity in all the brain voxels. This is well modeled as
being slowly changing and influenced by only a small number
of factors, rL .

Detecting anomalies in computer and social networks
Another application is in detecting anomalous connectivity patterns in social networks or in computer networks [23], [24].

Video Analytics

Original

Background

Dynamic MRI

Foreground

Original

Background

Sparse ROI

(b)

(a)

Figure 2. (a) A video analytics application. Video layering (foreground-background separation) in videos can be posed as an RPCA problem. This is often the first
step to simplify many computer vision and video analytics tasks (for one such example, see Figure 3). We show three frames of a video in the first column. The
background images for these frames are shown in the second column. Notice that they all look very similar and hence are well modeled as forming a low-rank
matrix. The foreground support is shown in the third column. This clearly indicates that the foreground is sparse and changes faster than the background. (Result
taken from [12]; code available at http://www.ece.iastate.edu/~hanguo/PracReProCS.html.) (b) Low-rank and sparse matrix decomposition for accelerated dynamic
MRI [22]. The first column shows three frames of cardiac cine data. The second column shows the slow-changing background part of this sequence, while the
third column shows the fast-changing sparse ROI. This is also called the dynamic component. These are the reconstructed columns obtained from eightfold
undersampled data. They were reconstructed using undersampled stable PC pursuit [22].

IEEE Signal Processing Magazine

July 2018

http://www.ece.iastate.edu/~hanguo/PracReProCS.html

Table of Contents for the Digital Edition of IEEE Signal Processing - July 2018