Signal Processing - September 2016 - 89

SPC Data

Overlapping Groups
of W Measurements
Inverse
Hadamard
Transform

Inverse
Hadamard
Transform

Low-Resolution Estimate
of the Frames
(√W × √W Pixels)

Estimate Motion Flow
Between Low-Resolution
Frames
Color Coding of
Motion Flow

Final Reconstructed Video
Using Motion Flow
Constraints (Full Resolution)

Figure 5. An outline of the CS multiscale video (CS-MUVI) recovery framework. Given a total number of T measurements, we group them into over-

lapping windows of size W , resulting in a total of F frames. For each frame, we first compute a low-resolution initial estimate using a window of W
neighboring measurements. We then compute the optical flow between upsampled preview frames (the optical flow is color coded as in [24]). Finally, we
recover F high-resolution video frames by enforcing a sparse gradient prior along with the measurement constraints as well as the brightness constancy
constraints generated from the optical-flow estimates. (Figure adapted from [31].)

Reconstruction-free inference
In this section, we discuss high-level inference problems that
can be solved directly in the compressive domain without the
need for reconstruction of the imagery. The primary example
we consider is that of action recognition, which is one of the
long standing research areas in computer vision with widespread applications in video surveillance and human-computer
interaction. In many applications of action recognition, one is
faced with resource constraints such as limited power supply,
limited storage, and limited computational capabilities, all of
which make CI a great solution. To this end, we focus specifically on action recognition from videos and review the results
that demonstrate that it is indeed possible to perform action
recognition at extremely higher compression ratios without
reconstructing the video in the first place.
The central idea that allows for detection/recognition problems to be solved directly in the compressed domain is the
Johnson-Lindenstrauss (JL) lemma and its application to compressive convolutional filtering (also referred to as smashed
filtering). Specifically, the JL lemma states that the correlation

between any two signals is nearly preserved even when the
data is compressed to a much lower-dimensional space. This
property allows one to evaluate the response of a signal to a
given convolutional kernel directly in the compressed domain.
This is extremely useful in practice since correlational features
have traditionally been used extensively in computer-vision
problems such as automatic target recognition, face recognition [35], palm-print identification [18], and even activity recognition [29]. Davenport et al. [25] introduced the concept of
smashed filters that provides a way to extract such correlational features in the compressed domain.
We first show that approximate correlational features can
be extracted directly from CS measurements even in very
challenging activity videos. Using this in conjunction with the
widely used correlational filters approach to recognition tasks
in computer vision, one can develop a spatiotemporal smashed
filtering approach to action recognition that results in robust
performance at extremely high compression ratios.
The overall algorithmic pipeline for directly extracting
correlational features from compressed video is illustrated in

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016