Signal Processing - September 2016 - 90

Figure 6. We start with pretrained space-time filters for each specific action. We chose to train maximum average correlation height
(MACH) filters for each action class [29]. The JL lemma then provides a way to evaluate the filter response for a given video directly in the compressed domain, thereby avoiding reconstructing the
frames of the test video. To reduce computational complexity, the
three-dimensional (3-D) response volume is calculated in the frequency domain via 3-D fast Fourier transform. For a given test
video, we obtain N A correlation volumes. For each correlation
volume, we adapt three level volumetric max pooling to obtain a
73-dimensional feature vector [30]. In addition, we also compute
peak-to-sidelobe ratio for each of these 73 max-pooled values.
This framework can be used in any reconstruction-free application from compressive cameras that can be implemented using
3-D correlation filtering. The action localization in each frame is
determined by a bounding box centred at location ^l max h in that
frame, where l max is determined by the peak response (i.e., the
response corresponding to the classified action) in that frame
and the size of the filter corresponding to the classified action. To
determine the size of the bounding box for a particular frame, the
response values inside a large rectangle of the size of the filter and
centered at l max in that frame are normalized so that they sum up
to unity. Treating this normalized rectangle as a 2-D probability
density function, we determine the bounding box to be the largest
rectangle centered at l max, whose sum is less than a value m. For
our experiments, we use m equal to 0.7.

Experimental results
We present sample results obtained on Weizmann [7] and the
University of Central Florida (UCF) sports [29] data sets. More
extensive results can be found in [22]. For all of our experiments, we use a measurement matrix z , whose entries are
drawn from i.i.d. standard Gaussian distribution, to compress

the frames of the test videos. We note that it is possible to use
more esoteric measurement matrices to improve either reconstruction and/or recognition performance. For example, variants of wavelet bases are better suited for reconstruction and
task-driven measurement operators are better suited for inference. In this section, we use the random Gaussian matrix to
level the playing field for reconstruction and inference.

Results on Weizmann data set
The Weizmann data set contains ten different actions, each performed by nine subjects, thus making a total of 90 videos. For
evaluation, we used the leave-one-out approach, where the filters
were trained using actions performed by eight actors and tested
on the remaining one. The results shown in Figure 7 indicate that
our method clearly outperforms the reconstruct-then-recognize
using the improved dense trajectories (IDT) method, a state-ofthe-art recognition algorithm. At compression ratios of 100 and
above, recognition rates are very stable for the compressive recognition framework, while reconstruct-then-recognize fails completely. The recognition rates are stable even at high compression
ratios and are comparable to the recognition accuracy for the
Oracle MACH (OM) method [1]. The average time taken by
both methods to process a video of 144 # 180 # 50 size are
shown in parentheses in Figure 7. Recon+IDT takes about 20-35
minutes to process one video, with the frame-wise reconstruction of the video being the dominating component. In contrast,
compressive inference takes only a few seconds. The sample
spatial localization results are shown in Figure 7(a) in a few
frames for various actions of the data set.

Results on UCF sports data set
The UCF sports action data set [29] contains a total of 150
videos across nine different actions. It is a challenging data

Compressively Sensed Actions/Scene

Scene
and Action
3-D MACH Filter

Random Lens
Pattern

"Single
Pixel"

Compressive
Measurements
Z (t )

Smashed Correlation
in Space+Time

Feature Vector Formation Vai
Nonlinear Operations Max-Pooling,
Peak-Sidelobe Ratio, etc.)

Bank of Space-Time Action Filters for Different Viewpoints

Action Classification Via Statistical
Methods (e.g., SVMs)

Figure 6. Compressive inference via smashed filters. (a) Every frame of the scene is compressively sensed by optically correlating random patterns with
the frame to obtain CS measurements. (b) An overview of our approach to action recognition from a compressively sensed test video. First, MACH [29]
filters for different actions are synthesized offline from training examples and then compressed to obtain smashed filters. Next, the CS measurements of
the test video are correlated with these smashed filters to obtain correlation volumes that are analyzed to determine the action in the test video. (Figure
adapted from [22].)
90

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016