IEEE Signal Processing - July 2018 - 83

D1

D1D2

(a)

(b)

D1D2D3

(c)

Figure 6. An ML-CSC model trained on the MNIST data set. (a) The local filters of the dictionary D 1 . (b) The local filters of the effective dictionary
D (2) = D 1 D 2 . (c) Some of the 1,024 local atoms of the effective dictionary D (3), which are global atoms of size 28 # 28.

Definition
The set of K-layered ML-CSC signals of cardinalities {k 1,
k 2, f, k K } over the convolutional dictionaries {D 1, D 2, f, D K }
is defined as MS "" D i ,Ki = 1, " k i ,Ki = 1 ,. A signal X belongs to this
set if it can be described as
X = D 1 C 1 s.t.
C 1 = D 2 C 2 s.t.
C 2 = D 3 C 3 s.t.
h
C K - 1 = D K C K s.t.

s
0, 3
s
C 2 0, 3
s
C 3 0, 3

C1

CK

s
0, 3

# k1
# k2
# k3
# kK .

(17)

Observe that, given the aforementioned relations, one obviously
gets X = D 1 D 2 gD K C K . Thus, our overall model is a special
case of Sparseland, with an effective dictionary being the multiplication of all the CSC matrices {D i} Ki = 1. It can be shown that
this effective dictionary has an overall convolutional form, as
shown in [75], and thus we could even go as far as saying that
ML-CSC is a special case of the CSC model.
Indeed, ML-CSC injects more structure into the CSC model by
requiring all the intermediate representations, C 1, C 2, f, C K - 1
to be locally sparse as well. The implication of this assumption is
the belief that X could be composed in various ways, all leading
to the same signal. In its most elementary form, X can be built
from a sparse set of atoms from D 1 by D 1 C 1. However, the very
same vector X could be built using yet another sparse set of elements from the effective dictionary D 1 D 2 by D 1 D 2 C 2 . Which
are the atoms in this effective dictionary? Every column in D 1 D 2
is built as a local linear combination of atoms from D 1, whose
coefficients are the atoms in D 2 . Moreover, due to the locality of
the atoms in D 2, such combinations are only of atoms in D 1 that
are spatially close. Thus, we could refer to the atoms of D 1 D 2

as molecules. The same signal can be described this way using
D 1 D 2 D 3, and so on, all the way to D 1 D 2 gD K C K .
Perhaps the following explanation could help in giving intuition to this wealth of descriptions of X. A human-being body can
be described as a sparse combination of atoms, but he/she could
also be described as a sparse combination of molecules, a sparse
composition of cells, tissues, and even body parts. There are many
ways to describe the formation of the human body, all leading to
the same final construction, and each adopting a different resolution of fundamental elements. The same spirit exists in the MLCSC model.
To better illustrate the ML-CSC and how it operates, Figure 6
presents a three-layered CSC model trained on the MNIST database of handwritten digits. The results presented here refer to a
dictionary-learning task for the ML-CSC, as described in [75].
We will not explain the details of the algorithm to obtain these
dictionaries, but rather concentrate on the results obtained. This
figure shows the m 1 = 32 filters that construct the dictionary
D 1, and these look like crude atoms identifying edges or blobs.
It also shows the m 2 = 128 filters corresponding to the effective
dictionary D 1 D 2, and its fundamental elements are elongated
and curved edges. Note that there is no point in presenting D 2
alone as it has almost no intuitive meaning. When multiplied by
D 1, it results with filters that are referred to as molecules. In the
same spirit, the m 3 = 1, 024 filters of D 1 D 2 D 3 contain large
portions of digits as its atoms. Any given digit image could be
described as a sparse combination over D 1, D 1 D 2, or D 1 D 2 D 3,
in each case using different fundamental elements set to form
the construction.

Pursuit algorithms for ML-CSC signals: First steps
Pursuit algorithms were mentioned throughout this article, making their first appearance in the context of Sparseland, and later

IEEE Signal Processing Magazine

|

July 2018

|

83
Table of Contents for the Digital Edition of IEEE Signal Processing - July 2018
