Signal Processing - January 2016 - 19

reference pointers to state-of-the-art research. Many topics are
inherently interrelated (see Figure 1):
■ Latent variable modeling and hierarchy are the central
themes of probabilistic or Bayesian modeling.
■ Hierarchy is a natural way to characterize random effects,
nonstationarity, and group variability.
■ Depending on the hyperprior/prior assumption or optimi-
zation criterion (e.g., posterior, marginal likelihood, and vari-
ational free energy), one can derive various exact or
approximate Bayesian inference algorithms.
■ Sparsity is a prior assumption imposed on the model,
which can also be used for model selection.
VB MEtHods
This section presents several efficient VB methods developed for EEG/
MEG signal processing. Let H denote collectively all the unknown
parameters to be inferred in (1). We focus on the VB-MF approxima-
tion, which seeks a factorable distribution q to approximate the true
posterior p by minimizing the KL divergence
KL (q p) = - # q (H) log ;

p (H | X)
E dH,
q (H)

(13)

where q (H) = % l = 1 q (H l), and H l (l = 1, g, L) denote the dis-
joint groups of variables in H. Direct minimization of (13) is diffi-
cult, but it is easy to show that the log marginal likelihood of the
data can be written as
L

log p (X) = L (q) + KL (q p),

(14)

with
L (q) =

# q (H) log ; pq(X(H, H) ) EdH.

(15)

Since KL (q p) $ 0, L (q) is a lower bound of the log mar-
ginal likelihood, which is termed the variational free energy.
In light of (14), minimizing the KL divergence is equivalent to
maximizing L (q), which is more computationally tractable
using numerical optimization algorithms. The most widely
used algorithm is coordinate ascent, which alternately updates
the approximate distribution of each parameter {q (H l)} until
convergence. In brief, this coordinate ascent can be thought of
as the solution for one approximate posterior marginal distri-
bution that is expressed in terms of the others. By stepping
through the different subsets of unknown parameters, we can
iteratively update the approximate marginals.
In the presence of latent variable Z, the variational
posterior is often assumed to have a factorial form:
q (Z, H) = q (Z) q (H) . Similarly, maximizing the variational
free energy with respect to two functions q (Z) and q (H) alter-
natively gives rise to the so-called VB-EM (expectation-maxi-
mization) algorithm [15]. In the VB-E (expectation) step, set
[2L /2q (Z)] = 0 and update the variational posterior q (Z); in
the VB-M (maximization) step, set [2L/2q (H)] = 0 and update

Model Specification
Hyperpriors p ( )
("VB Methods," "SBL Methods,"
and "NB Methods" Sections)

Priors p (Z | )
Parametric/Nonparametric
("VB Methods," "SBL Methods,"
and "NB Methods" Sections)

Variational Free
Energy

p (X |Z )

Marginal
Likelihood
p (X | )

Posterior
p (Z |X )

(q (Z, ))

Likelihood

Objective Function

VB Methods
maxq (q (Z, ))
with Mean-Field or
Lapace Approximation
("VB Methods" and
"SBL Methods" Sections)

Sampling Methods
Estimate p (Z |X ) with
Stochastic Approximation
("NB Methods" Section)

Penalized Likelihood Methods
maxz p (X |Z )p (Z )
("VB Methods" and
"SBL Methods" Sections)

EB
max p (X | )
("SBL Methods"
Section)

Inference Algorithm
[FIG1] a schematic illustration of Bayesian inference and the BML techniques introduced in this tutorial. Bayesian inference is carried out via
two phases: model specification and inference. In the model-specification phase, the likelihood L (Z) = p (X | Z), which describes how the
EEG/MEG data X are related to the unknown variables Z (including parameters and latent variables) is specified. Parametric or nonparametric
priors can be imposed on Z. the hierarchy is built up by imposing hyperpriors p (c) on the hyperparameters c in the priors. In the inference
phase, a particular algorithm is chosen to infer Z, based on deterministic or stochastic approximations. under specific conditions, empirical
Bayesian methods, penalized likelihood methods, and VB methods are equivalent to each other. see the section "Bayesian Inference: a Brief
overview" for details.

IEEE SIGNAL PROCESSING MAGAZINE [19] jANuARy 2016
Table of Contents for the Digital Edition of Signal Processing - January 2016
