IEEE Signal Processing - May 2018 - 26

The Pinsker-Kolmogorov expression, with a possible spectral
reweighting, provides a mechanism to determine those parts
of the signal that should be removed in an optimal encoding
subject to the bit-rate constraint.
This provides the minimal distortion in any system that is
used to recover a length T realization of X (t) having no more
than 26TR@ states. A special case of such a system is PCM of
the section "ADX Via Pulse-Code Modulation," and, therefore,
when X (t) is a Gaussian process, the distortion in (5) is bounded from below by (6).
In general, the optimal encoder that attains Shannon's
DRF operates in continuous time. Upon receiving a realization of X (t) over [-T/2, T/2], the encoder compares this
realization to each of the 26TR@ reconstruction waveforms
[33], [34]. We note, however, that Shannon's DRF is attainable even if this encoder is required to first map the analog
waveform to a discrete-time sequence. Indeed, this discretetime sequence can be the random coefficients in the analog signal's expansion according to some predetermined
orthogonal basis. Consequently, encoding and decoding may
be performed with respect to this discrete sequence without changing the fundamental distortion limit described by
the DRF in (6). We emphasize that the equivalence between
analog signals and coefficients in their basis expansion holds
regardless of whether the original process X (t) is bandlimited or not [40].
One commonly used example for such an orthogonal basis
is the Karhunen-Loèeve (KL) basis [41]. The latter's functions
are chosen as the eigenfunctions of the bilinear kernel defined
by the covariance of X (t) . As a result, the coefficients in this
expansion are orthogonal to each other and, in fact, independent in our case of Gaussian signals. This fact implies that the
KL expansion decomposes the process X (t) over the interval
6-T/2, T/2@ into a discrete Gaussian sequence of independent random variables, where the variance of each element is
proportional to the eigenvalue associated with the eigenfunction. Since X (t) is stationary, multiple sequences of this type
obtained from different length T blocks of X (t) are identically
distributed and, therefore, can be encoded using the same
block T encoder that essentially encodes multiple discrete
Gaussian sequences. The optimal encoding of such a sequence
using 6TR@ bits is achieved according to the water-filling principle, as described in "The Water-Filling Scheme." Moreover,
as T extends to infinity, the density of the KL eigenvalues is
described by the PSD S X (f) of X (t) , and the average distortion in encoding each block converges to (6) [41]. The prior
described coding procedure is one way to show that Pinsker
and Kolmogorov's water-filling expression (6) is attainable.
To implement any of the optimal encoding schemes of the
analog signal described previously, it is required to represent
it first by a discrete sequence of coefficients. However, the
implementation of this transformation is subject to practical
limitations. In particular, realizable hardware such as filters
and pointwise samplers are limited in the number of coefficient values they produce per unit time [7]. That is, for a time
lag T, there exists a number fs such that any system consist26

ing of these operations does not produce more than T fs analog
samples. In the next section, we explore the minimal distortion
that can be attained under this restriction. We are especially
interested in the minimal sampling rate fs that is required to
achieve Shannon's DRF.

ADX via sampling
We have seen that the optimal tradeoff between MSE distortion
and bit rate in the digital representation of an analog signal is
described by Shannon's DRF of the signal. In this section, we
explore the minimal distortion under the additional constraint
that the digital representation must be a function of the samples
of the analog signal, rather than the analog signal itself.

Lossy compression from samples
In the ADX setting of Figure 2, the encoder observes samples
of the source signal X (t) , and is required to encode these samples so that X (t) can be estimated from this encoding using
MMSE. Specifically, assuming that the sampler observes X (t)
for t ! [-T/2, T/2] , we denote by Y the 6Tfs@ -dimensional
random vector resulting from sampling X (t) at rate fs . The
encoder maps the vector Y to a digital word of length 6TR@
and delivers this sequence without errors to the decoder. The
latter provides an estimate Xt (t) for X (t) , t ! [-T/2, T/2] ,
based on only the digital sequence and the statistics of X (t) .
The distortion between X (t) and its reconstruction for a fixed
sampler S is defined by
D S (fs, R) = inf 1
T

#-TT//22 E^ X (t) - Xt (t)h2 dt.

(7)

The infimum in (7) is over encoders, decoders, and time
horizons T. We note that, under the assumption that X ($) and
its samples are stationary, any finite time-horizon encoding
strategy may be transformed into an infinite time-horizon
strategy by applying it to consecutive blocks. As a result,
increasing the time horizon cannot increase the distortion, and
the minimum over the time horizon in (7) can be replaced by
the limit T " 3.
As an example, in the PCM encoding described in the
"ADX Via Pulse-Code Modulation" section, S is a pointwise
sampler at sampling rate fs preceded by an LPF. The particular
encoder and decoder used in PCM was described in Figure 4.
Therefore, since the optimization in (7) is over all encoders and
decoders, for any signal for which pointwise sampling is well
defined, we have D S (fs, R) # D PCM (fs, R) .
Characterizing D S (fs, R) gives rise to a source coding problem in which the encoder has no direct access to the source
signal it is required to describe. Source coding problems of this
type are referred to as remote or indirect source coding problems [6]. More details on this class of problems is provided
in "Indirect Source Coding." Under the MSE criterion (7), the
optimal encoding scheme of most indirect source coding problems is obtained by a simple two-step procedure [43], [44], [6]:
1) Estimate X (t) from its samples Y subject to the MSE
criterion (7), i.e., compute the conditional expectation

IEEE Signal Processing Magazine

|

May 2018

|



Table of Contents for the Digital Edition of IEEE Signal Processing - May 2018

Contents
IEEE Signal Processing - May 2018 - Cover1
IEEE Signal Processing - May 2018 - Cover2
IEEE Signal Processing - May 2018 - Contents
IEEE Signal Processing - May 2018 - 2
IEEE Signal Processing - May 2018 - 3
IEEE Signal Processing - May 2018 - 4
IEEE Signal Processing - May 2018 - 5
IEEE Signal Processing - May 2018 - 6
IEEE Signal Processing - May 2018 - 7
IEEE Signal Processing - May 2018 - 8
IEEE Signal Processing - May 2018 - 9
IEEE Signal Processing - May 2018 - 10
IEEE Signal Processing - May 2018 - 11
IEEE Signal Processing - May 2018 - 12
IEEE Signal Processing - May 2018 - 13
IEEE Signal Processing - May 2018 - 14
IEEE Signal Processing - May 2018 - 15
IEEE Signal Processing - May 2018 - 16
IEEE Signal Processing - May 2018 - 17
IEEE Signal Processing - May 2018 - 18
IEEE Signal Processing - May 2018 - 19
IEEE Signal Processing - May 2018 - 20
IEEE Signal Processing - May 2018 - 21
IEEE Signal Processing - May 2018 - 22
IEEE Signal Processing - May 2018 - 23
IEEE Signal Processing - May 2018 - 24
IEEE Signal Processing - May 2018 - 25
IEEE Signal Processing - May 2018 - 26
IEEE Signal Processing - May 2018 - 27
IEEE Signal Processing - May 2018 - 28
IEEE Signal Processing - May 2018 - 29
IEEE Signal Processing - May 2018 - 30
IEEE Signal Processing - May 2018 - 31
IEEE Signal Processing - May 2018 - 32
IEEE Signal Processing - May 2018 - 33
IEEE Signal Processing - May 2018 - 34
IEEE Signal Processing - May 2018 - 35
IEEE Signal Processing - May 2018 - 36
IEEE Signal Processing - May 2018 - 37
IEEE Signal Processing - May 2018 - 38
IEEE Signal Processing - May 2018 - 39
IEEE Signal Processing - May 2018 - 40
IEEE Signal Processing - May 2018 - 41
IEEE Signal Processing - May 2018 - 42
IEEE Signal Processing - May 2018 - 43
IEEE Signal Processing - May 2018 - 44
IEEE Signal Processing - May 2018 - 45
IEEE Signal Processing - May 2018 - 46
IEEE Signal Processing - May 2018 - 47
IEEE Signal Processing - May 2018 - 48
IEEE Signal Processing - May 2018 - 49
IEEE Signal Processing - May 2018 - 50
IEEE Signal Processing - May 2018 - 51
IEEE Signal Processing - May 2018 - 52
IEEE Signal Processing - May 2018 - 53
IEEE Signal Processing - May 2018 - 54
IEEE Signal Processing - May 2018 - 55
IEEE Signal Processing - May 2018 - 56
IEEE Signal Processing - May 2018 - 57
IEEE Signal Processing - May 2018 - 58
IEEE Signal Processing - May 2018 - 59
IEEE Signal Processing - May 2018 - 60
IEEE Signal Processing - May 2018 - 61
IEEE Signal Processing - May 2018 - 62
IEEE Signal Processing - May 2018 - 63
IEEE Signal Processing - May 2018 - 64
IEEE Signal Processing - May 2018 - 65
IEEE Signal Processing - May 2018 - 66
IEEE Signal Processing - May 2018 - 67
IEEE Signal Processing - May 2018 - 68
IEEE Signal Processing - May 2018 - 69
IEEE Signal Processing - May 2018 - 70
IEEE Signal Processing - May 2018 - 71
IEEE Signal Processing - May 2018 - 72
IEEE Signal Processing - May 2018 - 73
IEEE Signal Processing - May 2018 - 74
IEEE Signal Processing - May 2018 - 75
IEEE Signal Processing - May 2018 - 76
IEEE Signal Processing - May 2018 - 77
IEEE Signal Processing - May 2018 - 78
IEEE Signal Processing - May 2018 - 79
IEEE Signal Processing - May 2018 - 80
IEEE Signal Processing - May 2018 - 81
IEEE Signal Processing - May 2018 - 82
IEEE Signal Processing - May 2018 - 83
IEEE Signal Processing - May 2018 - 84
IEEE Signal Processing - May 2018 - 85
IEEE Signal Processing - May 2018 - 86
IEEE Signal Processing - May 2018 - 87
IEEE Signal Processing - May 2018 - 88
IEEE Signal Processing - May 2018 - 89
IEEE Signal Processing - May 2018 - 90
IEEE Signal Processing - May 2018 - 91
IEEE Signal Processing - May 2018 - 92
IEEE Signal Processing - May 2018 - 93
IEEE Signal Processing - May 2018 - 94
IEEE Signal Processing - May 2018 - 95
IEEE Signal Processing - May 2018 - 96
IEEE Signal Processing - May 2018 - 97
IEEE Signal Processing - May 2018 - 98
IEEE Signal Processing - May 2018 - 99
IEEE Signal Processing - May 2018 - 100
IEEE Signal Processing - May 2018 - 101
IEEE Signal Processing - May 2018 - 102
IEEE Signal Processing - May 2018 - 103
IEEE Signal Processing - May 2018 - 104
IEEE Signal Processing - May 2018 - 105
IEEE Signal Processing - May 2018 - 106
IEEE Signal Processing - May 2018 - 107
IEEE Signal Processing - May 2018 - 108
IEEE Signal Processing - May 2018 - 109
IEEE Signal Processing - May 2018 - 110
IEEE Signal Processing - May 2018 - 111
IEEE Signal Processing - May 2018 - 112
IEEE Signal Processing - May 2018 - 113
IEEE Signal Processing - May 2018 - 114
IEEE Signal Processing - May 2018 - 115
IEEE Signal Processing - May 2018 - 116
IEEE Signal Processing - May 2018 - 117
IEEE Signal Processing - May 2018 - 118
IEEE Signal Processing - May 2018 - 119
IEEE Signal Processing - May 2018 - 120
IEEE Signal Processing - May 2018 - 121
IEEE Signal Processing - May 2018 - 122
IEEE Signal Processing - May 2018 - 123
IEEE Signal Processing - May 2018 - 124
IEEE Signal Processing - May 2018 - 125
IEEE Signal Processing - May 2018 - 126
IEEE Signal Processing - May 2018 - 127
IEEE Signal Processing - May 2018 - 128
IEEE Signal Processing - May 2018 - 129
IEEE Signal Processing - May 2018 - 130
IEEE Signal Processing - May 2018 - 131
IEEE Signal Processing - May 2018 - 132
IEEE Signal Processing - May 2018 - Cover3
IEEE Signal Processing - May 2018 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201809
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201807
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201805
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201803
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_201801
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1117
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0917
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0717
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0517
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0317
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0117
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1116
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0916
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0716
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0516
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0316
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0116
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1115
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0915
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0715
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0515
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0315
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0115
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1114
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0914
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0714
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0514
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0314
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0114
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1113
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0913
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0713
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0513
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0313
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0113
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1112
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0912
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0712
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0512
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0312
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0112
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1111
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0911
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0711
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0511
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0311
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0111
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1110
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0910
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0710
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0510
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0310
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0110
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1109
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0909
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0709
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0509
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0309
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0109
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_1108
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0908
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0708
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0508
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0308
https://www.nxtbook.com/nxtbooks/ieee/signalprocessing_0108
https://www.nxtbookmedia.com