IEEE Signal Processing - May 2018 - 37

constraints may also be inherent in the system model and
the estimation problem. As an example, consider the estimation of an analog signal describing the behavior of the
price of a financial asset. Although we assume that the price
follows some continuous-time behavior, the value of the asset
is only observed whenever a transaction is reported. This limitation on the observation can be described by a sampling constraint. If the transactions occur at nonuniform time lags,
then this sampling is nonuniform. Moreover, it is often assumed that the instantaneous change of the price is given by a
deterministic signal representing the drift plus an additive
infinite bandwidth and stationary noise [9]. Therefore, the
signal in question is of infinite bandwidth, and sampling occurs
below the Nyquist rate.
In addition to the sampling constraint, it may be the case
that the values of the transactions are hidden from us. The
only information we receive is through a sequence of actions
taken by the agent controlling this asset. Assuming the set of
possible actions is finite, this last limitation corresponds to a
quantization constraint. Therefore, the MMSE in estimating
the continuous-time price based on the sequence of actions is
described by the minimal distortion in the ADX.
While, in this case, we have no control over the actual way
the samples are encoded (into actions), the minimal distortion
in the ADX setting provides a lower bound on the distortion in
estimating the continuous-time price. This distortion can be
expressed by an additional noise in a model that makes decisions based on the estimated price.

Removing redundancy at the sensing stage

dant digital representation of the analog signal. For example,
oversampling in the PCM as in "ADX Via Pulse-Code Modulation" section leads to a redundant digital representation of
the quantized samples, since these become more correlated
with one another as the sampling rate increases. Indeed, the
properties of PCM imply that the optimal sampling rate that
minimizes the distortion also maximizes the entropy rate of
its digital output.
The counterpart of the PCM redundancy phenomena in the
more general setting of ADX is the representation attained by
optimal sampling at the critical rate. This optimal sampling
can be seen as a mechanism to remove redundancy at the sampling stage. It guarantees that the signal postsampling does
not contain any parts that would be removed under an optimal
lossy compression.
As an example of a system that benefits from operating
according to the previously discussed principle, we envision a
real-time voice-to-text transcriber based on an artificial neural
network [65]. Such a system consists of an artificial neural network that maps a sequence of bits to words, where this sequence
is obtained by an ADX unit, as shown in Figure 21. Since the
rate of information per unit time that can be processed by the
neural net is limited, an optimal design of the ADX would provide bits into the neural network consistent with this rate. The
challenge is, therefore, to sample and encode the audio signal
at the rate of the neural network processing so as to provide
the most relevant information subject to that rate constraint for
the network to perform its classification task. If we assume that
the most relevant information is described by a spectral psychoacoustic distortion function, then the optimal ADX scheme
with a signal PSD weighted by this distortion function provides
the most relevant information for classification, subject to the
processing constraint.

"Hello"

At the end of the section "ADX Via Pulse-Code Modulation,"
we concluded that, under an optimal encoder, oversampling
does not affect the fundamental distortion limit, since the
introduced redundancy is removed in the encoding. However,
oversampling may still be undesirable to the overall system
Conclusions
performance, since it results in redundant data that must be
The processing, communication, and/or digital storage of an anaremoved by additional processing. In fact, since analog signal
log signal is achieved by first representing it as a bit sequence.
processing is not constrained by memory or bit rate, when inforThe restriction on the bit rate of this sequence is the result of
mation originating in an analog signal is converted to a digital form, it
may bloat the system's memory
with a large amount of redundant
data. The processing of these data
0011101000110
requires additional resources that
1011110110101
are proportional to their size and
0100011110011
Neural
may severely restrict the system's
0101001010101
ADX
Network
ability to extract useful information.
1110010001010
Indeed, the lack of computational
1110001110101
resources for the extraction of use1110001010111
ful information from large data sets
1000101010101
is one of the most pressing issues of
Bit Rate ≤ Processing Rate
the digital age [64].
One way to address this big data
challenge is by collecting only rel- FIGURE 21. The bit rate of a digital representation of the sound of the word hello should not exceed the processing
evant information from the analog rate of the neural network. Sampling and lossy compression according to ADX preserves the most relevant part of
world, i.e., attaining a nonredun- the analog signal with respect to the distortion criterion and subject to the bit-rate constraint.
IEEE Signal Processing Magazine

May 2018

Table of Contents for the Digital Edition of IEEE Signal Processing - May 2018