IEEE Signal Processing - May 2018 - 38

constraints on power, memory, communication, and computation. In addition, hardware and modeling constraints in processing analog information imply that the digital representation is
obtained by first sampling the analog waveform and then quantizing or encoding its samples. That is, the transformation from
analog signals to bits involves the composition of sampling and
quantization or, more generally, lossy compression operations.
In this article, we explored the minimal sampling rate
required to attain the fundamental distortion limit subject to a
strict constraint on the bit rate of the system. We concluded that
when the energy of the signal is not uniformly distributed over
its spectral occupancy, the optimal signal representation can
be attained by sampling at a rate lower than the Nyquist rate,
which depends on the actual bit-rate constraint. This reduction
in the optimal sampling rate under finite bit precision is made
possible by designing the sampling mechanism to sample only
those parts of the signals that are not discarded because of
optimal lossy compression.
The characterization of the fundamental distortion limit
and the sampling rate required to attain it has several important
implications. Most importantly, it provides an extension of the
classical sampling theory of Whittaker, Kotelnikov, Shannon,
and Landau, as it describes the minimal sampling rate required
for attaining the minimal distortion in sampling an analog signal. It also leads to a theory of representing signals of infinite
bandwidth with a vanishing distortion. In particular, it provides
the average number of bits per sample, i.e., the ratio of the bit
rate (bits per unit of time) and the sampling rate (samples per
unit of time) so that, as the number of bits and samples per unit
of time extend to infinity, the ratio between the distortion under
optimal sampling and encoding and the DRF decreases to one.
Our results also indicate that sampling at the Nyquist rate is
not necessary when working under a bit-rate constraint for signals of either finite or infinite bandwidth. Such a constraint may
be due to hardware power, cost, or memory limitations. Moreover, sampling a signal at its critical sampling rate associated
with a given bit-rate constraint results in the most compact digital
representation of the analog signal and thus provides a mechanism to remove redundant information at the sensing stage.

Acknowledgments
This work was supported in part by the National Science
Foundation (NSF) under grant CCF-1320628, by the NSF's
Center for Science of Information grant CCF-0939370, and by
the U.S.-Israel Binational Science Foundation (BSF) under the
BSF Transformative Science grant 2010505.

References

[1] E. T. Whittaker, "On the functions which are represented by the expansions of
the interpolation theory," Proc. Roy. Soc. Edinburgh, vol. 35, pp. 181-194, July
1915.
[2] C. E. Shannon, "Communication in the presence of noise," Proc. IRE, vol. 37,
no. 1, pp. 10-21, 1949.
[3] H. Landau, "Sampling, data transmission, and the Nyquist rate," Proc. IEEE,
vol. 55, no. 10, pp. 1701-1706, Oct. 1967.

Authors
Alon Kipnis (kipnisal@stanford.edu) received his B.Sc.
degree in mathematics (summa cum laude) and his B.Sc.
degree in electrical engineering (summa cum laude), both in
in 2010, and his M.Sc. degree in mathematics in 2012, all
from Ben-Gurion University of the Negev, Israel. He recently
received his Ph.D. degree in electrical engineering from
Stanford University, California, where he is now a postdoctoral scholar in the Department of Statistics. His research focuses
38

on the intersection of signal processing, machine learning, and
statistics with data compression.
Yonina C. Eldar (yonina@ee.technion.ac.il) is a professor
in the Department of Electrical Engineering at the Technion-
Israel Institute of Technology, Haifa, where she holds the
Edwards Chair in engineering. She is also an adjunct professor at Duke University, Durham, North Carolina, and a
research affiliate with the Research Laboratory of Electronics
at the Massachusetts Institute of Technology, Cambridge, and
she was a visiting professor at Stanford University, California.
She is a member of the Israel Academy of Sciences and
Humanities and of the European Association for Signal
Processing. She has received many awards for excellence in
research and teaching, including the IEEE Signal Processing
Society Technical Achievement Award, the IEEE/Aerospace
and Electronic Systems Society Fred Nathanson Memorial
Radar Award, the IEEE Kiyo Tomiyasu Award, the Michael
Bruno Memorial Award from the Rothschild Foundation, the
Weizmann Prize for Exact Sciences, and the Wolf Foundation Krill Prize for Excellence in Scientific Research. She is
the editor-in-chief of Foundations and Trends in Signal
Processing and serves the IEEE on several technical and
award committees. She is a Fellow of the IEEE.
Andrea J. Goldsmith (andrea@wsl.stanford.edu) received
her B.S. degree in 1986, her M.S. degree in 1989, and her
Ph.D. degree in 1994, all in electrical engineering, from the
University of California, Berkeley. She is the Stephen Harris
professor of electrical engineering at Stanford University,
California. She also cofounded and served as chief technology
officer of Plume WiFi (formerly Accelera Inc.) and of
Quantenna Communications. She is a member of the National
Academy of Engineering and the American Academy of Arts
and Sciences. She has received several awards for her work,
including the IEEE Communications Society Edwin Howard
Armstrong Achievement Award and the Silicon Valley
Business Journal's Women of Influence Award. She has
authored three books on wireless communications and is an
inventor with 28 patents. Her research interests are in information and communication theory and their application to wireless communications and related fields.

[4] C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J.,
vol. 27, no. 4, pp. 379-423, 1948.
[5] C. E. Shannon, "Coding theorems for a discrete source with a fidelity criterion,"
IRE Nat. Conv. Rec., vol. 4, pp. 142-163, 1959.
[6] T. Berger, Rate-Distortion Theory: A Mathematical Basis for Data
Compression. Englewood Cliffs, NJ: Prentice Hall, 1971.
[7] Y. C. Eldar, Sampling Theory: Beyond Bandlimited Systems. Cambridge, U.K.:
Cambridge Univ. Press, 2015.
[8] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no.
7553, pp. 436-444, 2015.

IEEE Signal Processing Magazine

May 2018

Table of Contents for the Digital Edition of IEEE Signal Processing - May 2018