IEEE Circuits and Systems Magazine - Q3 2023 - 34

[92] J. Bradbury et al. (2018). JAX: Composable Transformations of
Python+NumPy Programs. [Online]. Available: http://github.com/google/jax
[93] T. Chen et al., " TVM: An automated end-to-end optimizing compiler for
deep learning, " in Proc. Oper. Syst. Des Implementation, 2018, pp. 580-594.
[94] TensorFlow Lite. [Online]. Available: https://www.tensorflow.org/lite
[95] X. Jiang et al., " MNN: A universal and efficient inference engine, " in
Proc. MLSys, 2020, pp. 1-13.
[96] NCNN: A High-Performance Neural Network Inference Computing
Framework Optimized for Mobile Platforms. [Online]. Available: https://
github.com/Tencent/ncnn
[97] NVIDIA TensorRT, an SDK for High-Performance Deep Learning Inference.
[Online]. Available: https://developer.nvidia.com/tensorrt
[98] A. Vaswani et al., " Attention is all you need, " in Proc. NeurIPS, 2017,
pp. 1-11.
[99] T. Chen et al., " Learning to optimize tensor programs, " in Proc. NeurIPS,
2018, pp. 1-12.
[100] A. Stoutchinin, F. Conti, and L. Benini, " Optimally scheduling CNN
convolutions for efficient memory access, " 2019, arXiv:1902.01492.
[101] B. H. Ahn et al., " Ordering chaos: Memory-aware scheduling of irregularly
wired neural networks for edge devices, " 2020, arXiv:2003.02369.
[102] H. Miao and F. X. Lin, " Enabling large neural networks on tiny microcontrollers
with swapping, " 2021, arXiv:2101.08744.
[103] M. Alwani et al., " Fusedlayer CNN accelerators, " in Proc. 49th Annu.
IEEE/ACM Int. Symp. Microarchitecture (MICRO), Oct. 2016, pp. 1-12.
[104] K. Goetschalckx and M. Verhelst, " Breaking highresolution CNN
bandwidth barriers with enhanced depthfirst execution, " IEEE J. Emerg.
Sel. Topics Circuits Syst., vol. 9, no. 2, pp. 323-331, Jun. 2019.
[105] O. Saha et al., " RNNPool: Efficient non-linear pooling for RAM constrained
inference, " 2020, arXiv:2002.11921.
[106] M. Tan and Q. Le, " EfficientNet: Rethinking model scaling for convolutional
neural networks, " in Proc. Int. Conf. Mach. Learn., 2019, pp.
6105-6114.
[107] M. Tan and Q. V. Le, " EfficientNetV2: Smaller models and faster
training, " 2021, arXiv:2104.00298.
[108] A. Gruslys et al., " Memory-efficient backpropagation through
time, " in Proc. NeurIPS, 2016, pp. 4132-4140.
[109] T. Chen et al., " Training deep nets with sublinear memory cost, "
2016, arXiv:1604.06174.
[110] K. Greff, R. K. Srivastava, and J. Schmidhuber, " Highway and residual
networks learn unrolled iterative estimation, " in Proc. Int. Conf.
Learning Representations, 2017, pp. 1-14.
[111] L. Liu et al., " Dynamic sparse graph for efficient deep learning, " in
Proc. Int. Conf. Learning Representations, 2019, pp. 1-18.
[112] Y. Wang et al., " E2-train: Training state-of-the-art CNNs with over
80% energy savings, " 2019, arXiv:1910.13349.
[113] N. Wang et al., " Training deep neural networks with 8-bit floating
point numbers, " in Proc. NeurIPS, 2018, pp. 1-10.
[114] X. Sun et al., " Hybrid 8-bit floating point (HFP8) training and inference
for deep neural networks, " in Proc. NeurIPS, 2019.
[115] A. Krizhevsky, I. Sutskever, and G. E. Hinton, " ImageNet classification
with deep convolutional neural networks, " in Proc. Adv. Neural Inf.
Process. Syst. (NIPS), 2012, pp. 1-9.
[116] Y. Cui et al., " Large scale fine-grained categorization and domainspecific
transfer learning, " in Proc. Conf. Comput. Vis. Pattern Recognit.,
Jun. 2018, pp. 4109-4118.
[117] S. Kornblith, J. Shlens, and Q. V. Le, " Do better ImageNet models
transfer better? " in Proc. Conf. Comput. Vis. Pattern Recognit., Jun. 2019,
pp. 2656-2666.
[118] A. Kolesnikov et al., " Big transfer (BiT): General visual representation
learning, " in Proc. Eur. Conf. Comput. Vis., 2020, pp. 491-507.
[119] K. Chatfield et al., " Return of the devil in the details: Delving deep
into convolutional nets, " in Proc. Brit. Mach. Vis. Conf., 2014, pp. 1-11.
[120] J. Donahue et al., " DeCAF: A deep convolutional activation feature for
generic visual recognition, " in Proc. Int. Conf. Mach. Learn., 2014, pp. 1-9.
[121] C. Gan et al., " DevNet: A deep event network for multimedia event
detection and evidence recounting, " in Proc. Conf. Comput. Vis. Pattern
Recognit., Jun. 2015, pp. 2568-2577.
[122] A. S. Razavian et al., " CNN features off-the-shelf: An astounding
baseline for recognition, " in Proc. Conf. Comput. Vis. Pattern Recognit. Workshops,
Jun. 2014, pp. 512-519.
[123] Q. Wang et al., " Melon: Breaking the memory wall for resource-efficient
on-device machine learning, " in Proc. 20th Annu. Int. Conf. Mobile
Syst., Appl. Services, 2022, pp. 450-463.
34
IEEE CIRCUITS AND SYSTEMS MAGAZINE
[124] D. Xu et al., " Mandheling: Mixed- precision on-device DNN training
with DSP offloading, " in Proc. 28th Annu. Int. Conf. Mobile Comput. Netw.,
(MobiCom), 2022, pp. 214-227.
[125] I. Gim and J. Ko, " Memory-efficient DNN training on mobile devices, "
in Proc. 20th Annu. Int. Conf. Mobile Syst., Appl. Services, 2022,
pp. 464-476.
[126] J. Frankle, D. J. Schwab, and A. S. Morcos, " Training BatchNorm
and only BatchNorm: On the expressive power of random features in
CNNs, " 2020, arXiv:2003.00152.
[127] P. K. Mudrakarta et al., " K for the price of 1: Parameter efficient
multi-task and transfer learning, " in Proc. Int. Conf. Learning Representations,
2019, pp. 1-15.
[128] A. Canziani, A. Paszke, and E. Culurciello, " An analysis of deep neural
network models for practical applications, " 2016, arXiv:1605.07678.
[129] H. Cai et al., " Once for all: Train one network and specialize it for efficient
deployment, " in Proc. Int. Conf. Learning Representations, 2020, pp. 1-15.
[130] G. Bender et al., " Understanding and simplifying one-shot architecture
search, " in Proc. Int. Conf. Mach. Learn., 2018, pp. 1-10.
[131] Z. Guo et al., " Single path one-shot neural architecture search
with uniform sampling, " 2019, arXiv:1904.00420.
[132] K. Simonyan and A. Zisserman, " Very deep convolutional networks
for large-scale image recognition, " in Proc. Int. Conf. Learn. Represent.,
2015, pp. 1-14.
[133] A. Howard et al., " Searching for MobileNetV3, " in Proc. Int. Conf.
Comput. Vis., Oct.-Nov. 2019, pp. 1314-1324.
[134] M. Everingham et al., " The Pascal visual object classes (VOC)
challenge, " Int. J. Comput. Vis., vol. 88, no. 2, pp. 303-338, 2010.
[135] J. Redmon and A. Farhadi, " YOLOv3: An Incremental Improvement, "
2018, arXiv:1804.02767.
[136] S. Yang et al., " WIDER FACE: A face detection benchmark, " in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 5525-5533.
[137] Y. Yoo, D. Han, and S. Yun, " EXTD: Extremely tiny face detector via
iterative filter reuse, " 2019, arXiv:1906.06579.
[138] Y. He et al., " LFFD: A light and fast face detector for edge devices, "
2019, arXiv:1904.10633.
[139] X. Zhao et al., " Real-time multi-scale face detector on embedded
devices, " Sensors, vol. 19, no. 9, p. 2158, 2019.
[140] S. Zhang et al., " S3FD: Single shot scale-invariant face detector, " in
Proc. IEEE Int. Conf. Comput. Vis., Oct. 2017, pp. 192-201.
[141] A. Burrello et al., " DORY: Automatic end-to-end deployment of
real-world DNNs on low-cost IoT MCUs, " IEEE Trans. Comput., vol. 70,
no. 8, pp. 1253-1268, Aug. 2021.
[142] E. Liberis, L. Dudziak, and N. D. Lane, " μNAS: Constrained neural
architecture search for microcontrollers, " 2020, arXiv:2010.14246.
[143] I. Fedorov et al., " SpArSe: Sparse architecture search for CNNs on
resourceconstrained microcontrollers, " in Proc. NeurIPS, 2019, pp. 1-13.
[144] M. Abadi et al. (2015). TensorFlow: Largescale Machine Learning on
Heterogeneous Systems. [Online]. Available: http://tensorflow.org/
[145] S. Ioffe and C. Szegedy, " Batch normalization: Accelerating deep
network training by reducing internal covariate shift, " in Proc. Int. Conf.
Mach. Learn., 2015, pp. 1-9.
[146] B. Jacob et al., " Quantization and training of neural networks for
efficient integer-arithmetic-only inference, " in Proc. Conf. Comput. Vis.
Pattern Recognit., Jun. 2018, pp. 2704-2713.
[147] A. Krizhevsky and G. Hinton, " Learning multiple layers of features
from tiny images, " M.S. thesis, Dept. Comput. Sci., Univ. Toronto,
Toronto, ON, Canada, 2009.
[148] D. P. Kingma and J. Ba, " Adam: A method for stochastic optimization, "
2014, arXiv:1412.6980.
[149] Y. You, I. Gitman, and B. Ginsburg, " Large batch training of convolutional
networks, " 2017, arXiv:1708.03888.
[150] J. Krause et al., " 3D object representations for fine-grained categorization, "
in Proc. IEEE Int. Conf. Comput. Vis. Workshops, Dec. 2013,
pp. 554-561.
[151] Y. Sun et al., " Test-time training with self-supervision for generalization
under distribution shifts, " in Proc. Int. Conf. Mach. Learn., 2020,
pp. 9229-9248.
[152] T. Dettmers et al., " LLM.int8 (): 8-bit matrix multiplication for
transformers at scale, " 2022, arXiv:2208.07339.
[153] G. Xiao et al., " SmoothQuant: Accurate and efficient post-training
quantization for large language models, " 2022, arXiv:2211.10438.
[154] A. Chowdhery et al., " PaLM: Scaling language modeling with pathways, "
in Proc. Mach. Learn. Syst., 2022, pp. 1-87.
THIRD QUARTER 2023
http://www.github.com/google/jax https://www.tensorflow.org/lite http://www.github.com/Tencent/ncnn http://www.github.com/Tencent/ncnn https://developer.nvidia.com/tensorrt http://www.tensorflow.org/
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
