IEEE Consumer Electronics Magazine - September/October 2019 - 13

challenging. Each individual has a unique ear
canal and our brains process and perceive audio
in a highly personalized way. And our audio
senses are attuned to visual cues, subconsciously anticipating changes in environmental
acoustics. Thus, in a large cathedral, you expect
echoes from your footsteps and voice; in a room
with carpets and soft furnishings, the acoustics
should be subdued. Thus, acoustic cues should
adapt to the surroundings of an MR-headset, or
the illusion of immersion is lost.
But energy usage on a headset is more critical
than on a smartphone, providing another excellent
use-case for Edge-AI: scene analysis36,37 and materials recognition38 combined with depth39,40 can
help build a detailed analysis of the surrounding
acoustic environment. And with state-of-the-art
Edge-AI, multiple neural networks can operate in
parallel at a fraction of the power budget for a GPU.

Image Signal Processing Pipeline in a Camera
Processing the raw Bayer data from an image
sensor is a classic example where camera engineers
and photographic experts have devoted many
years of effort to create a specialized image processing pipeline (IPP), illustrated in Figure 4. Bryce
Bayer's original patent41 filed in 1976 is a classic
example of an engineering compromise that works
so well that it has become the basis of modern digital imaging. Until recently, this "magic" happened
via complex set of image analysis and processing
algorithms, the IPP.42
But now, thanks to the magic of deep learning
methodologies, it has become feasible to replace
the IPP in an imaging system. This work began
with the demosaicking step of image conversion,43,44 followed by the idea to replace the key
steps of denoising and demosaicking from the IPP
with a single CNN network.45-47 Other authors
have progressed to complete replacement of the
IPP,48 or alternatively, to learn the detailed camera
model embodied in an existing IPP,49,50 which can
lead to reprogrammable IPPs. Imagine that you
can completely reprogram how your camera captures and develops raw images "on the fly." Well,
this approach is beginning to make its way into
actual products, so expect to see some exciting
new features in higher end digital cameras shortly!
Once you replace the IPP with a CNN, new
ideas emerge such as "Learning to see in the

September/October 2019

Figure 3. Off-axis iris regions; accurate per-pixel segmentation
is essential for user authentication.15

dark."51 In this particular example, researchers
have used a CNN to solve a key challenge for

Figure 4. Detailed view inside the IPP of a typical digital
camera.42

13



IEEE Consumer Electronics Magazine - September/October 2019

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - September/October 2019

Contents
IEEE Consumer Electronics Magazine - September/October 2019 - Cover1
IEEE Consumer Electronics Magazine - September/October 2019 - Cover2
IEEE Consumer Electronics Magazine - September/October 2019 - Contents
IEEE Consumer Electronics Magazine - September/October 2019 - 2
IEEE Consumer Electronics Magazine - September/October 2019 - 3
IEEE Consumer Electronics Magazine - September/October 2019 - 4
IEEE Consumer Electronics Magazine - September/October 2019 - 5
IEEE Consumer Electronics Magazine - September/October 2019 - 6
IEEE Consumer Electronics Magazine - September/October 2019 - 7
IEEE Consumer Electronics Magazine - September/October 2019 - 8
IEEE Consumer Electronics Magazine - September/October 2019 - 9
IEEE Consumer Electronics Magazine - September/October 2019 - 10
IEEE Consumer Electronics Magazine - September/October 2019 - 11
IEEE Consumer Electronics Magazine - September/October 2019 - 12
IEEE Consumer Electronics Magazine - September/October 2019 - 13
IEEE Consumer Electronics Magazine - September/October 2019 - 14
IEEE Consumer Electronics Magazine - September/October 2019 - 15
IEEE Consumer Electronics Magazine - September/October 2019 - 16
IEEE Consumer Electronics Magazine - September/October 2019 - 17
IEEE Consumer Electronics Magazine - September/October 2019 - 18
IEEE Consumer Electronics Magazine - September/October 2019 - 19
IEEE Consumer Electronics Magazine - September/October 2019 - 20
IEEE Consumer Electronics Magazine - September/October 2019 - 21
IEEE Consumer Electronics Magazine - September/October 2019 - 22
IEEE Consumer Electronics Magazine - September/October 2019 - 23
IEEE Consumer Electronics Magazine - September/October 2019 - 24
IEEE Consumer Electronics Magazine - September/October 2019 - 25
IEEE Consumer Electronics Magazine - September/October 2019 - 26
IEEE Consumer Electronics Magazine - September/October 2019 - 27
IEEE Consumer Electronics Magazine - September/October 2019 - 28
IEEE Consumer Electronics Magazine - September/October 2019 - 29
IEEE Consumer Electronics Magazine - September/October 2019 - 30
IEEE Consumer Electronics Magazine - September/October 2019 - 31
IEEE Consumer Electronics Magazine - September/October 2019 - 32
IEEE Consumer Electronics Magazine - September/October 2019 - 33
IEEE Consumer Electronics Magazine - September/October 2019 - 34
IEEE Consumer Electronics Magazine - September/October 2019 - 35
IEEE Consumer Electronics Magazine - September/October 2019 - 36
IEEE Consumer Electronics Magazine - September/October 2019 - 37
IEEE Consumer Electronics Magazine - September/October 2019 - 38
IEEE Consumer Electronics Magazine - September/October 2019 - 39
IEEE Consumer Electronics Magazine - September/October 2019 - 40
IEEE Consumer Electronics Magazine - September/October 2019 - 41
IEEE Consumer Electronics Magazine - September/October 2019 - 42
IEEE Consumer Electronics Magazine - September/October 2019 - 43
IEEE Consumer Electronics Magazine - September/October 2019 - 44
IEEE Consumer Electronics Magazine - September/October 2019 - 45
IEEE Consumer Electronics Magazine - September/October 2019 - 46
IEEE Consumer Electronics Magazine - September/October 2019 - 47
IEEE Consumer Electronics Magazine - September/October 2019 - 48
IEEE Consumer Electronics Magazine - September/October 2019 - 49
IEEE Consumer Electronics Magazine - September/October 2019 - 50
IEEE Consumer Electronics Magazine - September/October 2019 - 51
IEEE Consumer Electronics Magazine - September/October 2019 - 52
IEEE Consumer Electronics Magazine - September/October 2019 - 53
IEEE Consumer Electronics Magazine - September/October 2019 - 54
IEEE Consumer Electronics Magazine - September/October 2019 - 55
IEEE Consumer Electronics Magazine - September/October 2019 - 56
IEEE Consumer Electronics Magazine - September/October 2019 - 57
IEEE Consumer Electronics Magazine - September/October 2019 - 58
IEEE Consumer Electronics Magazine - September/October 2019 - 59
IEEE Consumer Electronics Magazine - September/October 2019 - 60
IEEE Consumer Electronics Magazine - September/October 2019 - 61
IEEE Consumer Electronics Magazine - September/October 2019 - 62
IEEE Consumer Electronics Magazine - September/October 2019 - 63
IEEE Consumer Electronics Magazine - September/October 2019 - 64
IEEE Consumer Electronics Magazine - September/October 2019 - 65
IEEE Consumer Electronics Magazine - September/October 2019 - 66
IEEE Consumer Electronics Magazine - September/October 2019 - 67
IEEE Consumer Electronics Magazine - September/October 2019 - 68
IEEE Consumer Electronics Magazine - September/October 2019 - 69
IEEE Consumer Electronics Magazine - September/October 2019 - 70
IEEE Consumer Electronics Magazine - September/October 2019 - 71
IEEE Consumer Electronics Magazine - September/October 2019 - 72
IEEE Consumer Electronics Magazine - September/October 2019 - 73
IEEE Consumer Electronics Magazine - September/October 2019 - 74
IEEE Consumer Electronics Magazine - September/October 2019 - 75
IEEE Consumer Electronics Magazine - September/October 2019 - 76
IEEE Consumer Electronics Magazine - September/October 2019 - 77
IEEE Consumer Electronics Magazine - September/October 2019 - 78
IEEE Consumer Electronics Magazine - September/October 2019 - 79
IEEE Consumer Electronics Magazine - September/October 2019 - 80
IEEE Consumer Electronics Magazine - September/October 2019 - 81
IEEE Consumer Electronics Magazine - September/October 2019 - 82
IEEE Consumer Electronics Magazine - September/October 2019 - 83
IEEE Consumer Electronics Magazine - September/October 2019 - 84
IEEE Consumer Electronics Magazine - September/October 2019 - 85
IEEE Consumer Electronics Magazine - September/October 2019 - 86
IEEE Consumer Electronics Magazine - September/October 2019 - 87
IEEE Consumer Electronics Magazine - September/October 2019 - 88
IEEE Consumer Electronics Magazine - September/October 2019 - 89
IEEE Consumer Electronics Magazine - September/October 2019 - 90
IEEE Consumer Electronics Magazine - September/October 2019 - 91
IEEE Consumer Electronics Magazine - September/October 2019 - 92
IEEE Consumer Electronics Magazine - September/October 2019 - 93
IEEE Consumer Electronics Magazine - September/October 2019 - 94
IEEE Consumer Electronics Magazine - September/October 2019 - 95
IEEE Consumer Electronics Magazine - September/October 2019 - 96
IEEE Consumer Electronics Magazine - September/October 2019 - 97
IEEE Consumer Electronics Magazine - September/October 2019 - 98
IEEE Consumer Electronics Magazine - September/October 2019 - 99
IEEE Consumer Electronics Magazine - September/October 2019 - 100
IEEE Consumer Electronics Magazine - September/October 2019 - 101
IEEE Consumer Electronics Magazine - September/October 2019 - 102
IEEE Consumer Electronics Magazine - September/October 2019 - 103
IEEE Consumer Electronics Magazine - September/October 2019 - 104
IEEE Consumer Electronics Magazine - September/October 2019 - Cover3
IEEE Consumer Electronics Magazine - September/October 2019 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20240102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20231112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20221112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20211112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202010
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202009
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202007
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202004
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202003
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202001
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201909
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201907
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201905
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201903
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201901
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201811
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201809
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201807
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201805
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201803
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_october2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_october2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2015
https://www.nxtbookmedia.com