IEEE Consumer Electronics Magazine - September/October 2022 - 68

Interacting with New York City Data
base mapisusedtoprovide a background of the
geographic context of our city model. We provide
three types of base maps including topography
base maps [see Figure 3(a)], imagery base maps
[see Figure 3(b)], and streets base maps [see
Figure 3(c)]. The topographic base map provides
a detailed and accurate representation of the elements
on the surface of the earth. An imagery base
map includes satellite imagery for the world, along
with high-resolution aerial imagery for many
areas, whereas a streets basemap shows road and
transit network details that are legible and accurate.
The " Info Layers " section provides geographical
information that can be displayed as tiles on
the base map. We divide all the info layers into
three aspects: environment, infrastructure, and
people. Users have a variety of options to choose
from in each aspect. For example, we provide the
industrial area [see Figure 3(d)], population density
[see Figure 3(e)] , and poverty distribution in
New York City in the " People " related layers and
subways [see Figure 3(f)], buildings, and transit
frequency situation in the " Infrastructure " related
layers. In our " Sightseeing " section, a few of wellknown
attractions are displayed as options, such
asTimes Square [see Figure 3(g)], Statue of Liberty
[see Figure 3(h)], BrooklynBridge [see Figure 3(i)],
and Central Park. When the base map with info
layers or one of the attractions is selected, users
will be able to see the detailed views.
QoE Metric Design
Modeling the QoE of XR systems has been
extensively researched.8 However, these QoE
models are primarily designed for evaluating
performances where frames are locally rendered
by XR devices. In remote rendering XR systems,
users are highly sensitive to latency, due to the
delay introduced by video encoding, network
delay, and video decoding, which are necessary
steps for real-time rendered video streaming
from the server. Unfortunately, few research
studies have been conducted on QoE metrics for
remotely rendered XR systems. In this article,
we propose a time window-based QoE model for
this purpose, which is defined as
QoE ¼
XN
n¼1
qðFnÞþ
XN
n¼1
pðRnÞ u
XN
n¼1
gðLnÞ
(1)
where qðFnÞ represents the level of a user's satisfaction
with the smoothness of the rendered
video, where Fn donates the average frame rate
at time window n. The level of satisfaction with
video quality is indicated by pðRnÞ, where Rn
represents the average resolution at time window
n. gðLnÞ is used to penalize the turnaround
latency between sending pose data from XR devices
to the remote server and displaying the
video frame for that pose data on the display,
where Ln donates the average latency at time
window n. Because the marginal improvement in
perceived quality decreases with higher frame
rates and resolutions,11 we used two logarithmic
functions to represent qðFnÞ and pðRnÞ, where
qðFnÞ¼ logðFn=FminÞ and pðRnÞ¼ logðRn=RminÞ.
Fmin and Rmin are the minimum values of the
frame rates and resolutions. Conversely, a user's
satisfaction significantly decreases more as the
total latency increases. Therefore, we used an
exponential function to denote gðLnÞ, i.e.,
gðLnÞ¼ eLn=Lmin and u is the latency penalty
factor.
Remote Rendering versus Local Rendering
We evaluated our proposed remote rendering
enabled XR system by comparing its performances
with local rendering by the HoloLens 2 itself.
For both cases (remote rendering and local rendering),
the system is required to achieve 60 FPS
with a resolution of 2048x1080 and a latency of
approximately 60 milliseconds (ms) to provide a
satisfactory experience.12 In our experiments,
we compared the performances of the systems
in both cases using the abovementioned parameters
as the target frame rate and the target resolution.
We also evaluated the case when the
target resolution is 1024x540 to further investigate
their performances in high-quality resolutions
and low-quality resolutions, respectively.
Four scenarios are tested and compared in
terms of frame rate, latency, and QoE (as shown in
Figure 5), including the remotely rendered highquality
videos (denoted as Remote_high), the
remotely rendered low-quality videos (Remote_low),
the locally rendered high-quality videos
(Local_high), and the locally rendered low-quality
videos (Local_low). We observe that the frame
rates are far below the required level (60 FPS)
when rendering the New York City model locally.
68
IEEE Consumer Electronics Magazine
IEEE Consumer Electronics Magazine - September/October 2022

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - September/October 2022
