IEEE Technology and Society Magazine - March 2022 - 36

Feature
a spatial audio rendering, with the aim of giving
the impression of a person being co-present in the
same space with the user.
Voxelvideo storytelling
The passing on of knowledge (ma¯tauranga) and
customs and protocols (tikanga) is traditionally done
by way of oral storytelling and is often supported by
something physical including whakairo (carving)
and tukutuku (lattice-work decoration) on the walls
which also have their own ko¯rero (narrative). While
other forms of knowledge sharing and preservation
have been used since early contact, such as written,
audio, and video accounts, the preferred and most
true-to-values approach is relating stories orally faceto-face.
Also, the storyteller's standing in the community,
expressed as mana, is of high importance and
will determine the depth and breadth of the story to
be shared. If true face-to-face storytelling is not possible
or feasible, like in our project scenario, alternative
forms have to be developed and provided.
Writing down and reading (aloud or to one-self)
stories is common practice since the early Pa¯keha¯
settlers arrived in New Zealand. While this form has
the potential to be an accurate representation of the
story to be told, it also requires a deep understanding
of Ma¯ori culture and how this can be translated
into (the Western form of) writing. A prominent and
highly important example is Te Tiriti o Waitangi-a
document settling the relationship between the
British Crown, the settlers, and the indigenous population
of Ma¯ori. It exists in te reo Ma¯ori and a translated
English version where specific interpreted
differences are debated to the present day. Hence,
for our purposes, written stories are a potential but
probably suboptimal option-we opt for a spoken
account of Ma¯ori storytelling over written stories.
Sound recordings of stories are one way to provide
rich potential for expression and adhere to the original
(oral) language; they can be easily recorded and
played back and have the ability to display the character
and mana of the speaker or storyteller. Their main
shortcoming is the lack of nonverbal expression, such
as gestures which normally accompany the story.
They also lack the integration of props, artifacts, and
the natural and built environment into the story. For
instance, when referring to carved or woven elements
of a wharerau wall, one cannot simply point at those
artifacts, but has to verbally describe them.
36
Videos are even more media-rich and have the
potential to not only present the words of the story in
combination with seeing the storyteller, but can also
depict parts of the environment, like the sky, details
of buildings, such as the interior of a wharerau, and
other objects and people. The main shortcoming
here is the lack of interactivity, in particular, the story
receiver's ability to explore the scene, as in real life,
and to interact with the storyteller in real-time and in
a meaningful way.
The use of recently researched and selectively
available volumetric video technology is addressing
some of the above shortcomings. These videos are
3-D representations of a scene and storyteller, usually
recorded with an array of specialized cameras and
experienced with head-worn displays. The viewer
can physically and virtually walk around within the
scene and can take any viewing angle they want.
Usually, the recording of volumetric videos is very
time- and labor-intensive and requires specialized
recording studios [15]. For instance, the Volucap
system [12], one of the most advanced volumetric
video capturing systems producing very high-quality
results, requires a studio environment with 16 carefully
calibrated stereo camera pairs for recording a
3 m × 3 m scene, where each minute of recording
requires hundreds of hours of post-production time
and by specially trained technicians.
We are using an alternative approach: voxelvideos.
Here, the volumetric video is recorded with
voxels (volumetric pixels) of rather coarse resolution,
using only three off-the-shelf RGB-D (color and
depth) cameras. The system can be used in a studio
setup with controlled lighting and so on, or in the
field using tripods, for example, inside the wharerau
on a marae. The achievable quality is significantly
inferior to approaches like the ones presented by
Schreer et al. [12] or Awanuia¯rangi [15], but there
is no significant post-processing time required, the
system can be operated by trained laypeople, and it
is a very affordable technique, making it ideal for our
partnership model enabling Ma¯ori to record stories
for Ma¯ori.
We have recorded about a dozen ko¯rero (stories)
with our voxelvideo system, including " Te Waka
o Tama Rereti, " a ko¯rero explaining how the stars
ended up in the sky, along with each of the eight
stories behind the walls in the wharerau. Viewers of
these stories experience them in a life-like and interactive
way. For example, we have received reports of
IEEE Technology and Society Magazine
IEEE Technology and Society Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2022
