IEEE Systems, Man and Cybernetics Magazine - April 2023 - 31

" Speaking to the subject/addressing the subject, "
labeled as " 1. " Speaking to the robot is a question forward-directed
to the robot or answer forward from the
subject when the robot asks a question. Speaking to the
other subject can be identified during conversations
between subjects standing in front of the robot. A
human annotator identifies these two activities manually
after observing and listening to the conversation. The
human annotators used the guidance illustrated in Table 2 to
distinguish speech activities from other speech-like
activities. The sample .xlsx annotated file is shown in
Table 3. Based on Figure 3, the table illustrates that the
annotated face subject ID 12 is talking to the robot. During
the annotation, the faces are only annotated when
there are speech activities or someone speaks to the
robot or another person. There is no annotation when
there is no speech activity (no highlighted waveform
area shown in Figure 3) or if the robot is speaking. After
the annotation, the accuracy timestamps were validated
by other nonannotators by playing each segment to
check if the sound was wrongly trimmed or oversegmented
(consisting of blank sound).
As shown in Figure 4, the E-MuMMER dataset consists
of a total of 6,193 speech segments. The speech segments'
length statistics are shown in Figure 4. The
dataset consists of short and long speech segments.
Most speech segments are between 15 and 60 frames or
1-2 s. The shortest segments in the dataset consist of 1
frame size (0.067-s duration), and the longest segment is
520 frames (34.67-s duration). A total of 24,131 human
faces are annotated as speaking to the robot, and 7,752
are identified as speaking to the subjects. We split the
dataset randomly: 60% for training, 20% for validation,
and 20% for testing.
Evaluation Metrics
To measure the prediction accuracy, we compare the
prediction response with the ground truth and compute
the accuracy of the proposed model in terms described
as follows.
Precision, Recall/True Positive Rate, and True Negative Rate
If TP, FP, and FN denote true positive, false positive,
and false negative, respectively, the formal definitions
of precision, recall/TP rate (TPR), and true negative rate
(TNR) are [36]
Precision TP FP
TP
=
+
Table 2. Guidance to identify speaking
instances and nonspeaking activities.
Speaking to Robot/Subject
* Any long-utterance communication
(e.g., asking the robot a question)
* Short utterances (e.g., " Yes, " " No, "
" Go, " " Hmm, " " Hi, " etc.)
* Vocalized communication intent
(e.g., screaming to attract attention)
* Fillers (e.g., " um " and " ah " )
Table 3. A single-row illustration of the
format of the .xlsx file.
Bbox
Frame ID
[275, 157, 83, 64] 15; ...; 6,636
Bbox: bounding box [x, y, w, h].
Subject
ID
12
Timestamp
(s)
28.2
Label
Nonspeaking Activities
* Sighs, coughs, laughs,
groans, and grunts
* Nonspoken communication
(e.g., gesturing
and waving)
* Humming
(1)
1,400
1,200
1,000
800
600
ts0 te0
ts1
te2
tsn
ten
Figure 3. The annotation interface for E-MuMMER.
Each face is annotated, whether speaking to the
robot or another person, based on the speech
activity in each time segment. Each speech
segment's starting and ending times were picked
manually, and the highlighted waveform is shown
below the window.
400
200
Speech Segment Frame Size
Figure 4. The E-MuMMER dataset annotated speech
segment statistics based on the frame size.
April 2023 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE 31
Number of
Speech Segments
[1, 5]
(90, 105]
(75, 90]
(60, 75]
(45, 60]
(30, 45]
(15, 30]
(10, 15]
(5, 10]
(105, 120]
(135, 150]
(120, 135]
>150
IEEE Systems, Man and Cybernetics Magazine - April 2023

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - April 2023
