IEEE Consumer Electronics Magazine - July 2018 - 19

Unity 3D. There are three specific steps of this movement data
binding: 1) the dance movement capture, 2) the encoding of
captured data, and 3) the binding of movement data to the virtual model. The motion-capture step can make the captured
movements bound in the character more accurate and natural.
Each tracking skeleton point of the captured motion is mapped
to the key point of the virtual character model accordingly to
make the model's skeleton driven by the captured motion data.
After movement binding and debugging, we import the established role model to the software and bind the motion data to
the virtual actor to drive the dance character animation. The
result of action binding is shown in Figure 4.

USER-TO-CHOREOGRAPHER
ALIGNMENT AND COMPARISON
It has been mentioned that the device for motion capture may
be different between the choreographers and users. The coordinate system of the skeletons needs to be aligned. The method of
coordinate system alignment is the use of the T pose. The
system can recognize the T poses of the choreographer and user
and align them, because the T pose can determine the x and y
axis. If the T-pose skeletons can be aligned, the coordinates of
the choreographer and user will be aligned too. After alignment
of the coordinate system, the movement will be aligned with
dynamic time warping to avoid error caused by a time lag.
Cosine eigenvalue and weighted Euclidean distance are
the two main methods for user-with-choreographer dance
movement comparison. Kinect can be used to capture the
motion data of users to control the virtual character in the
scene. The animator element of the instructor has been combined with the standard movement data of the professional
hand-waving dancer. Considering that the user's movements
evaluation is carried out through the comparison of the skeletons of the user and the instructor, it is necessary to get the
skeleton information for them. We can get the skeleton vectors information of the virtual user and instructor model and
calculate the differences of the bone eigenvector matrix. By
means of the human key points recording function of Kinect,
we built 19 bone vectors with the 20 key points of the human
skeleton by analyzing the bone rotation movement characteristics and structure of the human body skeleton. The structure
of the 19 bone vectors of the human skeleton is shown in Figure 5. Because the key skeleton vectors of the user and
instructor can be different, even if the user is performing the
same dance moves as the instructor, their height and weight
may be different. The dance movement represented by the
eigenvalues of direction cosine of the 19 skeleton vectors can
be used to remove the complexity of height and weight, i.e.,
the direction of each bone vector is taken to evaluate the
dance movement. We then take the skeleton vector head-
shoulder center (numbered 1 as shown in Figure 5) as an
example to analyze the evaluation algorithm. Assuming that
the 3-D coordinates of the key points of head-shoulder center
are ^ x 1, y 1, z 1h and ^ x 2, y 2, z 2h, the skeleton vector can be represented by (x 2 - x 1, y 2 - y 1, z 2 - z 1), the three direction angles
of the skeleton vector are a, b, and c in the Unity world coor-

Recognizing movement is a
complex process that involves
moving target detection, feature
extraction, and motion pattern
recognition.
dinates system, and the calculation formulas of the cosine
values are shown in Figure 6.
After extracting the cosine eigenvalues of the three direction angles (cosa, cos b, cosc) of the 19 skeleton vectors in
the virtual human model, we recorded them in a 19 # 3 matrix
as the eigenmatrix of the character model. Thereafter, the difference of the bone eigenvector matrix of the instructor and
user model is calculated, which can be a reference to the
evaluation of the user's dance movements.

(a)

(b)

(c)
FIGURE 4. The dance movement binding. (a) Photos of the key
movements by professional hand-waving dancer Chengjin Peng,
(b) the skeleton of the virtual instructor, and (c) the result of
movements binding.

july 2018

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - July 2018