The Data and Measurement Issue - 9

Digging In
Staying Grounded
To be full partners in evaluation, the core team spent time building shared values
and a common language.
We conducted research and interviews with key stakeholders, instructors, and
recent learners to better understand the needs of shovel supervisors and the
goals for the course. Digging in further, the team engaged in a series of six
two-hour collaborative workshops to develop foundational skills in the design
of metrics, studies, surveys, and scenarios and to apply these skills to our
project context.
We then developed appropriate metrics and measuring tools, using the LearningTransfer
Evaluation Model (LTEM) developed by Will Thalheimer to guide our work.
LTEM helps teams make decisions about the kinds of data to collect for a
multidimensional picture of a learning program's effectiveness. For instance, to
measure job performance, we identified key behavioral indicators (KBIs) that would
show if course objectives were really being applied on the job.
We planned a time-series study, in which we would compare the data before
training and at multiple points after the training, to assess the impact the training
had on the learners.
What We Wanted to Measure
Job performance
Task competence
Decision-making skills
Learner perceptions of performance
impact
How We Measured It
Anonymous self-evaluations
Capability checklists
Scenario questions
Performance-focused learner surveys
Learning evaluation projects comes in many shapes and sizes. Talk to TiER1 about
your goals, and we can help design the right solution for your team.
Developing the evaluation instruments was an iterative
process, involving multiple stages of team discussion,
review, and revision. For example, we validated scenario
questions with both experts and non-experts. Listening
to experts " think aloud " through their answers helped
us ensure that the questions would assess decisionmaking
based on the intended learning points. Testing
with non-experts helped us identify and remove wording
that could make the answers easier to guess.
Research-based evaluation methods such as LTEM help us leverage learning
and performance experiences of all kinds to uncover insights and achieve our
goals. This project illustrates how collaboration and learning science can
combine for a solution that delivers high-performance, develops teams, and
drives business results.
The project did face challenges, but the team persevered in excavating insights
and expanding their learning evaluation capability.
We ran into obstacles with implementing the time-series study as envisioned,
as the number of learners entering the program dropped significantly right
before our planned start for the study. Since our sample size was too small to
perform meaningful statistical analysis, we ultimately derived the most reliable
insights from research-inspired benchmarking based on our initial interviews
and observations.
Despite this setback, we still achieved several important outcomes:
* We identified strengths and opportunities for improvement in the shovel
supervisor training program.
* We developed evaluation tools that can be used in the future for additional
mine training programs.
* We informed the learning evaluation strategy for programs at the company.
* We supported team members in gaining skill and confidence in evaluating
learning programs, which they were able to transfer to other contexts.
Fueling Discovery
PERFORMANCE MATTERS / PG 9
The Data and Measurement Issue

Table of Contents for the Digital Edition of The Data and Measurement Issue
