Training Industry Magazine - Summer 2023 - 42

the same position. This was important
because it immediately indicated
(right after the training) if the content
was on target or had to be revised.
LEVEL
Level 2 measures participants' new
knowledge and skills that were acquired
during the training. Are they leaving with
new learning and capabilities that will
help them do their jobs better? This may
also include an attitude change or a new
way of looking at a problem. While we
typically conduct these assessments with
business clients by asking a few simple
questions about the new knowledge
they gained, it was especially important
for these government agencies to
understand the real difference between
where employees were before and after
the training.
To accomplish this without a rigorous
pre- and post-test at Time 1 and Time
2 for all training sessions, we used a
retrospective scale that carved out
each unique knowledge topic and
then asked participants, within the
same question, to rate where their
level of knowledge was before and
after the training. Thus, the difference
or percent increase in knowledge for
each session could be quantified using
only a post-training assessment.
Level 3 measures whether participants
are returning to their everyday jobs and
actually applying what they learned in
training. Level 3 is probably the most critical
measure because if behavior change
doesn't happen, then all the training
impact stops there. Conversely, if you can
demonstrate that your training improved
the way employees do something, these
improvements can then be correlated to
the critical metrics that matter most to the
organization. And as important as it is to
a business impact story, it's even more
critical to a government impact story
because these " mission-critical " behaviors
have historically defined the success or
failure of their training initiatives.
To measure this level for NASA and
the DOI, we asked questions about
improvement in crucial behaviors 6090
days post-training and asked raters
to use a scale from " no improvement "
to " exceptional improvement. "
We were then able to report the
percentage of participants, and other
42
1
2
3
4
5
6
MEASURES
Satisfaction: Did they like it?
Learning: Did they learn anything?
On-The-Job Improvements: Did they
do something differently or better?
Business Impact: Did it impact
business performance?
ROI: Was it worth it?
Transfer Climate: What factors
maximize the impact of the training?
rater groups, that observed each
level of improvement back on the job.
This level of measurement is crucial
because it shows that the learning at
Level 2 was transferred and applied on
the job at Level 3.
Level 4 typically measures whether
these behavioral improvements
actually lead to better business metrics
and higher performance. Here is
where we had to make a significant
divergence from business impact and
focus on organizational efficiencies
that could increase organizational
health by saving time, resources and/
or budget. To do this, we had to look at
the overarching objectives of the entire
agency, look at the mission and vision
of the agency and look for how the
behaviors taught in training specifically
fed into those larger objectives.
For instance, instead of asking about
how satisfied paying customers were
with a business service, we asked
questions about how the DOI was
able to make quicker decisions and
resolve federal land disputes, and
how NASA was able to create internal
efficiencies where participants could
have more productive and inclusive
hybrid
meetings,
spend
less
time
in unproductive meetings and thus
spend more time sending people into
the stratosphere.
Level 5 measures the extent to which
the benefits of a particular training
experience outweigh the costs of
that training experience. The ROI
calculation can only be done when
the benefits identified in Level 4 have
been monetized and turned into
dollar values for the organization.
Once monetized, they can then be
compared to the total costs, which are
already monetized.
How could we possibly do this for
agencies that aren't looking for
financial ROI, but still want to know
whether the training was " worth it " ?
Here we proposed translating at least
some of the efficiencies and outcomes
we found at Level 4 into a quantifiable
dollar benefit. For example, if issues are
being resolved quicker, or employees
are having more productive meetings,
then we can quantify the " hours saved "
into salary dollars saved.
Level 6 is an evaluation that simply
measures your learning climate and
Training Industry Magazine - Summer 2023

Table of Contents for the Digital Edition of Training Industry Magazine - Summer 2023
