Talking Stick - July/August 2017 - 52

2
Methods
and
Measures
When looking to make sense of unexpected results, it can be beneficial to
start with an exploration of the assessment measures used. Often, assessment results are presented with broad
labels or themes. The language may
contain words specific to department
goals or initiatives, yet the data may be
collected using a different language.
When the results do not make sense,
examining the exact measure - how
the question was asked - may provide
more clarity.
Consider, for example, a
survey of student perceptions regarding the broad
topic of campus safety. If the specific
questions used in the assessment
tool refer to a wide variety of topics,
including campus lighting, safety procedures, and feelings of security, the
results may be unexpected as assumptions about what is being measured do
not match the specific questions. In
this case, the language of a survey can
impact results.
Even the structure of a survey can
impact the results. For instance, survey
results may say that 25 percent of residents plan to leave housing because
of the cost. Yet if the survey question
is part of a branch and only asked of
residents who intend to leave, that 25
percent may only represent 1 percent
of the whole respondent group. If the
housing professional expects 1 percent
and sees 25 percent, the result is unex-

talking stiCk

pected but easily explained. Similarly,
broad themes are usually presented
with qualitative assessment results,
and unexpected results may be explained by a misunderstanding of the
theme. Examining specific comments
from respondents may be useful to
ensure a common understanding of
the broad theme.
In addition to looking at the measures, housing professionals should
explore the methods being used, since
unexpected results may come from assumptions about who the participants
are, the timing of the assessment, or
other factors related to the chosen
methods. Imagine a scenario in which
a campus housing official is dismayed
after she initially sees much higher
binge-drinking rates on her campus
than the national averages. However,
digging a little deeper and looking at
the survey's respondent demographics
clarifies the results. In this case, her
respondents are traditional-aged college students; few students are older
than 25 years old. The national data, in
contrast, includes a much larger proportion of students 25 years or older.
So, when she looks at only students
less than 25 years old, the results are
comparable.
In another instance, as one housing professional watched a research
presentation on his campus about
student alcohol use, he was surprised
at the low numbers. Although the
study used breathalyzers rather than
self-report, the data was collected
weekdays between 4 and 7 p.m. when
few people were drinking. Thus, the
numbers were low. Learning more
about the methods helped the professional understand the results.
Unexpected results can sometimes
be clarified by examining the measures
(how something is asked, operationalized, or coded) or methods (how the data
is collected and from whom). But when
these strategies do not help, then housing professionals need to turn to others.

3
Context
Matters
When facing unexpected results,
professionals often benefit from additional contextual information. This can
come in the form of a context defined
by the broader campus or by small
microcosms. For instance, a housing
professional who looked at perceptions of safety across different halls
was surprised to see low scores in one
particular hall, but, as it turns out, the
hall was located near an off-campus
convenience store that was robbed
during the timeframe of the survey.
Several open-ended comments even
mentioned the robbery. In this case,
the context (a nearby robbery) may
have contributed to the results.
In trying to assess how safe
students feel, serious incidents
such as recent deaths or acts of
violence may be of concern. But smaller
issues, like an incident of vandalism, can
also provide context to unexpected findings. A single case of vandalism in one
restroom may seem small, but it may
explain unexpectedly low responses on a
survey of facilities or hall safety. Similarly, responses analyzed by floor may
vary greatly in unforeseen ways based on
differences in how specific student staff
members handled the situation. It is not
unusual to see open-ended responses
that describe these situations and staff
responses.
In another example of the importance of context, a committee was
stunned to find that the response
rates to a yearly survey had dropped

Talking Stick - July/August 2017

Table of Contents for the Digital Edition of Talking Stick - July/August 2017