Talking Stick - January 2012 - 34

to observation and document analysis, consider which would be more appropriate given the circumstances of the program or intervention. Thinking about resources (such as staff and time) can help prioritize or determine how evidence should best be gathered. Literature and knowledge specific to a given purpose (e.g., student development theory, leadership models) is necessary to identify the dimensions and content to include in a rubric. Be sure to keep past experiences in mind, too. Piatkowski was surprised at how easy the process could be once staff began conversations to “look at gaps or voids to take [the] ambiguity of a community and make it tangible to measure as dimensions.” Brown underscores the importance of “sticking to it in a collaborative effort to make sure everything you’re looking for to be expressed and evaluated is actually in the final product” but warns that it can be easy to lose consistency in phrasing across dimensions or writing to advance descriptions through scale points. As a final consideration for the process, Severy advises everyone to “give yourself more time than you think you need.” Metarubrics can be used to review instruments, as well as peers or students. Severy encourages staff to “dry run it before you live run it.” Brown saw the testing or piloting phase as being extremely important,

as it allowed them to get feedback and improve the instrument, making it more specific and comprehensive before it was used to collect data. Albart reinforces the value of testing, saying that “if you don’t test the rubric, inevitably you’ll find things as you evaluate that are not working for you.” Moreover, Albart believes that “when you try to throw a rubric together and use it immediately, it will work but not be exactly what you want.” Consistency in rubric results is dependent on the consistency of the data input. Therefore, it is important to identify raters and hold trainings to educate them on the content and how to use the instrument. Those involved in the assessment should practice using the rubric and should gather feedback from raters not only to improve the instrument, but also to help insure reliability with raters. Albart believes that training is the most important part of the process. After all, she says, having many people involved in the process is only helpful if they understand what the descriptions and dimensions mean. Once consistency is established, it’s important to be strategic in the administration and collection of data. Consider the timing of data collection – both timing in relation to other projects, as well as the time required to collect all the data. Where applicable, consider using the instrument for multiple audiences (e.g., performance evaluations collecting self-, peer-, and supervisor-reported data). Severy encourages taking notes throughout the process to indicate if the evaluation or assessment captures what was intended, as well as other elements to consider for future instrument review. Finally, compiling aggregate data will allow the examination of trends and overall learning that occurred. To provide individual feedback, drill down to individual data. Consider additional statistical analysis if necessary. Prepare data to be shared and reported to various audiences and in various formats. Severy enjoys reviewing
conTInues 49

Talking STick

Talking Stick - January 2012

Table of Contents for the Digital Edition of Talking Stick - January 2012