This is an excerpt from a conference paper by Courtney Castle of the University of Kansas, which offers wonderful background and illustrations of Using Design Thinking to Create Human-Centered Assessments with an adapted Innovators’ Compass.
Why Use a Human-Centered Approach?
Ask any group of people how they feel about taking tests and you will receive a range of responses. Some will report that they relish the challenge, while others are filled with dread. There are many reasons that a person might have bad feelings about assessment such as test anxiety, lack of preparation, or feeling crunched for time. Whatever the reason, many people feel that traditional assessments don’t accurately reflect their skills and abilities. Psychometricians and assessment developers conduct their work far away from real test-takers, and therefore rarely contend with the impacts of testing on examinees. In contrast, when you ask teachers and students for their opinions about assessment, they give a much closer view of the impact of assessment. Here’s what they say:
“In high school, assessment was a negative spiral where I would be overthinking things so much that I would lose track of what was actually being asked of me, and that was really debilitating.”
“Now, as a teacher, dealing with people who need a little more practice, or it doesn’t come as quickly, I feel bad because assessment is negatively impacting them and making them feel like they can not do it.”
“80% b***s***, 20% useful. There is just too much going on with a traditional test to assess what the student really knows.”
In recent discourse, good assessment development is characterized as a process that promotes validity. To promote validity, assessment development begins by centering the construct to be measured and the use of the results. Psychometricians will often consult subject-matter experts to better understand the construct. They will also consult with the people who will be using the test results, such as teachers, administrators, or policy makers. Notably absent from these groups: the test-taker. By failing to know and understand test-takers, test developers fail to consider the human emotions, experiences, and situations that surround the testing process. To the extent that they consider the test-takers' experience, they refer to it as construct-irrelevant variance: a psychological or situational factor that regrettably (perhaps unavoidably) presents an obstacle to valid measurement of the construct (Messick 1984; Haladyna & Downing, 2004). This is a failure of both empathy and common sense; we know intuitively (from our own lives as humans) that the experience of taking a test involves so much more than the construct.
The experience of taking a test can bring up a lot of emotional baggage from the high stakes (or perceived high stakes), prior experiences, and situational and environmental challenges. These factors are not irrelevant; they can significantly alter a test-taker's performance. To overcome these obstacles, assessment development should consider the experiences of test-takers as a relevant and foundational consideration.
(Section skipped for blog brevity)
Using the Innovator’s Compass helps a psychometrician or assessment developer to understand the “problem” of assessment development in a new way.
1. Who’s involved? Test-takers, people who use test results to make decisions (teachers, administrators, policymakers), subject matter experts, psychometricians, etc. Whose voices are already heard, and who is left out?
2. What biases and assumptions do you have? How have I (the designer) experienced assessment? Was I a good test-taker or a bad test-taker myself? How has assessment benefitted and/or harmed me?
3. What’s happening? Why? Why do the test-takers have to take the test? How does the test affect people emotionally, financially, and logistically? The observations quadrant can also help understand the construct better: What’s already known about this construct? How is it typically measured? What are the benefits and challenges to measuring it this way? What other sources of evidence are there? What kinds of behaviors and actions are typically associated with the construct?
4. What matters most? Validity, making accurate decisions with assessment results, respecting time and resources, providing opportunities, fairness, feelings of safety and empowerment, engagement, relevance. (These are just a few of many possible principles, and some principles may contradict each other.)
5. What ways are there? Typical multiple-choice and constructed-response items. Innovative options such as verbal exams, creative products, and real-world projects.
6. What’s a step to try? Talk to a test-taker about their past experiences. Conduct a cognitive lab. Sketch a prototype. Research an innovative method.
The compass is an iterative process: after every experiment, continue to make observations by gathering test-taker feedback on their experience and examining statistical measures of item performance. Based on observations, refine principles, generate new ideas and experiments, and continue improving assessments. Using the compass in this way encourages the development of assessments that are equitable, engaging, and effective for all learners.
(Continues with illustrated examples in the paper).
