Hard to tell if Universities teach: The challenge of low-stakes testing

What a great idea!  Everybody who goes to University takes a test like the ACT or SAT.  Simply give it to them again as they’re graduating! Now you have a measure of impact — the change between the entrance test and exit test is the value added by a University.

Seems simple, but it doesn’t work. Students have a huge incentive to do well on the entrance exam, but zero incentive to do well on the exit exam.   A new study published in Education Researcher shows that the motivation really matters, and it calls into question the value of the Academically Adrift study that claimed that Colleges aren’t teaching much.  How do you know, if students don’t really have any incentive to do well on the post-intervention exams?

To test the impact of motivation, the researchers randomly assigned students to groups that received different consent forms. One group of students received a consent form that indicated that their scores could be linked to them and (in theory) help them. “[Y]our test scores may be released to faculty in your college or to potential employers to evaluate your academic ability.” The researchers referred to those in this group as having received the “personal condition.” After the students took the test, and a survey, they were debriefed and told the truth, which was that their scores would be shared only with the research team.

The study found that those with a personal motivation did “significantly and consistently” better than other students — and reported in surveys a much higher level of motivation to take the test seriously. Likewise, these student groups with a personal stake in the tests showed higher gains in the test — such that if their collective scores were being used to evaluate learning at their college, the institution would have looked like it was teaching more effectively.

via Study raises questions about common tools to assess learning in college | Inside Higher Ed.