Hard to tell if Universities teach: The challenge of low-stakes testing
January 8, 2013 at 7:46 am 15 comments
What a great idea! Everybody who goes to University takes a test like the ACT or SAT. Simply give it to them again as they’re graduating! Now you have a measure of impact — the change between the entrance test and exit test is the value added by a University.
Seems simple, but it doesn’t work. Students have a huge incentive to do well on the entrance exam, but zero incentive to do well on the exit exam. A new study published in Education Researcher shows that the motivation really matters, and it calls into question the value of the Academically Adrift study that claimed that Colleges aren’t teaching much. How do you know, if students don’t really have any incentive to do well on the post-intervention exams?
To test the impact of motivation, the researchers randomly assigned students to groups that received different consent forms. One group of students received a consent form that indicated that their scores could be linked to them and (in theory) help them. “[Y]our test scores may be released to faculty in your college or to potential employers to evaluate your academic ability.” The researchers referred to those in this group as having received the “personal condition.” After the students took the test, and a survey, they were debriefed and told the truth, which was that their scores would be shared only with the research team.
The study found that those with a personal motivation did “significantly and consistently” better than other students — and reported in surveys a much higher level of motivation to take the test seriously. Likewise, these student groups with a personal stake in the tests showed higher gains in the test — such that if their collective scores were being used to evaluate learning at their college, the institution would have looked like it was teaching more effectively.
via Study raises questions about common tools to assess learning in college | Inside Higher Ed.
Entry filed under: Uncategorized. Tags: computing education research, perception of university, teaching.
Computing Ed Research - Guzdial's Take 
1.
Bijan Parsia | January 8, 2013 at 8:48 am
Hmm. That study seems a it dodgy, ethically. They lied *on the consent* form in a way that could induce considerable anxiety in students. And it’s unnecessary: You could give incoming frosh an exam to compare against. Indeed, that’s better because it isolates the “in exam” motivation effects from the “prep” motivation effects.
Then that low stakes exam could be used against an end year low stakes exam.
It also seems that someone clever could look at post graduate entrance exams in comparison.
2.
Mylene | January 8, 2013 at 6:22 pm
I thought the Academically Adrift authors had in fact administered both the pre and post-test as “low-stakes”. Can anyone confirm? I suppose that that doesn’t really solve the problem. Although keeping the stakes consistent is less bad as a research design than comparing a high-stakes “pre” to a low-stakes “post”, it’s not necessarily clear that an increase in maximum capacity for reasoning would show up — perhaps there is a “default” level of everyday reasoning, used when people are indifferent about the results, that doesn’t change much?
Deborah Kelemen reports possibly related results in testing the frequency with which scientists accepted teleological statements. Although they did so much less frequently than a control group, their frequency doubled when they were rushed. John Burk posted an interesting discussion.
I can’t help thinking that a low-stakes test-taker would be likely to want to get through the test as quickly as possible, even if the test itself were not timed.
3.
alanone1 | January 8, 2013 at 8:59 am
In the 50s (and I don’t know how long thereafter), there was a national test everyone was supposed to take in their junior or senior year of college called the GRE (Graduate Record Exam). This provided a “somewhat measure” of what kind of thinking and knowledge had been attained. To my memory, there was quite a bit of motivation to do well on this test because it was looked at by both prospective employers and grad schools.
The first college I attended had a much stricter process, called “comps”, which was a week of comprehensive testing in your major and minor in your senior year, and that you had to pass in order to graduate. Comps were essentially like a final exam for each of the required courses over the four years but presented all at once some years after the courses had been taken. They required a lot of work! The plus side was that — once through the gauntlet — one gained a synthesis and perspective of the subjects that was very rich.
Cheers,
Alan
4.
dennisfrailey | January 8, 2013 at 9:02 am
Correction: once through the gantlet! A gauntlet is a glove.
5.
alanone1 | January 8, 2013 at 9:23 am
Here’s a good example of my research community putting in a lot of work to, among other things, allow anyone to instantly check most opinions with a few clicks — yet people persist in living inside their heads.
People who really like words will know that “gantlet” was an original form, but the accepted and most used word for many years is “gauntlet”.
To make it easier for you to check this, here is the link:
http://en.wikipedia.org/wiki/Gauntlet_(punishment)
A more important link to check out is the one on “relevance”:
http://en.wikipedia.org/wiki/Relevance
6.
Dennis J Frailey | January 8, 2013 at 5:51 pm
Hmm. Wikipedia notes: Robert Hartwell “The Grumbling Grammarian” Fiske asserts in The Dictionary of Disagreeable English that the word “gantlet” (the form of punishment) has been incorrectly conflated with “gauntlet” (the medieval glove covered with metal plates) and should be used separately.
OK, I’m a curmudgeon. Call me “Henry Higgins”. But I agree with Fiske. I still like to use correct English, not the bastardized English that is slowly gaining acceptance. The reason that “gauntlet” is now accepted as an alternative to “gantlet” (by some but not all dictionaries) is that it was been misused (conflated) for so long.
7.
dennisfrailey | January 8, 2013 at 9:00 am
These tests (SAT, ACT) are supposed to measure aptitude, which in theory should not change as a result of education. So students should not be expected to change their scores. College potentially gives them knowledge and experience but that’s not the same as aptitude.
8.
Bijan Parsia | January 8, 2013 at 9:59 am
I don’t think that’s right. Aptitude may be fixed (by definition, but meh), but your ability to perform a given task isn’t merely a function of aptitude. I might have great aptitude for learning languages, but give me a test on the vocabulary of a language I’ve never heard of and even a poor language aptitude student will crush me.
So, you might not expect a difference between subjects before and after college, but you should expect a within subject difference if college is adding value.
9.
Dennis J Frailey | January 8, 2013 at 5:54 pm
The “A” in SAT stands for aptitude, not knowledge or skill or experience.
10.
Bijan Parsia | January 8, 2013 at 6:02 pm
And that’s irrelevant to my point.
The SAT doesn’t have a magic aptitudoscope that can directly measure aptitude.
Just as failure to perform a certain task doesn’t determine your aptitude for such tasks per se, neither does success. This is basic.
(And people do improve their SAT scores! By studying! OMG!)
11.
Fred Martin (@fgmart) | January 8, 2013 at 6:58 pm
SAT no longer stands for anything.
At first it was Aptitude (’cause it was supposedly measuring something fixed), then it became Achievement (’cause people were obviously studying and doing better), then finally they threw in the towel.
http://www.time.com/time/magazine/article/0,9171,136829,00.html
SAT measures SES 🙂
12.
dennisfrailey | January 8, 2013 at 9:29 am
Does studying, listening to or performing music increase one’s knowledge of and skill at performing music? Presumably yes, but the level of improvement varies with the individual. And talent makes a huge difference with many musical skills. Some musicians are successful with no formal training whereas others benefit from it. (And some “musicians” are successful despite lack of significant musical skill or talent.) The more complex and difficult the music (classical vs rap, for example) the more value in formal musical education.
Similar comments can be made about the merits of going to the gym for building up one’s physical fitness.
Perhaps a college education is similar. It makes a huge difference with some, especially those with appropriate talent, and a marginal difference with others. Fields like engineering or science or medicine are complex enough that formal training is pretty well mandatory for most (hence things like accreditation in these fields) whereas in other fields college may educate primarily for personal edification.
People go to college for different reasons. For some it may simply be a way to avoid “real work” for a few years. For others it may be a way to pick up a credential that will gain them entry to some later opportunities. College provides the opportunity to learn but it cannot force people to learn. So if a student doesn’t learn, we ought to put blame on the student (for not taking advantage of the oppoprtunity) and the coillege’s admissions process (for not filtering out a poor prospect).
Perhaps one of the best evaluators of the effect is the individual student. Perhaps another is the employer, although having recruited and employed people for over 40 years I’d say it’s 99% the person and at most 1% the college they went to that matters (in the US – because most of our colleges are at least fairly good — this is not necessarily true for other countries).
13.
Alfred Thompson | January 8, 2013 at 9:57 am
I completely agree. I worry that many big companies miss out of some great hires because they didn’t go to big name schools. The same is true of some graduate schools as well. But recruiting from the big name schools is an easy and cheaper way to do things for many organizations.
14.
Alfred Thompson | January 8, 2013 at 9:55 am
The first year I taught APCS my students decided the exam itself was meaningless to them and they all decided to “blow it off.” Of course this resulted in some very poor test scores. The students themselves had learned a lot and several went on to very successful college and professional careers in CS. Several of them went on to help peers in college who had 4s and 5s on the exam but still struggled in college.
But the exam results almost killed the APCS course at the school and probably would have if they next year’s class hadn’t decided to take the exam more seriously. This experience makes me look at exams that are high stakes for teachers but low stakes for students somewhat skeptically.
15.
Cecily | January 8, 2013 at 10:48 am
Alfred- I love your story about AP CS because I had similar experiences teaching programming at the high school level; my students simply had no interest in taking the AP test because they were so far beyond it.
It seems to me that there are a few fundamental flaws with the “low stakes exit exam” concept. First, most of what I teach in my classes has very little to do with what is on the ACT. Giving the ACT to my students would be like giving a carpentry exam to piano students- the exam needs to match what in theory the students are supposed to be learning. Second, students have varying degrees of motivation depending on what opportunities are available to them. The “hot ticket” here seems to be an opportunity to work in the new Adobe building at point of the mountain in Lehi. Adobe bought out Omniture Analytics and several of my students have expressed interest working there. We also have several that go on to work in mobile apps and web development. We have very little opportunity in the way of computer organization, design, or architecture in Utah, and motivating even the best students is a major challenge in those courses. We do have an exit exam that is a national exit exam (from ETS I think) that we require our students to pass to graduate, but I am not sure we get any meaningful feedback from it as far as how to improve the program. I think it would be fabulous if there were more standardized exams that we used as finals in our individual courses that graduate schools and employers could use to compare our students.