Archive for October 11, 2011

NYTimes takes on Cognitive Tutors: What can we really prove with studies?

I looked up the Department of Education’s report on Carnegie Learning, and the NYTimes article was actually kinder than it might have been. Three of the four papers that were reviewed actually showed that the cognitive tutor had a negative impact on the outcome measure! Their standards for what studies counted and which didn’t were a little odd to me. Like this comment from the paper:

Plano, G. S., Ramey, M., & Achilles, C. M. (2005). Implications for
student learning using a technology-based algebra program
in a ninth-grade algebra course. Unpublished manuscript. The
study is ineligible for review because it does not use a sample
aligned with the protocol—the sample is not within the specified age or grade range.

What does that mean? The ninth graders being studied might not have actually been in ninth grade? Or somebody should have checked?

I wonder whether either the “hype” of the salespeople or the whole “What Works Clearinghouse” make any sense at all. I raised this question in my piece for Greg’s Making Software. We have studies where Media Computation has worked well in terms of impacting student retention. So? That shows that it can work. That is no guarantee that it will work. The WWC says, “Let’s use randomized trials, of both students and teachers.” I’d make the same claim. Even the greatest teacher can be stymied by a class of poor and starving students, and even the greatest textbook can be completely ineffective with an unprepared teacher and unmotivated students. Is it possible to prove that any intervention will always work?

The federal review of Carnegie Learning’s flagship software, Cognitive Tutor, said the program had “no discernible effects” on the standardized test scores of high school students. A separate 2009 federal look at 10 major software products for teaching algebra as well as elementary and middle school math and reading found that nine of them, including Cognitive Tutor, “did not have statistically significant effects on test scores.”

via A Classroom Software Boom, but Mixed Results Despite the Hype –

October 11, 2011 at 9:45 am 5 comments

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,184 other subscribers


Recent Posts

Blog Stats

  • 2,049,107 hits
October 2011

CS Teaching Tips