Comparing performance in learning computer science between countries
May 6, 2019 at 7:00 am 2 comments
Imagine that you are a high school chemistry teacher, and you’re convinced that you have developed a terrific way to teach the basic introductory to chemistry course. Your students do terrific on all your assessments and go on to success in chemistry in college. You decide that you want to test yourself — are your students really as good as you think they are?
You reach out to some friends in other schools and ask them to give your final exam to their students. You are careful about picking the other schools so that they’re really comparable along dimensions like student wealth, size of school, and student demographics. Your friends are willing, but they just have a few of their students take the test. You don’t know really how they pick. Maybe it’s the best students. Maybe it’s the students who need remedial help. Maybe it’s a punishment for students in detention. Of course, all of your students take the final exam.
In the end, you have lots of YOUR students who took YOUR exam, and you have a handful of other students. Your friends (who likely don’t teach like you) give you a few tests from their students. Is it at all surprising that your students will likely out-score the friends’ students?
That’s how I read this paper from Proceedings of the National Academy of Sciences of the US: “Computer science skills across China, India, Russia, and the United States.” The authors are quite careful about picking schools to compare, along dimensions of how “elite” the schools are. I’m quite willing to believe that there is a range of schools with different results along an “elite” spectrum.
They over-sample from the United States, compared to the population of these countries:
Altogether, 678 seniors from China (119 from elite programs), 364 seniors from India (71 from elite programs), and 551 seniors from Russia (116 from elite programs) took the examination…We also obtained assessment data on 6,847 seniors from a representative sample of CS programs in the United States (607 from elite programs).
The test they use is the “Major Field Test” from ETS. I don’t know that it’s a bad test. I do suspect that it’s US-centric. It’s like the final exam from our Chemistry teacher in my example. Compare that to the TIMSS assessments that go to great lengths to make sure that the data are contextualized and that the assessments are fair for everyone.
Maybe the results are true. Maybe US computer science students are far better than comparable CS students in Russia, China, and India. I’m just not convinced by this study.
Entry filed under: Uncategorized. Tags: computing education, computing education research, undergraduates.
Computing Ed Research - Guzdial's Take 
1.
Alfred Thompson | May 6, 2019 at 12:52 pm
I want to believe but like you I am skeptical. Thanks for the perspective. I think most people are going to take that chart at face value.
2.
Norman J Barnes | December 18, 2019 at 4:38 pm
I think the point needs to be made here that this was an international research team not merely an American one and that it included Russian, Chinese and Indian academics. Furthermore, the study has been published in the Proceedings of the National Academy of Sciences. Had there been a problem in methodology in, for example, sampling sizes, or the fact that the test was somehow US-centric, I think we would have heard early and very loud complaints and criticism from multiple reputable sources in those countries and elsewhere (other than from a few exhibiting sour grapes). To my knowledge we have not.
The fact is this was an eye-opening study. US computer science students whose K-12 education almost certainly did not prepare them as well for post-secondary computer science programmes as their peers from China and Russia certainly and maybe not India mostly, outperformed students from those countries by the end of their university CS programme. This suggests strongly that US university programmes, in computer science at least, are high quality and add very high value compared to the three other countries in the study. It’ll be interesting to see if this applies to other STEM fields.
Rather than sour grapes I imagine this study will stimulate some soul-searching in India and China, in particular, in the way they’re teaching computer science. In reading some Indian newspaper accounts online where the study was picked up more quickly than in the US, that process may already be starting.