The problem with sorting students into CS classes: We don’t know how, and we may institutionalize inequity

One of the more interesting features of the ACM SIGCSE ITiCSE (Innovation and Technology in CS Education) conference are “working groups” (see description here). Groups of attendees from around the world work together before and at the conference on an issue of interest, then publish a report on what happened. This is the mechanism that Mike McCracken used when he organized the first Multi-Institutional, Multi-National (MIMN) study in CS Ed (see paper here). This year’s Working Group #9 caught my eye (see list here).

The description of what the group wants to explore is interesting: How can we measure what will lead to success in introductory computer science?

The main issues are the following.

• The ability to predict skill in the absence of prior experience
• The value of programming language neutrality in an assessment instrument
• Stigma and other perception issues associated with students’ performance, especially among groups underrepresented in computer science

It’s a deep and interesting question that several research groups have explored. Probably the most famous of these is the “The Camel has Two Humps.” If you read that paper, be sure to read Caspersen et al’s (unsuccessful) attempt to replicate the results (here), Simon’s work with Dehnadi and Bornat to replicate the results (again unsuccessful, here), and then finally the retraction of the original results (here). Bennedsen and Caspersen have a nice survey paper about what we know about predictive factors from ICER 2005, and there was a paper at SIGCSE 2019 that used multiple data sources to predict success in multiple CS courses (here). The questions as I see it are (a) what are the skills and knowledge that improve success in learning to program, (b) how can we measure to determine if they are there, and (c) how can we teach those skills explicitly if they are not.

Elizabeth Patitsas explored the question of whether there are innate differences between students that lead to success or failure in introductory CS (see paper here). She does not prove that there is no so-called Geek Gene. We can’t prove that something does not exist. She does show that (a) that grades at one institution over many years are (mostly) not bimodal, and (b) some faculty see bimodal grade distributions even if the distribution is normal. If there was something else going on (Geek Gene, aptitude, whatever), you wouldn’t expect that much normality. So she gives us evidence to doubt the Geek Gene hypothesis, and she gives us a reasonable alternative hypothesis. But it isn’t definitive proof — that’s what Ahadi and Lister argued at ICER 2013. We have to do more research to better understand the problem.

Are Patitsas’s results suspect because they’re from an elite school? Maybe. Asking that question is really common among CS faculty — Lecia Barker found that that’s one of the top reasons why CS faculty ignore CS Ed research. We discount findings from places unlike ours. That’s why Multi-Institutional, Multi-National (MIMN) is such a brilliant idea. They control for institutional and even national biases. (A MIMN study showing the effectiveness of Peer Instruction is in the top-10 SIGCSE papers list.)

In my research group, we’re exploring spatial reasoning as one of those skills that may be foundational (though we don’t yet know how or why). We can measure spatial reasoning, and that we can (pretty easily) teach. We have empirically shown that wealth (more specifically, socioeconomic status (SES)) leads to success in computing (see this paper), and we have a literature review identifying other privileges that likely lead to success in CS (see Miranda Parker’s lit review).

I am concerned about the goals of the working group. The title is “Towards an Ability to Direct College Students to an Appropriately Paced Introductory Computer Science Course.” The first line of the description is:

We propose a working group to investigate methods of proper placement of university entrance-level students into introductory computer science courses.

The idea is that we might have two (or more) different intro courses, one at a normal pace and one at a remedial pace. The overall goal is to meet student needs. There is good evidence that having different intro courses is a good practice. Most institutions that I know that have multiple introductory courses choose based on major, or allow students to choose based on interest or on expectations of abilities. It’s a different thing to have different courses assigned by test.

If we don’t know what those skills are that might predict success in CS, how are you going to measure them? And if you do build a test that sorts students, what will you actually be sorting on?  It’s hard to build a CS placement test that doesn’t actually sort on wealth, prior experience, and other forms of privilege.

If the test sorts on privilege, it is institutionalizing inequity. Poor kids go into one pile, and rich kids go into the other. Kids who have access to CS education go into one pile, everyone else into another.

Why build the test at all?  To build a test to sort people into classes presumes that there are differences that cannot be mitigated by teaching. Building such a test presumes that there is a constant answer to “What does it take to succeed in CS1?” If we had such a test, would the results be predictive for classes that both use and don’t use pair programming? Peer instruction? Parsons problems?

I suggest a different perspective on the problem.  We can get so much further if we instead improve success in CS1. Let’s make the introductory course one that more students will succeed in.

There’s so much evidence that we can improve success rates with better teaching methods and revised curriculum. Beth Simon, Leo Porter, Cynthia Lee, and I have been teaching workshops the last four years to new CS faculty on how to teach better. It works — I learned Peer Instruction from them, and use it successfully today. My read on the existing literature suggests that everyone benefits from active learning, and the less privileged students benefit the most (see Annie-Murphy Paul’s articles).

One of the reasons why spatial reasoning is so interesting to explore is that (a) it does seem related to learning computing and (b) it is teachable.  Several researchers have shown that spatial skills can be increased relatively easily, and that the improved skills are long-lasting and do transfer to new contexts. Rather than sort people, we’re better off teaching people the skills that we want them to have, using research-informed methods that have measurable effects.

Bottom line: We are dealing with extraordinary enrollment pressures in CS right now. We do need to try different things, and multiple introductory courses is a good idea. Let’s manage the enrollment with the best of our research results. Let’s avoid institutionalizing inequities.