Where CS Teachers See Bimodality: Guest Blog Post from Elizabeth Patitsas

When I saw Elizabeth’s debrief, I asked her if I could share it on here. She graciously prepared this guest blog post, with more detail to explain what she did. Thanks, Elizabeth!

About a year ago, I used a number of venues to recruit participants for an anonymous study about the distributions of grades in CS classes. The study involved a minor deception, and because we do not have the emails of all the participants, I’m posting the debrief openly.

——————————

Dear study participant,

You’re getting this email because about a year ago you participated in my research project, “An investigation of the grades distributions in university computer science”. In this project, I showed you a series of six histograms and I asked you how often you saw that distribution’s shape in your own teaching, as well as how you’d categorize the distribution (normal, bimodal, etc).

I’m writing to you know to let you know this study involved a minor deception. We were actually most interested whether you’d label some ambiguous distributions as “normal” versus “bimodal”. We’ve now completed our analysis, and we want to debrief you before we write up our results for publication.

As you may know, there is a common perception amongst CS educators that grades distributions are bimodal. However, upon statistical analysis of the grades distribution available to us, we discovered that most of our grades distributions pass statistical tests of normality and very few of them of them pass the statistical tests of bimodality (see link for more).

We were curious why the perception of bimodal grades is so prevalent, even when grades may actually be normally-distributed. Ahadi and Lister argued at ICER 2013 that the perception of bimodal grades comes from CS educators believing that some students possess an innate gift/talent to do computer science. Regular readers of Mark’s blog would know this as the “Geek Gene Hypothesis”: the notion that when it comes to CS, you either can get it, or you can’t.

Ahadi and Lister argued that when people see two “peaks” in their grades, it’s because they’re expecting to see two different populations to be represented: the students who get it, and the students who don’t. Usually, bimodal distributions represent data where you’ve sampled two different populations together at the same time. If CS grades were bimodal, that could imply we have two different populations of students (e.g. those who get it + those who don’t). Whereas if CS grades are normal, it would imply (but not prove) our students form a spectrum, where most students understand some — but not all — of the material.

In the study you participated in, all six histograms were randomly-generated normal distributions with a small sample size (and so looked noisy). We wanted to test two things:

1. RQ1. Are CS educators more likely to label ambiguous distributions as “bimodal” if they believe that some students are inherently predisposed to do well in CS?
2. RQ2. If we tell CS educators that it’s a commonly-held belief that CS grades are bimodal, will educators be more likely to label ambiguous distributions as “bimodal”?

For a random half of the participants, before you categorized the distributions, we asked you “It is a commonly-held belief that CS grades distributions are bimodal. Do you find this to be the case in your teaching?” The other half of the participants saw this question after categorizing the distributions. This priming was used to test RQ2.

If our consent form had said the true intent of the study, then all participants would have been primed, rather than a random half. Our minor deception about the purpose of the study was necessary to answer this research question.

We delayed the debriefing until after our analysis was complete, on the assumption you’d want to know the preliminary results of the study. We indeed found that participants who agreed more strongly with the statement “Some students are innately predisposed to do better at CS than others” were statistically significantly more likely to label ambiguous distributions as bimodal.

Participants who had been primed were more likely to label distributions as bimodal — and this effect was stronger if they also agreed that CS ability was innate.

To ensure participant anonymity, we did not collect names and emails on the SurveyMonkey survey. As a result we have no way to link your responses to your identity.

We will be submitting our results for publication in the coming months, and will happily disseminate the paper once it is published. If you’d like to receive a copy of the paper once it’s published, add your email to this form. If you have any further questions, please contact us.

Thank you,

Elizabeth Patitsas

Jesse Berlin

Michelle Craig

Steve Easterbrook