A Challenge to Computing Education Research: Make Measurable Progress

In computing education research, we repeatedly conduct studies showing just how little students are learning in our classes.  We’ve been doing it for almost 30 years.  I have a challenge to our community: Let’s show that we can do better!

I had two experiences this week that made me think of this challenge.

Students learn little in CS1: Allison Tew is defending her dissertation at the end of this week.  I’ll blog on that soon.  As I mentioned previously, Allison built the first language independent measure of CS1 knowledge, which she hoped to show was reliable and valid.  To establish the validity of her language independent exam (which used pseudo-code), she ran subjects through a test in their “native” CS1 language and in her pseudo-code, equivalent test. Allison has taken some criticism (from her SIGCSE 2010 paper) for defining a small definition of CS1, only the very most common topics across multiple styles of CS1.  She ran over 950 subjects, across Java, MATLAB, and Python classes at multiple institutions in two countries, making this (I believe) the largest study of CS1 knowledge ever.

The important bottomline was whether her test really works, but the bottomline that I want to focus on here is: The majority of students did not pass.  The average score on the pseudo-code test was 33.78%, and 48.61% on the “native” language test.

That really shouldn’t surprise us.  Elliot Soloway ran hundreds of subjects through his rainfall problem, and always found that students did really badly at it. Every attempt to replicate that study that I know of has found roughly the same thing. The McCracken working group study showed that students can’t design.  The Lister working group study showed that students couldn’t answer simple multiple choice questions about loops.  What’s surprising is that Allison continues to narrow the focus, to the barest of minimums — and still students can’t pass.

People don’t know computing. Erika Poole just submitted her final dissertation document this last week.  She did a fascinating study of families dealing with a variety of technology related challenges that she set out for them (and paid them to attempt), as a way of discovering how they sought out help.  The families in her study did much worse than I might have guessed, e.g., only 2 of 15 families were able to configure their wireless router.  Some of the reasons were because of absolutely brain-dead user interfaces (how about a virtual keyboard that was missing some keys?!?).  But others were more subtle.

One of her challenges was to edit a Wikipedia page.  When you edit a Wikipedia page for the first time, without an account, you get this warning message:

You are not currently logged in. Editing this way will
cause your IP address to be recorded publicly in this
page's edit history. If you create an account, you can
conceal your IP address and be provided with many other
benefits. Messages sent to your IP can be viewed on your
talk page.

Erika writes:

This message was confusing or concerning to these participants, who did not necessarily understand whether IP address exposure was harmful or innocuous.  …

Of the three non-completers, Nyree described the process of editing an encyclopedia and reading this warning message as something that made her ―feel like a criminal. She chose not to save the change she made.  Jamar, uncertain about whether exposing his IP address would open him up to harm,  called a computer savvy friend to help him interpret this message;  his friend told him that he should not complete the task.  Janine Dunwoody, too, was alarmed by this message, but decided she did not want to create an account, either, because she did not want to disclose personal information to Wikipedia.

Should we expect people who edit Wikipedia to know what an “IP address” is?  Is it a user interface mistake to expect that people should know what an “IP address” is?  At the very least, we can probably all agree that any student who takes a course in computing principles should come away knowing what an “IP address” is.

Stop replicating, start improving. Most of the studies described here have tried to understand what students and common users know about computing.  That’s a really important science goal.  However, we are also engineers — we attempt to create solutions to learning problems.  As computing education researchers, we have both goals, to understand and to improve.

After 30 years, why hasn’t somebody beaten the Rainfall Problem?  Why can’t someone teach a course with the explicit goal of their students doing much better on the Rainfall Problem — then publish how they did it?  We ought to make measurable progress.

I don’t think that this is an impossible goal.  In fact, I bet that some of the existing research projects in computing education could “beat” (generate published reports with better results) these current studies.

• The TeachScheme approach focuses on design based on data.  I bet that their students could beat the Rainfall Problem or the McCracken working group problem.
• I bet that the Problets, Practice-It!, or CodingBat folks could get better results on either the Lister problem set or Allison’s new test.  My bet is that lots of examples and practice in-the-small are what it would take to get students to understand those concepts.
• Students who take the new AP CS “Computer Science: Principles” should be able to complete all of Erika’s challenges successfully, and especially, know what an IP address is.

If someone makes an attempt and doesn’t beat the problem, that’s worth publishing, too.  We need to set out some yardsticks and measure progress against them.  It’s great to know exactly what’s going on in our classrooms, even if it’s not great news.  There are lots of people trying innovative approaches to computing education.  We need to close this loop, and see if the innovative approaches are making progress on those same challenges that showed us the bad news.  Let’s make measurable progress.