Lister and Spohrer, Plans and Schema and MCQ’s

In response to my piece on Millenials, Raymond Lister pointed out that his ITICSE working group report was about reading code while the work by Jim Spohrer and Elliot Soloway was on writing code.  Of course, he’s right.  But his passing comment had had me thinking all night.  How different are those tasks?  I worked with Elliot for (mumble, mumble) years, and I read all that I could by him and Jim.  I just leafed through Jim’s dissertation again this morning (yeah, it sits on my shelf — I really am a total geek!  But now it’s on Google Books, too!).  I feel like there is such a similarity between what Lister’s group was working on and what Spohrer was working on, but I can’t quite put my finger on what’s so similar.

Let me quote a bit from Raymond et al.’s paper, if he doesn’t mind.  (If he does, I’ll delete these later.)  This is Question 5 of the multiple choice question (MCQ) instrument that Raymond’s group created and presented to students.  It was the easiest question, in terms of how students actually performed on it.

First off, I don’t find this question particularly easy.  It involves careful tracing, but I do agree that it feels like something that students should be able to do at the end of the first semester of Computer Science.  Why is that?  Why does it look like something we expect students should be able to do?

Elliot’s group at Yale explored a model for how programers write programs that focused on goals and plans. Goals were what you wanted to do, and plans were how you did them.  Sometimes you set a goal, and tried to implement a plan, but failed (reached an impasse). So you tried a different plan.  Plans were models of code, typical pieces of code that you wrote or saw a million times.  Rob Rist had a similar theory, structured around schemas instead of plans.  Tom-a-to, Tom-ah-to — it feels similar to me.

Imagine you saw this code:

i = 0;
while (i < a.length){
    // Do something with a[i]
}

Say I showed this to you and said, “It’s not working — what did I do wrong?”  I’m guessing that every reader of this blog would immediately say, “You forgot to increment i inside the loop!”  How did you know that?  Because you traced the code?  I suggest that you all know the index-across-the-array-with-a-while plan, and know that a common problem is forgetting to increment the index inside the loop.  You didn’t have to trace this code as much as you saw the problem, because you know this plan.

Why does Question 5 look so reasonable to us?  Because it looks like reversing an array?  Because it looks like the code for swapping elements of an array?  Both of those might be true.  In general, it looks like code we’ve seen many times.  We feel like we recognize those plans.

Here’s Question 8, the question that students performed the worst on.

This one feels harder to me.  Why?  The code looks like the code for a sort, like a Bubble Sort, but it’s not really.  It triggers my plan-recognizer enough to say, “Sure, I should be able to do that.” But when I try to match my plans to this, it doesn’t quite work.  So I have to do step-by-step reasoning, what Cognitive Scientists call weak methods.  If I immediately recognize a problem and can provide a ready-made answer, that answer is often right — that’s a strong method.  If I can’t do that, I fall back on my backup approaches, like tracing code line-by-line, which is more often going to fail.  Did students recognize that they had to trace to solve this problem, or did they just plug in what looked the most like Bubble Sort to them?  And once they recognized that they had to use a weak method, did they apply that method correctly or did they make a mistake?

On a side note, this question feels a bit like cheating on the “it’s about tracing, not writing code” claim.  Isn’t picking which expression or statement fits very much like composing the right plan into the code?  At what point does identifying code cross from a “tracing” task into a “writing” task?  Is it “tracing” because I didn’t have to type the characters in or fight the syntax?

In the end, I think that these two explorations, 12 years apart, feel similar to me because I can use the theory from the first one to explain the second one.  That is something that I don’t think we do enough of in computing education research today: Make theory, use theory to generate predictions, then test those predictions.  Many of the ICER conference papers present descriptions and analyses.  Those are great and important.  Many of the conclusions of ICER papers make recommendations for how we teach.  That, too, is great and important.  But science is about making theory, and progress in society comes from using scientific theories to make useful predictions.  We need to get to the point where we can be predictive in computing education research.