Lister and Spohrer, Plans and Schema and MCQ’s

April 23, 2010 at 12:36 pm 8 comments

In response to my piece on Millenials, Raymond Lister pointed out that his ITICSE working group report was about reading code while the work by Jim Spohrer and Elliot Soloway was on writing code.  Of course, he’s right.  But his passing comment had had me thinking all night.  How different are those tasks?  I worked with Elliot for (mumble, mumble) years, and I read all that I could by him and Jim.  I just leafed through Jim’s dissertation again this morning (yeah, it sits on my shelf — I really am a total geek!  But now it’s on Google Books, too!).  I feel like there is such a similarity between what Lister’s group was working on and what Spohrer was working on, but I can’t quite put my finger on what’s so similar.

Let me quote a bit from Raymond et al.’s paper, if he doesn’t mind.  (If he does, I’ll delete these later.)  This is Question 5 of the multiple choice question (MCQ) instrument that Raymond’s group created and presented to students.  It was the easiest question, in terms of how students actually performed on it.

First off, I don’t find this question particularly easy.  It involves careful tracing, but I do agree that it feels like something that students should be able to do at the end of the first semester of Computer Science.  Why is that?  Why does it look like something we expect students should be able to do?

Elliot’s group at Yale explored a model for how programers write programs that focused on goals and plans. Goals were what you wanted to do, and plans were how you did them.  Sometimes you set a goal, and tried to implement a plan, but failed (reached an impasse). So you tried a different plan.  Plans were models of code, typical pieces of code that you wrote or saw a million times.  Rob Rist had a similar theory, structured around schemas instead of plans.  Tom-a-to, Tom-ah-to — it feels similar to me.

Imagine you saw this code:

i = 0;
while (i < a.length){
    // Do something with a[i]

Say I showed this to you and said, “It’s not working — what did I do wrong?”  I’m guessing that every reader of this blog would immediately say, “You forgot to increment i inside the loop!”  How did you know that?  Because you traced the code?  I suggest that you all know the index-across-the-array-with-a-while plan, and know that a common problem is forgetting to increment the index inside the loop.  You didn’t have to trace this code as much as you saw the problem, because you know this plan.

Why does Question 5 look so reasonable to us?  Because it looks like reversing an array?  Because it looks like the code for swapping elements of an array?  Both of those might be true.  In general, it looks like code we’ve seen many times.  We feel like we recognize those plans.

Here’s Question 8, the question that students performed the worst on.

This one feels harder to me.  Why?  The code looks like the code for a sort, like a Bubble Sort, but it’s not really.  It triggers my plan-recognizer enough to say, “Sure, I should be able to do that.” But when I try to match my plans to this, it doesn’t quite work.  So I have to do step-by-step reasoning, what Cognitive Scientists call weak methods.  If I immediately recognize a problem and can provide a ready-made answer, that answer is often right — that’s a strong method.  If I can’t do that, I fall back on my backup approaches, like tracing code line-by-line, which is more often going to fail.  Did students recognize that they had to trace to solve this problem, or did they just plug in what looked the most like Bubble Sort to them?  And once they recognized that they had to use a weak method, did they apply that method correctly or did they make a mistake?

On a side note, this question feels a bit like cheating on the “it’s about tracing, not writing code” claim.  Isn’t picking which expression or statement fits very much like composing the right plan into the code?  At what point does identifying code cross from a “tracing” task into a “writing” task?  Is it “tracing” because I didn’t have to type the characters in or fight the syntax?

In the end, I think that these two explorations, 12 years apart, feel similar to me because I can use the theory from the first one to explain the second one.  That is something that I don’t think we do enough of in computing education research today: Make theory, use theory to generate predictions, then test those predictions.  Many of the ICER conference papers present descriptions and analyses.  Those are great and important.  Many of the conclusions of ICER papers make recommendations for how we teach.  That, too, is great and important.  But science is about making theory, and progress in society comes from using scientific theories to make useful predictions.  We need to get to the point where we can be predictive in computing education research.

Entry filed under: Uncategorized. Tags: , .

Compose Your Own — Music and Software Alan Kay on Hoping That “Simple” is not “Too Simple”

8 Comments Add your own

  • 1. Raymond Lister  |  April 23, 2010 at 4:37 pm

    I’m in the midst of packing up my apartment in Vancouver, before returning to Australia. We need to be out of the apartment by April 30, so I’m not going to write a long thoughtful reply. But then, I’m not sure that the
    original posting was thoughtful either.

    Mark wrote, “… Question 8 … this question feels a bit like cheating on the “it’s about tracing, not writing code” claim.”

    You haven’t read the paper carefully. The paper presents two types of questions, tracing questions (called
    fixed code questions” in the paper) and skeleton code questions. Question 8 is of the latter kind.

    Mark wrote, “… these two explorations, 12 years apart, feel similar to me …”.

    To a man with a hammer, every problem will look like a nail.

    Mark wrote, “… science is about making theory …”

    Indeed it is, but as they also teach in History and Philosophy of Science classes, not every theory turns out to be useful.
    Schemas, at least in the epistemological form touted by Soloway and his acolytes, turned out to be as useful as the geocentric models of the planets.

    For an alternative schema theory that is more recent and in my opinion more promising, see “roles of variables” at …

    There is also a great deal of literature in cognitive science, educational psychology, and philosophy of mind, much of it published since Soloway et al., that critique and offer alternatives to the general idea of a schemas. My own experimental work is motivated by that non-schema literature.

    For my more recent work on the role of tracing in programming, see the following two papers …

    Lister, Fidge, and Teague, 2009. “Further evidence of a relationship between explaining, tracing and writing skills in
    introductory programming. In Proceedings of ITiCSE ’09, pp. 161-165.

    Venables, Tan, and Lister, 2009. A closer look at tracing, explaining and code writing skills in the novice programmer.
    In Proceedings of ICER ’09. pp. 117-128.

    The discussion/conclusion sections of those papers contain some speculations on how tracing/explaining/writing code relate, of relevance to the original blog posting. Those sections can probably be read without first reading
    the rest of those two papers.

    Now, back to packing …

    • 2. Mark Guzdial  |  April 23, 2010 at 9:53 pm

      Hi Raymond! I did read the paper (several times now). I do realize that you made this distinction, but even “skeleton code” problems aren’t “coding” questions. I think they’re closer to what Spohrer was exploring, but I took your previous comment to suggest that they’re closer to “tracing” questions. Sorry I misunderstood.

      I’ll look into your non-schema literature — thanks for the link!

      While not every theory turns out to be useful, the process of making and disproving theories is pretty useful.

    • 3. Mark Guzdial  |  April 26, 2010 at 9:37 am

      Raymond, I’ve been digging into some of the literature that you referenced in this comment. I knew of the Roles of Variables work before, and while it offers an alternative schema theory, it is a schema theory. One of the 2007 papers referenced on the Roles of Variables site starts out: “Expert programmers possess schemas, abstractions of concrete experiences, which help them solve programming problems and lessen the load on their working memory during problem solving. Possession of schemas is a key difference between novices and experts, which is why instructors need to help students construct them. One recent tool for facilitating schema formation in introductory programming are roles of variables, which represent stereotypes of variable use in computer programs (Sajaniemi, 2002).” When I was in Elliot’s group, a common topic of discussion was where data fit into a goals-plans analysis. It’s important to explore these schema that are organized around data rather than process. Most of the literature that I see on roles of variables is focused on novices, not experts. I wonder how experts analyze their data? Do they think about it in terms of roles, too? Same roles?

      Could you point me to some of non-schema cognitive science literature you’re describing, please? I’m looking through your papers on tracing, explaining, and writing, but I’m not recognizing which of your references is explaining the non-schema theory of cognition that you’re describing here.

  • 4. owen astrachan  |  April 23, 2010 at 8:35 pm

    So there’s a guy who’s a curmedgeon. I know him. Here’s what he says.

    Question 5 isn’t easy, it’s not hard, it’s just ridiculous. When we ask students to quote-trace-unquote code, it should (IMHO) be code whose purpose is understood. In general we don’t debug code that’s a mystery, though for students sometimes I wonder. We debug code whose purpose we know — because we had an intention in writing the code and the bug shows that our intention wasn’t realized in reality.

    In the case of question #5: this is a test-taking-skills question. If we had to enter the result rather than do multiple choice, there would be a different story to tell. Ok, students did well on this. As you write “it smells like swapping”. There are only two answers with 3 as the first element, both of these have 0 as the last element. If we know “swap” we’re immediately reinforced with something that can lead us to correctness. We have the added advantage that we only have to differentiate among two answers that might be correct if our reasoning is correct. Only one entry is different, we can immediately concentrate on identifying why that entry is either 1 or 2, so we look at the 2*temp which is “smelly”, it’s not what we expect. So if you’ve seen a swap, you know that smells, you identify the problem, and you get the answer. If you’re answering this by actually tracing instead of eliminating answers *while* examining the code you don’t have good test-taking skills at all.

    On to Question #8. You say “it feels harder”. If you have to do step-by-step reasoning, you’re like the students who didn’t do well on this. I conjecture that the “good” students immediately recognize that an inversion requires identifying an entry with later entries that are smaller. That’s just like lots of problems we give in our CS1 course, it’s like finding the min in an array, for example: compare yourself to everything after you, it’s like selection sort. But mostly it’s like comparing an entry with one that comes later. If you pick answer (a) or (b) you don’t get it, they compare i with an earlier-j. Again, you don’t trace, you eliminate incorrect answers. Now you’re down to simply the end condition of the array: do you want to stop one-before-the-end or at-the-end. You might get confused because the outer-loop stops one before the end. If you recognize the need for this, because that leaves something after it (as opposed to the last element which has nothing after it) then you immediately clue in to the correct answer.

    So, I think this is as much about multiple-choice test-taking skills as it is about recognizing code patterns. If you’re really tracing, you’re making a huge mistake. If you eliminate/trace/verify/trace hypothesize-in-your-head, verify sort of stuff, which is a test-taking skill, you’ll do much better than if you try to trace.

    If you really want to test the ability to apply patterns in previously seen code/examples, I think you have to ask students to answer without choices-to-choose-from or you need to ask them to write code. Tracing != writing.

    All this is related to what you’re writing about, but multiple choice questions are confounding in trying to ascribe rational reasoning to how students approach these problems or what they know. If you know how well students do on standard MC tests, you’d learn a lot toward understanding how they do on this test.

    Or so I assert without a formal study

    • 5. Mark Guzdial  |  April 26, 2010 at 9:24 am

      Owen, isn’t some of that “test-taking skill” what you want to test here? Yes, the “good” students immediately recognize that (a) and (b) are wrong — but that’s the point of the test, to separate the “good” students from the ones who “don’t get it.” The fact that students performed so badly on Q8 is interesting. Alright, maybe not to you, given your long history with the AP and MCQ’s for testing CS. But to most of us, it’s another indication that students aren’t learning all that we think that they’re learning in CS1.

  • 6. Aaron Lanterman  |  April 24, 2010 at 7:25 pm

    Reminds me of my differential equations class; the tests were all multiple choice with something like 10 choices per problem.

    On most questions, I first eliminated the answers that didn’t satisfy the initial conditions; I then took the remaining answers and plugged them into the original diff eq to see which one solved it. I didn’t do many of the problems according to the official techniques for solving them.

  • 7. Aaron Lanterman  |  April 24, 2010 at 7:30 pm

    These sorts of programming questions remind me of my main beef with the circuit exercises in most introductory circuits texts: the circuits don’t actually *do* anything. They’re vaguely helpful for teaching node-voltage equations, mesh-current equations, Thevenin equivalents, or whatever, but no one would ever actually construct such circuits since they don’t actually do anything useful.

    It’s even worse when terribly unreleastic values are used, such as a 1 ohm resistor or a 1 farad capacitor or a 1 Henry inductor. I think students should get a feel for what real, typical values are.

  • […] the same thing. The McCracken working group study showed that students can’t design.  The Lister working group study showed that students couldn’t answer simple multiple choice questions about loops. […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,038 other followers


Recent Posts

Blog Stats

  • 2,014,277 hits
April 2010

CS Teaching Tips

%d bloggers like this: