Parsons Problems have same Learning Gains as Writing or Fixing code, in less time: Koli Calling 2017 Preview

November 17, 2017 at 7:00 am 12 comments

On Saturday, Barbara Ericson will be presenting at Koli Calling her paper (with Lauren Margulieux and Jeff Rick), “Solving Parsons Problems Versus Fixing and Writing Code.”

The basic design of her experiment is pretty simple.  Everybody gets a pretest where they answer multiple-choiced questions, write some code, fix some code, and solve some Parsons problems.  (I’ve written about Parsons Problems here before.)

Then there are three instructional treatments with three different kinds of problem-solving practice:

  • One group gets Parsons Problems with distractors in them — blocks that should not be dragged into the solution.
  • One group gets the same code to fix — same code as in the Parsons Problems but all the distractors are there.  They have to fix the broken code in the distractor to get to the same code as the correct block in the Parsons.
  • One group gets to write the code to solve the same problem.

Then they take an isomorphic (same basic problems with context and constants changed) post-test, go away, and come back one week later for a retention test (which is isomorphic to both the pretest and the first posttest: multiple choice questions, Parsons, fix code, write code).  So we have students who study with Parsons Problems getting tested by writing and fixing code.

Here’s the bottom line from their abstract: “We found that solving two-dimensional Parsons problems with distractors took significantly less time than fixing code with errors or than writing the equivalent code. Additionally, there was no statistically significant difference in the learning performance, or in student retention of the knowledge one week later.”

That’s it. It’s simple but profound.  Below is the timing table from the paper. The Parsons Problems took effort, but always less time — sometimes they took only half the time of fixing or writing code, and other times it was only a few percentage less. But it was always less.

One takeaway idea is: If Parsons leads to the same learning in less time, why wouldn’t every teacher use more Parsons problems?  A second one that we’ve been thinking alot about is: Can we provide more Parsons problems so that in the same amount of time that students were writing code, they actually learn more? Efficiency matters, as Elizabeth Patitsas’s work suggests — more efficient learning may mean less belief in Geek Gene by CS teachers.


Entry filed under: Uncategorized. Tags: , , .

Royal Society Report on CS in English Schools: The Challenge of Reaching Everyone Keeping the Machinery in Computing Education: Back to the Future in the Definition of CS

12 Comments Add your own

  • 1. Anders S  |  November 17, 2017 at 7:53 am

    Very cool! It’d be interesting to see how you could use Parsons Problems w distractors in tech conference tutorials and other adult-focused trainings where teaching time is short and so efficiency could really pay off

  • 2. Michael  |  November 17, 2017 at 11:14 am

    Where do they say they have the same learning gains? From the paper…

    > Participants also performed better on the fix and write code problems on both post-tests than on the pre-test.

    > There were no significant differences from the pretests to the posttests on the multiple-choice questions or the Parsons problem.

    • 3. Mark Guzdial  |  November 17, 2017 at 11:30 am

      These statements are talking about the pre-and post-tests. The treatment came between those tests. For the MCQ and Parsons problems on the tests, there was no significant difference pre/post — for all three treatments. There was significant difference pre-to-post on the fixing and writing code problems on the tests. That there was a significant difference pre-to-post is considered evidence of learning. Thus, across all three treatments, there was learning, as measured by the code fixing and code writing problems. There was no difference between treatments. That warrants the claim of same learning gains.

      • 4. Michael  |  November 17, 2017 at 10:22 pm

        Ah yes, sorry–I thought they were talking about conditions, and not item types!

  • 5. Alfred Thompson  |  November 17, 2017 at 6:49 pm

    The one reason I don’t use them is that I really don’t understand how to set them up. I suppose I could figure it out from articles but I have so much on my plate that doing so takes more cycles than I’ve got. I’d love a workshop on them. Perhaps at SIGCSE, or better for me, CSTA.

    • 6. Mark Guzdial  |  November 17, 2017 at 7:27 pm

      One of the items on the list for the NSF proposal I’m writing!

    • 7. evan  |  March 19, 2019 at 7:28 am

      a bit late, but this might help others who are having the same issue. Here’s a site for generating parsons problems from copy-pasted snippets of code, you can also share permalinks for curated examples

  • 8. Parson Problems – Web Explorations  |  December 9, 2017 at 4:33 pm

    […] his Computing Education Blog, Parsons Problems have the same learning gains as writing or fixing code, Mark Guzdial highlighted how useful Parsons Problems can be for interactive learning […]

  • […] wife and research partner, Barbara Ericson, is finishing her PhD on adaptive Parsons problems.  She is going to shift her emphasis from being Director of CS Outreach to more […]

  • […] solving Parsons problems, fixing code, and writing code. (Editor’s note: I blogged on her first study here.) I also tested two forms of adaptation. For the second study, I added […]

  • 11. Parson's Problems for Interactive Assessment -  |  June 20, 2018 at 12:05 pm

    […] his Computing Education Blog, Parson’s Problems have the same learning gains as writing or fixing code, Mark Guzdial highlighted how useful Parson’s Problems can be for interactive learning […]

  • […] problems, and through solving her new adaptive Parsons problems. She had a control group this time (different from her Koli Calling paper) that did turtle graphics between the pre-test and post-test, so that she could be sure that there […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,184 other subscribers


Recent Posts

Blog Stats

  • 2,054,191 hits
November 2017

CS Teaching Tips

%d bloggers like this: