Eric Mazur’s Keynote at ICER 2011: Observing demos hurts learning, and confusion is a sign of understanding

August 17, 2011 at 9:41 am 31 comments

Our keynote for ICER 2011 was Eric Mazur, famous Harvard physics education researcher.  Mazur maintains a terrific website with his publications and talks, so the slides from his talk are available as well as the papers which serve as the content for this talks.  His keynote talk was on “The scientific approach to teaching: Research as a basis for course design.”  I was hoping that he might give us some advice, from older-and-wiser physics education research to up-start, trying-to-learn-to-walk computing education research.  He didn’t do that. Instead, he told us about three of his most recent findings, which were great fun and intriguing.

The first set of findings were about peer instruction, which we’ve talked about here.  He spent some time exploring findings on the Force Concept Inventory (FCI), particularly with respect to gender.  In the US and Belgium (as one place where he’s explored this), there is a huge, statistically significant gap between men and women on the FCI.  In Taiwan, he didn’t find that same gap, so it is cultural, not innate.  With peer instruction, the gap goes away.  Good stuff, but not shocking.

The second set of findings were on physics demonstrations, when teachers make spark and lights, balance weights, make things explode (if you’re lucky), and do all kinds of things to wake you up and make you realize your misconceptions.  Do they really help?  Mazur tried four conditions (rotated around, so students would try them each): No demo, observing a demo, observing a demo after making a prediction of what you thought would happen, and then having a discussion afterward.  The results were pretty much always the same (here are the results from one study):

Yeah, you read that right — observing a demo is worse than having no demo at all!  The problem is that you see a demo, and remember it in terms of your misconceptions.  A week later, you think the demo showed you what you already believed.  On some of the wrong answers that students gave in Mazur’s study, they actually said “as shown in the demo.”  The demo showed the opposite!  The students literally remember it wrongPeople remember models, not facts, said Mazur.  By recording a prediction, you force yourself to remember when you guessed wrong.

That last line in the data table is another really interesting finding — talking about it didn’t improve learning beyond just making the prediction.  Social doesn’t help all learning.  Sometimes, just the individual is enough for learning.

This result has some pretty important ramifications for us computing educators.  When we run a program in class, we’re doing a demonstration.  What do students remember of the results of that program execution?  Do they even think about what they expect to see before the program executes?  What are they learning from those executions?  I think live coding (and execution) is very important. We need to think through what students are learning from those observations.

Third finding: Students praise teachers who give clear lectures, who reduce confusion.  Student evaluations of teaching reward that clarity.  Students prefer not to be confused.  Is that always a good thing?  Mazur tried an on-line test on several topics, where he asked students a couple of hard questions (novel situations, things they hadn’t faced previously), and then a meta-question, “Did you know what you were doing on those questions?”  Mazur and his colleagues then coded that last question for “confusion” or “no confusion,” and compared that to performance on the first two problems.

Confused students are far more likely to actually understand.  It’s better for students to be confused, because it means that they’re trying to make sense of it all.

I asked Mazur if he knew about the other direction: If a student says they know something, do they really?  He said that they tried that experiment, and the answer is that students’ self-reported knowledge has no predictive ability for their actual performance.  Students really don’t know if they understand something or not — their self-report is just noise.

About these ads

Entry filed under: Uncategorized. Tags: , , .

Indiana is hiring in CS Education research! App Inventor goes to MIT

31 Comments Add your own

  • […] Guzdial has posted a good summary of this year’s ICER keynote by physics education guru Eric Mazur, in which he reported the results of several recent […]

  • 2. Jung Choi  |  August 17, 2011 at 11:29 am

    Great post! Several other studies indicate that students are poor judges of their own knowledge & learning. I think the concept of “desirable difficulty” can go a long way to explaining these results. It’s disturbing that student misconceptions filter or affect what they take away from demos, and that the demos apparently reinforced their misconceptions?

  • 3. Darrin Thompson  |  August 17, 2011 at 1:40 pm

    Demos only reinforce misconceptions. Brilliant.

    I don’t remember how but in my day to day software work, especially when debugging I picked up an apparently good habit. I’ll often state out loud that something like, “After stepping over this line I expect blockDev to contain ‘/dev/sda1.'” Based on this research I’ll have to keep this habit, as debugging is all about discovering one’s own misconceptions.

    Is it really fair for Mazur to code “Did you know what you were doing…” as confused or not? Seems like “confidence” is what was measured and it’s not quite the same thing.

    • 4. Mark Guzdial  |  August 17, 2011 at 7:48 pm

      I didn’t get the prompt right, Darrin. I paraphrased. The actual question (for the study results I described) was: “3. Please tell us briefly what points of the reading you found most difficult or confusing. If you did not find any part of it difficult or confusing, please tell us what parts you found most interesting.”

  • 5. Neil Ernst (@neilernst)  |  August 17, 2011 at 2:27 pm

    Forgive me if this is answered elsewhere on the blog. Is it generally safe to assume that the results from physics education are applicable to CS?

    • 6. Mark Guzdial  |  August 17, 2011 at 5:18 pm

      Nope, it’s not safe to assume that — it’s an active area of research, and it’s one of the reasons why there were several replication studies at ICER 2011 this year. In general though, yes, most STEM education involves similar learning mechanisms and can be facilitated by similar teaching mechanisms. For example, Beth Simon has successfully replicated some of the peer instruction work from Physics in CS classes. At this year’s ICER, Beth reported on a study of PeerWise (invented for CS) which worked even better in Physics classrooms. How is CS learning different than Physics learning? We could conjecture lots of differences, but the studies of what works the same and what doesn’t are mostly yet-to-be-done.

  • 7. sesquiotic  |  August 17, 2011 at 2:39 pm

    This has some very interesting implications for the value of student evaluations of professors and courses, too.

  • 8. R. Wright  |  August 17, 2011 at 3:42 pm

    “Students praise teachers who give clear lectures, who reduce confusion. Student evaluations of teaching reward that clarity. Students prefer not to be confused… Students really don’t know if they understand something or not — their self-report is just noise. ”

    Yet somehow the teaching quality of most college professors in the U.S. is still judged almost entirely on student popularity. And we wonder why the value of a college education has gone down the toilet.

  • 9. Understanding Confusion « h walks into a bar   |  August 18, 2011 at 6:24 pm

    […] Eric Mazur’s Keynote at ICER 2011: Observing demos hurts learning, and confusion is a sign of unde…, Mark Guzdial @Computing Education Blog […]

  • 10. Algebra++  |  August 22, 2011 at 2:56 pm

    Dr. Mazur came to our department, 20 years ago when I was in grad school. He was just getting rolling. In science education, I was taught the uses of ‘discrepant events’. When we encounter counters to our intuition/expectation/prediction, it wakes us from sleepwalking – jogs our equilibrium. Physics may just have more of these counters than other fields. In the first physics classes: 1) Heavier objects SHOULD fall faster than light objects; Aristotle said so. 2) If I sling a weight around horizontally, I’d swear I was being pulled to the OUTside. For a full physicist, it doesn’t let up. You move into quantum mechanics and relativity, where particles tunnel through impossibly high barriers, and time passes differently for different observers. Perhaps confusion is a sign of wakefulness.

  • 11. Dean Townsley  |  August 22, 2011 at 3:35 pm

    About the headline “Observing demos hurts learning”… This is a bit more subtle than is being represented here. I don’t know what Mazur said in the lecture, but the data on the linked slides disproves the simplest interpretation of this assertion. The next table in Mazur’s slides after the first one discussed in this blog post, which reports exam data on a slightly different question (rather than “online test” data on a question apparently identical to the demo) shows a clear improvement even with just observing the demo (42% correct for just observe demo vs. 31% correct for no demo).

    Also I should point out that “correct” for the table mentioned in the blog post appears to require “mentions torque”, which only 24% of students did. In fact 60+% of students got the right answer, just not all mention torque in their justification, most using “proportional reasoning” (which a torque argument actually reduces to for this particular problem).

    That being said, the “observe only” demo mode provided only marginal improvement in students using correct methodology. Also it didn’t do nearly as well as doing it “right” with prediction of the demo outcome and follow-up discussion, especially for getting correct methodology (85% vs. 55% for this question).

    I think the take-away is that students will tend to remember the demo rather than the physics it is trying do demonstrate. That generally helps them remember at least something, but for questions too much like the demo, they will be unable to give proper reasoning. However, proper use of demos (with prediction and discussion) overcomes this and does much more.

  • 12. Lee  |  August 23, 2011 at 7:57 am

    It seems to me that the confused / not confused question is more likely to be a correlation based on the Dunn Kruger effect than on the ‘fact’ that confused students learn better

    • 13. William Doane  |  August 27, 2011 at 2:39 pm

      Lee- I don’t think anything in Mazur’s reporting so far makes a claim about the actual confusion level (an unknowable), but rather focuses on the self-report of being confused… so, I don’t think it’s incompatible with your observation about the Dunning-Kruger effect. -Wil

  • […] Eric Mazur’s Keynote at ICER 2011: Observing demos hurts learning, and confusion is a sign of understanding (via Computing Education Blog) Our keynote for ICER 2011 was Eric Mazur, famous Harvard physics education researcher.  Mazur maintains a terrific website with his publications and talks, so the slides from his talk are available as well as the papers which serve as the content for this talks.  His keynote talk was on "The scientific approach to teaching: Research as a basis for course design."  I was hoping that he might give us some advice, from older-and-wiser physics educat … Read More […]

  • 15. Alex R  |  August 27, 2011 at 2:28 pm

    Is there a paper published about Mazur’s findings with demonstrations, or do the results just exist in his presentation? I wasn’t able to find a paper on his website.

  • 17. dwees  |  August 27, 2011 at 10:29 pm

    I wonder what happens with interactive demos? For example, what if the student gets to control the demo (also known as a science lab) if that makes any difference to their learning?

    I remember my science labs in university, and they all seemed so pointless, because I had these prescriptive steps I was supposed to follow, and no freedom to actually experiment. What happens if students are asked to prove using an experiment that their misconceptions are actually true?

  • 18. BraveNewCurrency  |  August 28, 2011 at 8:37 pm

    Darwin observed “Ignorance more frequently begets confidence than does knowledge”. This effect has a name:

  • 19. Alex  |  August 31, 2011 at 9:47 am

    Well, I think the key to making the demonstration of an action useful to learning is that people actually WATCH the demonstration and pay attention.
    The discovery of mirror neurons in the brain has shown that the main way how humans learn is through watching and observing others. The observed action is mirrored by the observers brain in a form of internal body loop and the neuronal effect on the brain is similar as if the observed action has been executed by the observer himself.
    Of course it’s always hard to actually proof anything that goes on in the brain but in this case I would highly doubt the outcome of the study.

  • 20. Problem Solving « justasnapshot2011  |  September 3, 2011 at 12:24 pm

    […] to understand the questions more because they started off confusing. The article entitled “Confusion is a Sign of Understanding” allows me to not stress when I am confused, but to continue through frustration to really […]

  • 21. Money Problem Solving « life in math  |  September 15, 2011 at 12:06 pm

    […] on our own and then having help doing it seemed less confusing. which brings me to the article Observing demos hurts learning, and confusion is a sign of understanding by Eric Mazur. The article talks about confusion being a sign of understanding. Through out my own […]

  • […] trick: How do you know that the students learned what they were supposed to learn?  We know that self-assessment is a bad way of judging that learning.  That’s the contribution that I see the Stanford AI class making – doing assessment, […]

  • […] perhaps all those qualities held so highly by search engine designers are not so hot. The article, Eric Mazur’s Keynote at ICER 2011: Observing demos hurts learning, and confusion is a sign of unde…, on Computing Education Blog, explains how just the opposite may be needed for search users to […]

  • 24. Confused? Good. That Means You’re Understanding [Obvious]  |  October 4, 2011 at 12:02 pm

    […] Mazur, famous Harvard physics education researcher, recently did a keynote at ICER 2011 (you can read more about the whole keynote here). Of course, he talks about education – I also hear he’s a pretty smart […]

  • […] may learn actually less from lectures accompanied by demonstrations or experiments, because they will map their misconceptions onto what they […]

  • […] Schank has been saying for years that learning only occurs through failure.  Mazur’s results about physics demonstrations support Schank’s claim — you only learn from a physics demonstration if you first make […]

  • 27. Why Self-Reported Evaluations Are Evil | HAPGOOD  |  April 21, 2012 at 10:09 pm

    […] not quite. But this from Mazur is pretty depressing (though thoroughly expected if you read the literature on this sort […]

  • 28. Predicting Live Coding | Academic Computing  |  May 11, 2012 at 12:22 pm

    […] Guzdial, who was in turn talking about an ICER 2011 keynote speech by Eric Mazur (got that?). Guzdial has already written a nice summary of the speech, but in short: Mazur and his co-authors found that practical demonstrations had poor outcomes in […]

  • 29. Science Education Research | Academic Computing  |  July 9, 2012 at 4:32 am

    […] idea that old bad knowledge hangs around cropped up again in the work of Eric Mazur. Mazur looked at the effects of live demonstrations of science, long thought to be beneficial due to their engaging nature. Mazur and colleagues found that live […]

  • […] research being described in the below post was discussed here previously, and is related to the predict-before-demo work that Eric Mazur presented at last year’s ICER.  The uppermost bit here is that data mining can’t get at this […]

  • 31. Some Interesting Reading | Teaching Software Carpentry  |  April 10, 2013 at 11:15 am

    […] Eric Mazur’s Keynote at ICER 2011: Observing demos hurts learning, and confusion is a sign of … […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

Recent Posts

August 2011
« Jul   Sep »


Blog Stats

  • 1,054,189 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 3,571 other followers

CS Teaching Tips


Get every new post delivered to your Inbox.

Join 3,571 other followers

%d bloggers like this: