Taking a test is better than studying, even if you just guess: We need to flip the flipped classroom

July 30, 2013 at 1:47 am 15 comments

The benefits of testing for learning are fascinating, and the result described below makes me even more impressed with the effect.  It suggests even more strongly that the critical feature of learning is trying to understand, trying to generate an answer, even more than reading an answer.

Suppose, for example, that I present you with an English vocabulary word you don’t know and either (1) provide a definition that you read (2) ask you to make up a definition or (3) ask you to choose from among a couple of candidate definitions. In conditions 2 & 3 you obviously must simply guess. (And if you get it wrong I’ll give you corrective feedback.) Will we see a testing effect?

That’s what Rosalind Potts & David Shanks set out to find, and across four experiments the evidence is quite consistent. Yes, there is a testing effect. Subjects better remember the new definitions of English words when they first guess at what the meaning is–no matter how wild the guess.

via Better studying = less studying. Wait, what? – Daniel Willingham.

These results mesh well with a new study from Stanford.  They found that the order of events in a “flipped” classroom matters — the problem-solving activity (in the classroom) should come before the reading or videos (at home). The general theme is the same in both sets of studies: problem-solving drives learning, and it’s less true that studying prepares one for problem-solving.

A new study from the Stanford Graduate School of Education flips upside down the notion that students learn best by first independently reading texts or watching online videos before coming to class to engage in hands-on projects. Studying a particular lesson, the Stanford researchers showed that when the order was reversed, students’ performances improved substantially.

via Classes should do hands-on exercises before reading and video, Stanford researchers say.

Entry filed under: Uncategorized. Tags: , .

The challenges of integrated engineering education Writing programs using ordinary language: Implications for computing education

15 Comments Add your own

  • 1. Donna Llewellyn  |  July 30, 2013 at 7:50 am

    What about a sandwich class? Read (some entry level of prep), problem solve, read (further investigation).

    • 2. Mark Guzdial  |  July 30, 2013 at 11:02 am

      Of course, I can’t answer any of the “what about…” questions, because I’m reporting on two other studies. I can offer my interpretation, and play the suggestions against that model.

      Learning is a hard and cognitively expensive activity, especially accommodation (vs. assimilation) and conceptual change. Humans avoid it, if they can. I agree, Alan, that good studying skills can reduce the cost of learning, but it’s harder than other cognitive activities. From that perspective, it makes sense that you would engage learning in response to a problem whose solution you did not already have. Simply studying, especially if it’s not intrinsically interesting, does not make it obvious what to learn (as Sneezypb points out).

      The shared problem that all undergraduates have is, “Earn a degree.” You can certainly get them to read something, if they believe it is necessary for solving that problem. What I hear these two studies saying is that engaging students in a problem (a test, a visualization exercise) that they find engaging will motivate later studying and book-learning. Donna’s suggestion of a “sandwich class” could certainly work, but might be even more effective swapping the bread-and-filling: Do a small exercise in class to point out to students what they don’t know, go read and prep, and follow it up with another exercise where they succeed.

      Bri, I see your point about frustration, and find it interesting that neither of these studies mentioned measuring motivation or self-efficacy. Probably is a problem, but it may also be a matter of how far a cognitive leap you’re asking. If I put you in an operating room and demanded you do brain surgery, you would have some doubts about your efficacy. But if I asked you to program in a language you hadn’t met yet, I’d bet that you’d figure out >75% of it and would be prompted to learn the rest at home.

      Richard, your “problem” is about a change of perspective, a reframing of the problem in a Schön sense. That’s a “wicked” problem in the cognitive science sense, and seems to be beyond what I’m reading in these studies.

  • 3. Bri Morrison  |  July 30, 2013 at 9:09 am

    When reading the first part of your post I was thinking how well it meshed with Eric Mazur’s work of asking students to predict what happens before they run the experiment. The second half of the post is more in line with Just-in-time teaching. But then when I try to apply it to teaching programming: how do you have the students solve the problem when they don’t yet know the construct that’s needed? In other words, there’s a problem that requires a loop for the solution yet they haven’t read or learned about loops yet. Seems to me that would lead to frustration in the classroom as they try to solve something without the necessary tools. And how does the instructor know when to stop the problem solving and start the instruction of the new construct? Some students are likely to be frustrated while others are still lost. I still believe the better approach are paired worked examples: one worked example followed by one they solve on their own.

  • 4. Richard Barke  |  July 30, 2013 at 9:15 am

    What about areas of learning in which there is not “a problem” that can be “solved”? I use a class exercise in which student teams are told to “devise the optimal campus parking policy.” Students begin with “free parking for everyone everywhere,” then realities are introduced (by the students or, when needed, the instructor). Optimal for whom, what are the costs, and how are they financed, what are the unintended consequences, etc.? In the end, students produce a somewhat well-structured set of practices.

    Then I point out that they were solving the wrong problem (which I gave them, just as in the real world they’ll often be given the wrong problem to start). They should have been asking “how do we get people from A to B?” The campus parking “problem” is an artifact of decisions about housing patterns, local schools, public transit, bike lanes, staff working hours, internships and coops, nearby amenities, etc. About one in five groups of students understands this rather quickly, but most need to be reminded that “the problem” is malleable and that they’re allowed (expected!) to question the professor’s formulation. Type III errors are rife in the policy world. Obviously.

    For the definition of a vocabulary word, solving a differential equation, or designing a circuit, there usually is “a” solution. We should be cautious about using pedagogies that assume that “problem solving” has a binary outcome.

    • 5. gasstationwithoutpumps  |  July 30, 2013 at 12:18 pm

      Designing a circuit definitely has more than one solution. Indeed, type III errors abound in circuit design (often the circuit is not needed at all, once the problem is properly identified—nowadays it is often better to do most of the functions traditionally done in analog circuits with software).

  • 6. alanone1  |  July 30, 2013 at 9:36 am

    What are they actually testing here?

    Are they testing a method, or simply finding out that today’s students have little to no studying skills? (I.e they have not built internally what good “studyers” have: the heuristic behaviors that provide a wide variety of ways to learn things on one’s own.)

    These are the kind of experiments and conclusions that soft areas of study should not do. The softer the area, the more rigorous the investigations should be — and in so much of the educational “research” literature this is just not the case. They get some kind of correlation, but they don’t even identify for certain the correlates.


  • 7. sneezypb  |  July 30, 2013 at 10:04 am

    Could it be that by guessing first and getting immediate feedback, the student has a clear understanding of whether or not they know the material and what to pay attention. Just reading a text, they do not have any idea what they should be learning.

    In college, study advice from someone was to look at the study questions in a text book. They would be a guide to what is important in the reading. This seems like a similar concept.

  • 8. derekbruff  |  July 30, 2013 at 10:23 am

    My takeaway from that Stanford study is that we need to be more intentional about what kinds of activities we ask students to engage in before class. The point of the flipped classroom is to have students do the “hard” part of learning during class, when they can call on their peers and instructor for help. What happens before class is meant to prepare them for the in-class experience.

    Most faculty who flip their classrooms ask students to prepare for class by reading their textbook or watching lecture videos, but, as the Stanford study points out, those might not be the best ways to give students “first exposure” to a topic. I see two key questions here: (1) What’s the best way to introduce students to a given topic? (2) What’s the best way to use limited face-to-face class time?

    Mike Winiski from Furman University had a great post on this theme last year: http://mikewiniski.org/blog/?p=267.

  • 9. jchoigt  |  July 30, 2013 at 10:36 am

    One of the standards for discipline-based education research is to measure learning gains with pre- and post-tests. These results imply that the act of pre-testing will affect student learning, and must be carefully accounted for.

    • 10. David Klappholz  |  July 30, 2013 at 8:54 pm

      @jchoigt: These studies appear to suggest that simply taking a pre-test improves learning. (What, after all, is a pre-test, from the student’s point of view, other than an attempt to solve the problems being tested?)

  • 11. Helmi Reid  |  July 30, 2013 at 8:20 pm

    Interesting piece. At what level of education is this study actually referring to?

  • 12. Kajal Sengupta  |  August 2, 2013 at 4:20 am

    I agree fully with the comment that flipped classrooms matter. Large number of teachers are using it successfully. Read about some of them in http://www.wiziq.com/e-book/82-6-things-teachers-do-to-flip-the-classroom

  • 14. Mark Miller  |  August 9, 2013 at 3:41 pm

    This is just a cursory first impression, but what might be going on here is developing an inquiry process in the students (the Stanford article suggests this). By getting them to guess, they have to develop some notion of what the thing is, and what’s important about it, whether it’s accurate or not. Whether they have some background in how to learn in different subject areas that would allow them to make an educated guess (as opposed to something superficial and purely subjective) is not answered here. By having them form a notion, and then study about it, they get to engage in a kind of “self discussion.” By engaging in this exercise of comparison, that may more clearly define the available knowledge for them.

    Perhaps the Stanford article is just providing a snapshot, and there’s more to the course that’s not revealed there, but what bothered me about the description was that the students (and by extension the school) appeared to be pretending that a computer model is the equivalent of working on a real brain and sensory nervous system, making the exercise (study this, then study that) appear to be an almost pure abstraction.

  • […] post by Cathy Moore (and another that I came across not too long ago here at Computing Education Blog ) struck a chord with me. In essence, they are both saying that learners can benefit by having […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,184 other subscribers


Recent Posts

Blog Stats

  • 2,054,360 hits
July 2013

CS Teaching Tips

%d bloggers like this: