Practice is better for learning facts, worked examples are better for learning skills
April 4, 2012 at 6:56 am 18 comments
Fascinating piece in US News and World Report on the LearnLab work at Carnegie Mellon University. Since I’m exploring worked examples research and the implications for CS Education these days, I found the below section of the interview with Ken Koedinger intriguing. Practice helps you learn facts, but worked examples help you learn skills. Isn’t learning to program mostly about learning skills? We should be providing lots more worked examples of programming (not just the code — the process) to teach programming skills.
In math, for example, traditionally, students receive a list of math problems to solve. But this approach “gives novice learners too little support in constructing new knowledge,” Koedinger says. “It’s not as effective as replacing about half of those problems with example solutions. Rather than guessing their way through problems, these worked-out examples allow students to focus on grasping the thinking needed so they can solve future problems on their own.”
Thus, “if every other problem contains a step-by-step solution, students learn more robust skills,” he adds. “Even better is adaptive computer-based practice that adjusts to individual students, providing more worked-out solution steps initially, but then gradually challenging a student with more problems as he or she increases in understanding and skill.”
But Koedinger is quick to point out that using more worked examples is not the answer for all learning goals. “They are best for skills, but pure practice is better for facts,” he says. “For deeper concepts and principles, more emphasis on providing explanations is important, but should these explanations simply be given to students?”
via LearnLab Explores Teaching and Learning – US News and World Report.
Entry filed under: Uncategorized. Tags: computing education, computing education research, educational psychology, worked examples.
Computing Ed Research - Guzdial's Take 
1.
jchoigt | April 4, 2012 at 11:17 am
Very interesting thoughts, but I’d like to see the supporting evidence that worked examples are better for students learning skills. I’ve seen the published evidence that retrieval practice is better than repeated study for factual recall, but I don’t know of studies comparing practice vs worked examples.
2.
Mark Guzdial | April 4, 2012 at 8:38 pm
Me too! Looking…
3.
Barry Brown | April 4, 2012 at 8:56 pm
http://ies.ed.gov/ncee/wwc/practiceguide.aspx?sid=1
This brochure from the Institute of Education Sciences lists several recommendations for improving student learning. Recommendation #2 is exactly what is being described: Interleave worked example solutions with problem-solving exercises. Contains references to research literature.
4.
Mark Guzdial | April 5, 2012 at 7:47 am
Thanks, Barry. I have the Trafton and Reiser paper that talks about the optimal interleaving of worked examples and practice. What I haven’t read yet is the evidence that Ken is talking about in this article: That practice helps with declarative learning (“facts”), and worked examples are better for developing procedural learning (“skills”). That’s an intriguing claim.
5.
chaikens | April 4, 2012 at 12:10 pm
In elementary computer programming, what are facts and what are skills? The skills include using facts. In less elementary programming, one important skill is looking up facts. To start at elementary programming, which facts are best learned from context free practice and which are best learned from their appearances in worked examples?
6.
chaikens | April 4, 2012 at 1:19 pm
On reading Mark’s 2009 post
https://computinged.wordpress.com/2009/10/02/question-everything-how-we-teach-intro-cs-is-wrong/
and replies, I’m reminded of those railroad-track syntax diagrams in some Pascal books. I taught that material in lower division post CS1 courses (like CS2) because it expresses elements of how computer scientists think and it introduces a particular classical CS idea (context free grammars in more attractive clothing), in addition to teaching how to deal with Pascal syntax. It appears that other CS ideas (such as object oriented models) have taken priority over CFGs for inclusion in our beginning courses. We all say CS1 is more than just programming and a particular language. Although I really liked syntax diagrams, I cannot honestly say how well they helped the students to learn to program.
Over time, and experience with more elementary students, I now think that learning a programming language really is like learning a second natural language. So examples and drill are more effective than instruction from principles, at the elementary level. But I do teach statements, informally, in terms of their parts (like the body of a loop or conditional, or the left side of an assignment) because I must convey an accurate way of thinking about what they mean…in addition to plenty of examples. Maybe questions in English, like “What happens immediately after the increment-part in a for-loop?” are effective in (either or both?) exercises and exams.
Whether or not the CFG idea is introduced in CS1, I think, is independent of the question of whether some level of formalistic discussion or technique applied to our teaching language’s grammar is effective to make our students capable of using that language. Certainly computer science majors are expected to be taught this topic and to apply it to programming, parsing, using modern specifications, theory, etc. eventually.
7.
Mark Guzdial | April 4, 2012 at 8:38 pm
Moti Ben-Ari showed me a great poster last summer at ITICSE. They had developed railroad-track grammars for Java, and they got the compiler to point out where on the railroad track the students had gone wrong. The poster was just on the technology, but I’d love to see a study of how effective that might be to help students understand syntax errors in Java.
8.
Jack Toole | April 4, 2012 at 11:42 pm
Do you have any examples about how to effectively use worked examples in CS? I’ve read a lot about their benefits on your blog, but I’m finding it difficult to imagine how to apply them concretely.
In specifics:
Where do you see worked examples fitting into a CS course’s curriculum? Do you use them during lecture? For discussion/labs? For homework assignments? Do you usually code them in front of students, or provide already-written sample functions on worksheets or homework assignments?
Specifically about writing code in front of students: How slow is this for you? Would it translate well to languages more verbose than Python? Our class uses C++ (it’s CS2, and arguable worth having exposure to before graduation), and whenever I write code in front of the class, it seems to go painfully slowly due to the typing overhead.
We have a couple worked examples provided as example functions on labs. The two problems that seem to mitigate their effectiveness are that students (a) often copy-paste the code for some of the examples, and (b) never read some of the examples. In both of these cases I doubt how much students are learning from them.
9.
Mark Guzdial | April 5, 2012 at 7:43 am
There are few examples of worked examples in CS Ed. The Pirolli work (and the follow-up work that he did with Mimi Recker) in the early 1980’s is the best: Most detailed, best evaluated. Caspersen and Bennedsen use worked examples as part of the design of their course. Leigh Ann Sudol DeLyser is using worked examples in her dissertation work at CMU. That’s about it. What we don’t have are design guidelines with empirical results, helping us to figure out what to use where.
Yes, when I code in front of class in lecture, it goes slow. Based on the questions I get, I think I’m finally going at their pace. I just did coded live in class yesterday (Java data structures), and got a bug that I couldn’t figure out before the end of class. But some students stayed after, and we figured it out. I think that was a good process.
10.
Jack Toole | April 5, 2012 at 8:38 am
Thanks! I’ll go look at those!
11.
Stephen Downes | April 5, 2012 at 7:39 am
If you want your programmer to learn how to do a bubble sort, give him a worked example of a bubble sort. Result: programmer efficiently learns how to create bubble sorts.
If you want your programmer to develop novel sorting algorithms to meet real-world design challenges, have him figure out bubble sorts for himself (may need a little scaffolding, depending on the programmer). Result: programmer less efficiently learns to bubble sort, but programmer *also* learns how to deal with novel situations (not on the bubble sort test, sadly, but a more valuable real-world skill).
Beware researchers pointing to instructional methods that are ‘more efficient’. They achieve this efficiency by focusing on simplified tasks and basic knowledge, at the expense of deeper real-world and problem-solving skills.
12.
Jack Toole | April 5, 2012 at 9:00 am
I have yet to read all the articles linked in these posts, but I think it’s possible to test that hypothesis with research: show students a worked example of bubble sort, then ask them a question about insertion sort, versus asking students a question about bubble sort followed by a question about insertion sort. (Of course, both questions and example would need a certain amount of scaffolding).
My instinct agrees with you (and I think, in any case, there’s probably value from both methods in differing circumstances). But my instinct would also have believed the world was flat. Science is important to question or validate both of these ideas. I think it would be a shame to conclude that, since research is imperfect, it should be abandoned altogether. CS education still is that way at many institutions – it’s why reading off 60-slide PowerPoint presentations often counts as “teaching” material. I think our current system errs far more to the side of not trusting research than to the side of blindly following it.
13.
Mark Guzdial | April 5, 2012 at 10:48 am
Stephen, how do you interpret the Sweller and Cooper results that students who do practice problems and those who see worked examples perform the same on transfer problems? That’s counter-intuitive for me, but follow-up studies have supported that finding (e.g., meta-analysis by Atkinson, Derry, et al. 2000). Teaching for transfer seems to be orthogonal to problem-solving or examples.
14.
Stephen Downes | April 5, 2012 at 11:41 am
Transfer problems aren’t the same as coming up with nvel or creative solutions. Transfer involves applying the same algorithm to new situations, when what we want is the development of new algorithms.
15.
Big-D “Design” for education and online courses: Let’s build more and different « Computing Education Blog | May 14, 2012 at 8:35 am
[…] was just talking about). The other one, Examplify, is about getting students to self-explain worked examples. Turadg’s thesis is practice-oriented and practical. For example, he actually figured out […]
16.
Instructional Design Principles Improve Learning about Computing: Making Measurable Progress « Computing Education Blog | June 5, 2012 at 7:30 am
[…] other educational psychology themes in here. We give them instructional material with a complete worked example. By calling out the mental model of the process explicitly, we reduce cognitive load associated […]
17.
Big Data vs. Ed Psychology: Work harder vs. work smarter | Computing Education Blog | January 31, 2014 at 1:35 am
[…] And on worked examples, […]
18.
Launching our Teacher Ebook for learning Python and CS Principles | Computing Education Blog | April 1, 2015 at 7:50 am
[…] ebook is hosted by Brad Miller’s Runestone tools and site. We use worked examples (as mentioned here) interleaved with practice, as Trafton and Reiser recommend. We have coding in the book as well […]