How do we encourage retention of knowledge in computing?

June 19, 2012 at 8:30 am 18 comments

The scenario described in the experiment below has been repeated many times in the education literature: Students are asked to read some material (or listen to a lecture), and are then asked to do something with that material (e.g., take a quiz, write down everything they can remember, do a mind-mapping exercise), and some time later, they take a test to measure retention. In the experiment described below, simple writing beat out creating a mental map. Interesting, but it’s an instance of a case that I wanted to raise.

This pattern of information+activity+retention is common, and really does work. Doing something with the knowledge improves retention over time.

So how do we do this in computer science? What do we ask our students to do after lecture, or after reading, or after programming, to make it more likely that they retain what they learned? If our only answer is, “Write more programs,” then we missed the point. What if we just had our students write down what they learned? Even if it was facts about the program (e.g., “The test for the sentinel value is at the top of the loop when using a WHILE”), it would help to retain that knowledge later. What this particular instance points out is that the retention activity can be very simple and still be effective. Not doing anything to encourage retention is unlikely to be effective.

But two experiments, carried out by Dr Jeffrey Karpicke at Purdue University, Indiana, concluded that this was less effective than constant informal testing and reciting.Dr Karpicke asked around 100 college students to recall in writing, in no particular order, as much as they could from what they had just read from science material.Although most students expected to learn more from the mapping approach, the retrieval exercise actually worked much better to strengthen both short-term and long-term memory.The results support the idea that retrieval is not merely scouring for and spilling out the knowledge stored in one’s mind — the act of reconstructing knowledge itself is a powerful tool that enhances learning about science.

via Old school teaching better for retaining knowledge – Telegraph.

Entry filed under: Uncategorized. Tags: , .

Good academic leadership as a model for good teaching 25 years of HyperCard—the missing link to the Web | Ars Technica

18 Comments Add your own

  • 1. David Wees  |  June 19, 2012 at 9:09 am

    I remember when this study came out. Not tested: How well students used the knowledge they learned in other contexts than a test.

    My suspicion is that you need a balance of writing your own code, and learning how other people have solved their code problems. Imagine an extreme example of this pedagogical technique – students spend none of their time constructing code and all of their time learning through the retrieval method. Would they be able to program anything themselves later?

    Personally, I think your pedagogical technique has to match your goals at the end.

    • 2. Mark Guzdial  |  June 19, 2012 at 9:49 am

      Sweller and Pirolli suggest that you can do just that: Study other people’s code, and then successfully code on the test. Sweller did this with algebra: reviewing worked examples only, then successful algebra problem-solving on the test. (He did test for transfer — being able to use the knowledge elsewhere — but both traditional and worked examples algebra students did badly on that.) Pirolli taught recursion in Lisp, and he intermixed some problem-solving with reviewing worked examples, because there is evidence that some problem-solving intermixed with worked examples is optimal. Mimi Recker used self-explanations with worked examples to get at understanding and retention: Have the students explain the code to themselves (or presumably, to someone else). I’m planning to use a worked examples with self-explanations technique this summer, and based on what I read in the literature, I predict that students will be better prepared for coding (than lecture + textbook alone). This is just me teaching a class, not a study, but I’m interested to see how the students react and how well they perform.

    • 3. Bonnie  |  June 19, 2012 at 12:25 pm

      I agree with this. I give lots of mini quizzes in my CS1 and CS2 courses. Generally the students do quite well on them, but then fall apart during the in-class programming assignments. They can spout back facts, but don’t know how to use them. I haven’t tried having them read programs in any kind of structured way, although I am aware of the idea and may try to use it in the future. I expect MASSIVE resistance though.

  • 4. Steve Thomas  |  June 19, 2012 at 9:49 am

    First thanks for the link to Carl Wieman in your previous post. Found a lecture of his here: which really caused me to rethink parts of how I teach.

    I would change your question from what do we do “after” the lecture/reading/programming, to what do we ask them to do “during” the these activities. I think some form of Wieman’s “Clicker” approach (which could be done with laptops) with not just multiple choice, but also some open ended questions as well.

    Another thing I think would be useful is to be able to easily “watch” all the students as they program. I was teaching some kids via Google hangout and I had them all “screen share” and I was able to watch and see certain common mistakes or areas they were struggling with and use that as the next thing to teach. Now two things, it was a small class 5 kids and they were using Etoys so it was easier to “see” what they were doing because its visual scripting tiles. So not sure how to scale this.

    I am starting to look through Carl Wieman’s some of his CWSEI information ( do you have any other references to concrete examples of questions to ask and things to do during (and after) lectures that is effective, that would be much appreciated.

  • 6. gasstationwithoutpumps  |  June 19, 2012 at 5:32 pm

    I agree with other commenters here. I don’t care if students can regurgitate “The test for the sentinel value is at the top of the loop when using a WHILE.” What I care about is whether they can figure out when looping is needed and correctly construct loops without off-by-one errors. I don’t think that such memory exercises and testing help much with the skills I’m interested in seeing them develop. Unfortunately, the skills I’m interested in are extremely difficult to test in a short 3-hour test, so are unlikely to be the subject of any educational research.

    • 7. Mark Guzdial  |  June 19, 2012 at 6:35 pm

      Here’s a thought experiment. A student struggles with an assignment, sometimes even doing a syntactic random walk to get it to compile. Finally, after many iterations at both compilation and execution, the program works. Along the way, the student figured out a bunch of things that we all want the student to know–not about syntax, but about how the program needs to be structured to work.

      How do we get the student to retain that knowledge? “She will just remember, because of the struggle to get there” is unlikely to be correct, given what we know about memory and learning. How do you get students to highlight and remember the lessons learned?

      Briana Morrison is doing an experiment this summer that will be exploring how novices read code. We have lots of evidence that they don’t notice the same things as experts. If that’s true, why should we expect them to notice and remember the important lessons from their own programs?

      Retention isn’t about remembering syntax rules for a test. Retention is remembering, period.

      • 8. gasstationwithoutpumps  |  June 19, 2012 at 8:32 pm

        Why are assignments being given that can be solved by random walks in code space?

        What knowledge do you want students to retain? Is it codifiable with the sort of statements you used as examples? (I don’t believe so, but could be convinced otherwise—particularly if the knowledge you want students to retain is different from the skills I’m thinking of.)

        I do think that there is value in having students write stuff down, though I think that a lot of the benefit comes from having to formulate it clearly, rather than from simple recall exercises. That is why I feel that it is essential that students learn to write good comments from the beginning of learning to program—so that they capture their thinking about the variables and algorithms in a way that is distinctly different from the program. Unfortunately, most of the examples of commenting they see are truly horrible—designed to teach about the syntax of the programming language rather than being exemplars of clear documentation.

        So if I were to do an exercise trying to get students to retain more, I would concentrate on improving their in-code documentation. Of course, that means that a competent programmer who understands how to write in-program documentation has to read it and provide feedback, and many of our TAs and CS professors are not up to the task—based on the crappy feedback most students seem to have gotten on the programs they’ve turned in for prior courses.

        • 9. Mark Guzdial  |  June 19, 2012 at 9:04 pm

          I used to teach really large classes (150-300). I never meant for any assignment to be solvable by a random walk in code space. With that many students, some students will try. I confused the point by mentioning that. My main point is that students learn something, from some process (even if random), when programming. If not, why did we make the assignment at all?

          How do we get students to retain the knowledge they just gained? I agree that writing comments is one form of reflection — if you write them after the code.

          Retention comes from reflection, from practice, from revisiting ideas. If you give several programming assignments that address the same learning objectives, then absolutely — that is likely to work for achieving retention. If not, what else do we do?

          I’m raising the issue in this post. There is little empirical data about retention in computer science. Most SIGCSE Symposium or ICER papers don’t measure learning at all, let alone retention some time later. Lauren’s paper is one of the few. I suspect that it is likely that most of us don’t do anything explicitly to encourage retention, and whatever we are doing, there is probably little empirical evidence that it works.

          • 10. katrinafalkner  |  June 21, 2012 at 7:46 pm

            I think reflection, as you state above, is the key here – getting students to think about what they learnt from what they have just completed. One thing we have been doing is to get our students who have just completed first year to start writing down advice for someone just starting – what would they do differently if they were to start again, and what is the most important thing that they have discovered from their experiences. While this has already been interesting, I’d like to expand this further to have our students do this as a regular activity – and sharing this knowledge with others is a strong motivator.

          • 11. mrstevesscience  |  June 23, 2012 at 5:46 pm

            Agree with Katrina reflection is the key and I like your idea of asking the kids to write down advice for someone who just started. Does anyone else have any suggestions to get kids in the habit of reflection?

          • 12. Cecily  |  June 26, 2012 at 2:36 am

            I make my students do a final project presentation on the last day of class. The first semester, they kind of didn’t say too much, so the second semester I had them all answer a few questions. This has been a great “reflection” activity for most of them after writing their first serious piece of code. They seem to learn a lot, and it is interesting to hear the vastly different lessons they learn from the same assignment(e..g I learned I want to be a CS major(instead of a math major) vs. I learned I do NOT want to be a CS major(instead of a math major). They also tell what the hardest part was, what they learned(how important sequencing code is, when to ask for help, importance of mentors, etc.)

  • 13. guy  |  June 20, 2012 at 10:22 am

    1945, How to Solve It, G. Polya

    The final step: Looking Back – Examine the solution obtained…

    and, regarding comments, I find that promoting very meaningful procedure and variable names to be more important than comments, unless the comments bring to light stuff that’s not obvious in the code.

  • 14. guy  |  June 21, 2012 at 10:26 am

    I have a little more time today, and after revisiting my copy of How to Solve It, I thought that I should have included a bit from the text.

    First, all four phases: (1) understanding the problem, (2) devising a plan, (3) carrying out the plan, and (4) looking back.

    Even though Polya’s book is about solving math problems, his phases can be applied to introductory programming exercises. Here is his first paragraph of the description of the “looking back” phase.

    Even fairly good students, when they have obtained the
    solution of the problem and written down neatly the argument,
    shut their books and look for something else. Doing so, they
    miss an important and instructive phase of the work. By
    looking back at the completed solution, by reconsidering and
    reexamining the result and the path that led to it, they could
    consolidate their knowledge and develop their ability to solve
    problems. A good teacher should understand and impress on
    his students the view that no problem whatever is completely
    exhausted. There remains always something to do; with
    sufficient study and penetration, we could improve any
    solution, and, in any case, we can always improve our
    understanding of the solution.

    He then follows this with more that in introductory programming would be testing/verifying the solution and even examining the elegance of the solution (i.e., can the solution be reduced, made clearer, …).

    So, yes… students reflecting on what they have just done, maybe as comments in their code, is probably very valuable…

    • 15. gasstationwithoutpumps  |  June 21, 2012 at 6:27 pm

      I think that what is needed is more along the lines of a writing course: after the assignment is turned in and feedback added, students get it back to improve. Going through several drafts of the program until it is good (as opposed to marginally functional) would be a major change for most CS classes.

  • 16. Bri Morrison  |  June 21, 2012 at 5:42 pm

    Given a lot of students today, I wonder if the results are any different if they *type* down what they remember versus writing it down (physically, pen and paper old school style). Maybe I’m just old (or old-school) but I do believe there’s something to the “in the ear, use the hand to put to paper, remembered by the brain”. And I just don’t know about typing…same internal memory triggers?

    • 17. gasstationwithoutpumps  |  June 21, 2012 at 6:26 pm

      Yes, I believe that typing is just as effective as hand writing for building retention, and it is less likely to cause writer’s cramp.

  • […] How do we encourage retention of knowledge in computing? « Computing Education Blog. […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,052 other followers


Recent Posts

Blog Stats

  • 2,030,790 hits
June 2012

CS Teaching Tips

%d bloggers like this: