## Using a Worksheet to Address a CS Education Problem

I just gave a midterm exam in my class on data structures using media computation.  The results are disappointing, and have prodded me into trying something new: I made a worksheet.

Two of the problems on the midterm were tracing problems.  One had a posterize method (reduces the possible number of colors in a picture), and I provided some RGB values for some sample pixels: What new RGB values will be in those pixels after this method executes?  The second problem had three graphics images, and three programs: Which program generated which image?  I had several people review the tracing problems — they weren’t trivial.

One of the problems was a code writing problem.  I gave them the Sound methods for reverse() and for increaseVolume().  They were to write a new method that reversed the first half the sound, then increased the volume in the second half the sound.  I literally expected them to copy the bodies of the two loops, and just tweak the ranges on the loops.

The average score on the two tracing problems (adding the percentages together and dividing by 200) was 96%.  15 of the 16 students in the class got over 90% on the two problems.  The average score on the one writing problem was 75%.  Five of the 16 students got less than 70% right.  These students are quite good at reading code.  Some of these students were unsuccessful writing code.

What happened?  The research literature on CS education has lots of examples of how reading and writing skills develop separately.  Pete Pirolli reported in the early 1980’s that, in several studies, Anderson’s lab at CMU found no correlation between reading and writing skills.  Raymond Lister and his colleagues have written several papers about students having one or the other, but not both sets of skills.  So, it’s not a surprising result, but it’s one that I need to address.

How?  The students are working hard on programming assignments.  Programming assignments are so heavy — they take so much time, they require so much effort.  I do require students to do a weekly out-of-class quiz, often oriented around videos.  This semester, two of the quiz assignments were to use Problets on for and while loops.  That may have helped with the reading skills, but doesn’t seemed to have helped much with writing skills.

I decided that I needed to try out one of the ideas in CSLearning4U, that is more examples and more lighterweight practice.  I made a worksheet for this week’s quiz. The worksheet has four completely worked out array manipulation programs (with outputs provided) and two more sound manipulation programs (for analogous actions as on the arrays), to make clear the connections.  There are five more programs that are nearly complete, including all outputs, but with some blanks. The blanks are mostly where range manipulations would take place.  The students have to complete the blanks.  I’m offering two points each for the 10 blanks, where the first 10 points go towards the quiz grade and the second possible 10 points are bonus points toward the midterm exam.

I made very clear that there was no partial credit.  The students get them right, or they don’t.  I explicitly said that the smartest thing to do is to type in the program, then fill in the blanks such that the output is right.  This is a lighterweight activity than programming from scratch.

This is unusual for me, and maybe for other higher-education CS teachers.  Worksheets are very common in high schools, and even in higher-education in other STEM disciplines.  I’ve seen physics labs and chemistry pre-labs in worksheet formats.  I can’t remember seeing them in CS, but I’m sure that somebody has done them.  What I’m doing here is something that K-12 teachers are taught to do: See a particular learning problem, then find or invent an activity to address that problem.  This is the first time I’ve created a worksheet as that activity.  In CS, we tend to assign some more programming to address a learning problem.  I’m looking for something with less cognitive load: More example, and more lightweight practice.

I’m still figuring out how I’ll evaluate if this worked.  My guess is that I’ll have a slightly more heavyweight version of this on next week’s quiz, e.g., I provide a partial program, with inputs and outputs specified, and I’ll ask them to fill in the rest of the program.  That will give me a sense if they’re developing the writing skills I hope to see.

Entry filed under: Uncategorized. Tags: , , , .

• 1. Software Carpentry » New Kinds of Content  |  February 14, 2012 at 9:11 am

[…] Guzdial, whose blog on CS education is always interesting, recently posted about using worksheets to help people learn to write programs. As he says, research going back 30 years shows that reading and writing skills develop […]

• 2. Cecily  |  February 14, 2012 at 10:22 am

I have done a few worksheets in my CS1 class. I wrote a bunch (3-4) of study guides to help students with the reading(we’re using Dietel and Dietel, objects late), and I wrote a multiple choice worksheet with a half a dozen questions on arrays to help prep them for our ABET assessment. Last semester study guides were mandatory, but I got really mixed feedback from the students, so this semester they can choose do do a study guide or an extra programming problem each unit. The vast majority of the students have chosen a programming problem. The ABET worksheet is still mandatory, but we haven’t gotten there yet. I totally agree that we need more examples. I currently do two-four problems with my students, and I think the most common negative feedback was they wanted more examples, so next semester I will be working towards 6-8 together plus six on their own.

As far as testing test questions, I have a TA that had the class last semester, and I am having her pre-test my test questions, and that is helping quite a bit with calibration. I am wondering who you had review the tracing questions- was it faculty or students, and if students was it recent students or advanced students?

• 3. Mark Guzdial  |  February 14, 2012 at 10:54 am

My TA took the class last semester, and she reviewed the exam. I also had Barb (my co-author on the text) review the exam, one of my PhD students (who teaches CS1 and CS2 at another local university), and a Sophomore who just took the CS version of this course. (My course is the non-majors data structures course.)

That’s really quite interesting that the students chose the option of the extra programming problem. What were the grade distributions like on the study guides? I wonder if the students perceive the worksheets as harder than the programming. My bet is that they take less time, but they may be more cognitively demanding.

• 4. nickfalkner  |  February 16, 2012 at 7:01 am

I was wondering how much you thought the structure of the worksheet helped to frame the information? As it stands, you have exemplars and then some simple worked exercises which, combined in this way, potentially reduce the cognitive burden through an easy-to-follow association.

Do you think that there would be any value in providing the exemplars, as is, and then moving the ‘fill-in-the-blanks’ activity to a small group or tutorial activity to allow people to work together for some of it (sharing their insights) and then completing the final activities separately? My apologies if this is close to what you’ve done and I’ve just misread it.

I was wondering about the perception problem as well. One of the burdens that we shoulder in our Puzzle Based Learning course is the number of students who cannot solve a problem because it’s impossible. (Until they are told that it isn’t) Almost any type of puzzle activity is seen as harder than a programming problem, even if the analytical difficulty of the programming exercise is far harder.

Thank you for a very interesting post.

• 5. Mark Guzdial  |  February 16, 2012 at 9:47 am

Hi Nick,

I don’t know much about designing worksheets for CS education. I know of very little literature on the subject. Probably the closest we have is the work on algorithm visualization, which is (in a sense) a scaffolded learning activity that is less than programming-from-scratch. There is a lot of literature on similar activities in STEM education. We need to borrow those lessons learned and try them in CS.

Sure, doing some kind of discussion may be useful. I couldn’t in this context because my “quizzes” are all out-of-class over the course of a week. While I’ve done work in CSCL, the most recent findings suggest that discussion doesn’t really buy you much. There was a paper at ICER (Beth Simon was a coauthor) that showed that the discussion phase of peer instruction doesn’t add anything to the learning (though I’m not quite sure about the research method), and Eric Mazur’s keynote pointed out that, in their work on predictions, discussion predictions didn’t help with learning.

Cheers,
Mark

• […]  Me dará un sentido si están desarrollando las habilidades de escritura que espero ver.. version original aqui […]

• 7. Leigh Ann Sudol-DeLyser  |  February 20, 2012 at 6:45 pm

I’m a strong believer not in worksheets, but in structured practice.

Let me explain. Consider calculus problem sets. Normally the problems start with one-offs, problems that come from the textbook or the notes with a small change. Lets think about our introductory problem sets for a minute. How many of them are one-offs? If you take into account the cognitive load of the syntax, prior un-perfectly learned material, and then try and combine those items in new ways.. well what you get is often “I guess this is right” by the student. I’ve been doing a lot of think-aloud protocols with CS1 students for my thesis work and I can tell you that the weaker students do not have enough working memory to represent even a 6-7 line algorithm at one time. They see the PARTS of the program, not the connected whole. I hope in my thesis to connect the parts for students through feedback when they struggle.

Also, I am not surprised by the low correlation studies for reading and writing code – but I’m not sure how many of them were rigorous in the structure of the code they were testing. How many students were asked to read and write the same algorithms? From prior studies I’ve learned that changing even one small piece of an algorithm can (even within just code reading exercises) break statistical methods that rely on those correlations. Our problems often involve too many skills and features to look at the same way as other domains.

When I was teaching HS I engaged in “worksheet making” structured practice, and later as a college professor I used my assignments that way. Each assignment had 1-2 problems that were just one-off from either the textbook or the notes – and referred to the starting code for students! They also involved reflective questions – why was this different and what did you have to change? The assignments went on to move further from given examples, but because we show it once (or twice) in class does not mean students internalized it. Especially with multiple components. Although our working memory can do 7 plus or minus 2, remember that as experts we “chunk” things and while the for loop to traverse an array may count as 1 in our working memory, odds are that it counts as 4 in our novices.

(Yes Mark – I’ll let you know when this gets published 😉

• 8. Mark Guzdial  |  February 21, 2012 at 10:16 am

Pirolli’s work was rigorous in making sure that the reading and writing was similar — that was the main point of his paper. Isn’t structure practice a kind of worksheet activity? I just did a Blog@CACM piece on this theme, where I defined “worksheets” pretty broadly. What definition do you use of each?

• […] Teaching is a great job.  I particularly appreciate how teaching keeps me thinking and questioning, which is particularly important for an education researcher.  I’m teaching two classes this semester.  I’ve mentioned recently how my data structures class has me thinking about new kinds of practice activities. […]

• […] and World Report on the LearnLab work at Carnegie Mellon University.  Since I’m exploring worked examples research and the implications for CS Education these days, I found the below section of the interview with Ken Koedinger intriguing.  Practice […]

This site uses Akismet to reduce spam. Learn how your comment data is processed.