Measuring progress on CS learning trajectories at the earliest stages

I’ve written in this blog (and talked about many times) how I admire and build upon the work of Katie Rich, Diana Franklin, and colleagues in the Learning Trajectories for Everyday Computing project at the University of Chicago (see blog posts here and here). They define the sequence of concepts and goals that K-8 students need to be able to write programs consisting of sequential statements, to write programs that contain iteration, and to debug programs. While they ground their work in K-8 literature and empirical work, I believe that their trajectories apply to all students learning to program.

Here are some of the skills that appear in the early stages of their trajectories:

• Precision and completeness are important when writing instructions in advance. 
• Different sets of instructions can produce the same outcome. 
• Programs are made by assembling instructions from a limited set. 
• Some tasks involve repeating actions. 
• Programs use conditions to end loops.  
• Outcomes can be used to decide whether or not there are errors.
• Reproducing a bug can help find and fix it.
• Step-by-step execution of instructions can help find and fix errors.

These feel fundamental and necessary — that you have to learn all of these to progress in programming. But it’s pretty clear that that’s not true. As I describe in my SIGCSE keynote talk (the relevant 4 minute segment is here), there is lots of valuable programming that doesn’t require all of these. For example, most students programming in Scratch don’t use conditions to end loops — still, millions of students find expressive power in Scratch. The Bootstrap: Algebra curriculum doesn’t have students write their own iteration at all — but they learn algebra, which means that there is learning power in even a subset of this list.

What I find most fascinating about this list is the evidence that CS students older than K-8 do not have all these concepts. One of my favorite papers at Koli Calling last year was  It’s like computers speak a different language: Beginning Students’ Conceptions of Computer Science (see ACM DL link here — free downloads through June 30). They interviewed 14 University students about what they thought Computer Science was about. One of the explanations they labeled the “Interpreter.” Here’s an example quote exemplifying this perspective:

It’s like computers speak a different language. That’s how I always imagined it. Because I never understood exactly what was happening. I only saw what was happening. It’s like, for example, two people talking and suddenly one of them makes a somersault and the other doesn’t know why. And then I just learn the language to understand why he did the somersault. And so it was with the computers. 

This student finds the behavior of computers difficult to understand. They just do somersaults, and computer science is about coming to understand why they do somersaults? This doesn’t convey to me the belief that outcomes are completely and deterministically specified by the program.

I’ll write in June about Katie Cunningham’s paper to appear next month at the International Conference of the Learning Sciences. The short form is that she asked Data Science students at University to trace through a program. Two students refused, saying that they never traced code. They did not believe that “Step-by-step execution of instructions can help find and fix errors.” And yet, they were successful data science students.

You may not agree that these two examples (the Koli paper and Katie’s work) demonstrate that some University students do not have all the early concepts listed above, but that possibility brings us to the question that I’m really interested in: How would we know?

How can we assess whether students have these early concepts in the trajectories for learning programming? Just writing programs isn’t enough. 

• How often do we ask students to write the same thing two ways? Do students realize that this is possible?
• Students may realize that programming languages are “finicky” but may not realize that programming is about “precision and completeness.” 
• Students re-run programs all the time (most often with no changes to the code in between!), but that’s not the same as seeing a value in reproducing a bug to help find and fix it. I have heard many students exclaim, “Okay, that bug went away — let’s turn it in.” (Or maybe that’s just a memory from when I said it as a student…)

These concepts really get at fundamental issues of transfer and plugged vs unplugged computing education. I bet that if students learn these concepts, they would transfer. They address what Roy Pea called “language-independent bugs” in programming. If a student understands these ideas about the nature of programs and programming, they will likely recognize that those are true in any programming language. That’s a testable hypothesis. Is it even possible to learn these concepts in unplugged forms? Will students believe you about the nature of programs and programming if they never program?

I find questions like these much more interesting than trying to assess computational thinking. We can’t agree on what computational thinking is. We can’t agree on the value of computational thinking. Programming is an important skill, and these are the concepts that lead to success in programming. Let’s figure out how to assess these.

Making Desktops on the way to Dynabooks: Happy 80th Birthday to Alan Kay!

Yesterday was Alan Kay’s 80th birthday.

As Ben du Boulay and I wrote in our history of computing education research (see here), Alan and the Learning Research Group group at Xerox PARC asked a question in the 1970’s that has changed how all of us interact with computers, “How would we change computing to make it about learning?” Their efforts to build the Dynabook gave us the desktop user interface that we have today. It’s the greatest public impact of research in computing and learning.

Alan has been a mentor to me for years, and he still influences me, such as in the comments he contributes in this blog . I’m grateful for those opportunities to learn from and be inspired by him. I spoke in my SIGCSE 2019 Keynote (see the clip here) about how finding “Personal Dynamic Media” by Adele Goldberg and Alan (see link here) set the direction of my career. I first met Alan when he came to a workshop I helped organized in 1995. Here’s a picture that Ben Shneiderman took and shared with me:

That’s Alan on the right, and that’s me in the middle. The guy on the left is Wally Feurzeig, the guy who implemented the first Logo (with Danny Bobrow).

Wishing Alan happy days and many more years!

Thought Experiments on Why Face-to-Face Teaching Beats On-Line Teaching: We are Humans, not Econs

With everything moving on-line, I’m seeing more discussion about whether this on-line life might just be better. Amy Ko recently blogged (see post here) about how virtual conferences are cheaper, more accessible, and lower carbon footprint than face-to-face conferences, ending with the conclusion for her “it is hard to make the case to continue meeting in person.” My colleague, Sarita Yardi, has been tweeting about her exploration of “medium-independent classes” where she considers (see tweet here), “Trying to use the block of class time just because that’s how we’ve always taught seems like something to revisit. Less synchronous time; support short, frequent individual/small group interaction, less class time.”

It’s hard to do on-line education well. I used to study this kind of learning a lot (see post on “What I have learned about on-line collaborative learning”). I recently wrote about how we’re mostly doing emergency remote teaching, not effective on-line learning (see post here). I am concerned that moving our classes on-line will hurt the most the students who most need our help (see post here).

It should come as no surprise then that I don’t think that we know how to do on-line teaching or on-line conferences in a way that is anywhere close to the effectiveness of face-to-face learning. I agree with both Amy and Sarita’s points. I’m only focusing on learning outcomes.

Let me offer a thought experiment on why face-to-face matters. How often do you…

• Look at the movie trailer and not watch the movie.
• Watch the first few minutes of a show on Netflix but never finish it.
• Start a book and give up on it.
• Start watching a YouTube video and immediately close it or click away.

Now contrast that with: How often do you…

• Get up from a one-on-one meeting and walk out mid-discussion.
• Get up in the middle of a small group discussion and leave.
• Walk out of a class during a lecture.
• Walk out of a conference session while the speaker is still presenting (not between talks or during Q&A).

For some people, the answers to the first set are like the answers for the second set. I tried this thought experiment on my family, and my wife pointed out that she finishes every book she starts. But for most people, the first set is much more likely to happen than the second set. This is particularly hard for professors and teachers to recognize. We are good at self-regulated learning. We liked school. We don’t understand as well the people who aren’t like us.

There are a lot of people who don’t really like school. There are good reasons for members of minority groups to distrust or dislike school. Most people engage in higher-education for the economic benefit. That means that they have a huge value for the reward at the end, but they don’t particularly want to go through the process. We have to incentivize them to be part of the process.

Yes, of course, many students skip classes. Some students skip many classes. But the odds are still in favor of the face-to-face classes. If you are signed up for a face-to-face class, you are much more likely to show up for that class compared to any totally free and absolutely relevant to your interests lecture, on-campus or on-line. Enrolling in a course is a nudge.

For most people, you are much more willing to walk away from an asynchronous, impersonal event than a face-to-face, personal event. The odds of you learning from face-to-face learning are much higher simply because you are more likely to show up and less likely to walk out. It’s a great design challenge to make on-line learning opportunities just as compelling and “sticky” as face-to-face learning. We’re not there yet.

I would be all in favor of efforts to teach people to be more self-regulated. It would be great if we all were better at learning from books, lectures, and on-line resources. But we’re not. The learners with the best preparation are likely the most privileged students. They were the ones who were taught how to learn well, how to learn from school, and how to enjoy school.

Here’s a second thought experiment, for people who work at Universities. At any University, there are many interesting talks happening every week. For me, at least a couple of those talks each week are faculty candidates, which I am highly encouraged to attend. Now, they’re all on-line. How many of those did you attend when they were face-to-face, and how many do you attend on-line? My guess is that both are small numbers, but I’ll bet that the face-to-face number is at least double the on-line number. Other people see that you’re there face-to-face. There are snacks and people to visit with face-to-face. The incentives are far fewer on-line.

On-line learning is unlikely to ever be as effective as face-to-face learning. Yes, we can design great on-line learning, but we do that fighting against how most humans learn most things. Studies that show on-line learning to be as effective (or even more effective) than face-to-face classes are holding all other variables equal. But holding all other variables equal takes real effort! To get people to show up just as much, to give people as much (or more) feedback, and to make sure that the demographics of the class stay the same on-line or face-to-face — that takes significant effort which is invisible in the studies that are trying to just ask face-to-face vs on-line. The reality is that education is an economic endeavor. Yes, you can get similar learning outcomes, at a pretty high cost. At exactly the same cost, you’re unlikely to get the same learning outcomes.

We are wired to show-up and learn from face-to-face events. I would love for all of us to be better self-regulated learners, to be better at learning from books and from lecture. But we’re not Econs, we’re Humans (to use the Richard Thaler distinction). We need incentives. We need prompts to reflect, like peer instruction. We need to see and be seen, and not just through a small box on a 2-D screen.

SIGCSE 2020: Papers freely available, AP CSA over AP CSP for diversifying computing, and a tour of computing ed research in one hour

My Blog@CACM post for this month was about my first stop on my tour of SIGCSE 2020 papers (see link here). While the SIGCSE 2020 conference was cancelled, the papers are freely available now through the end of June — see all the proceedings here. I’ve started going through the proceedings myself. The obvious place to start such a tour is with the award-winning papers. My Blog@CACM post is on the paper from An Nguyen and Colleen M. Lewis of Harvey Mudd College on the negative impact of competitive enrollment policies (having students enroll to get into CS, or requiring a higher-than-just-passing GPA to get into the computing major) on students’ sense of belonging, self-efficacy, and perceptions of the department.

I said that this was the first stop on my tour, but that’s not really true. I’d already looked up the paper Does AP CS Principles Broaden Participation in Computing?: An Analysis of APCSA and APCSP Participants (see link here), because I’d heard about it from co-author Joanna Goode. I was eager to see the result. They show that AP CS Principles is effectively recruiting much more diverse students than the AP CS A course (which is mostly focused on Java programming). But, AP CS A students end up with more confidence in computing and much more interest in computing majors and tech careers. Maybe CSA students had more interest to begin with — there is likely some selection bias here. This result reminds me of the Weston et al result (see blog post here) showing that the female high school students they studied continued on to tech and computing majors and careers if they had programming classes.

I’ve been reading The Model of Domain Learning: Understanding the Development of Expertise (see Amazon link) which offers one explanation of what’s going on here. Pat Alexander’s Model of Domain Learning points out that domain knowledge is necessary to have sustained interest in a domain. You can draw students in with situational interest (having activities that are exciting and engage novices), but you only get sustained interest if they also learn enough about the domain. Maybe AP CSP has more situational interest, but doesn’t provide enough of the domain knowledge (like programming) that leads to continued success in computing.

In my SIGCSE 2020 Preview blog post (posted just two days before the conference was posted), I mentioned the cool session that Colleen Lewis was organizing where she was going to get 25 authors to present the entire 700+ page Cambridge Handbook of Computing Education Research in 75 minutes. Unfortunately, that display of organizational magic didn’t occur. However, in a demonstration of approximately the same level of organizational magic, Colleen got the authors to submit videos, and she compiled a 55 minute version (which is still shorter than reading the entire tome) — see it on YouTube here.

There are lots of other great papers in the proceedings that I’m eager to get into. A couple that are high on my list:

• Dual-Modality Instruction and Learning: A Case Study in CS1 from Jeremiah Blanchard, Christina Gardner-McCune, and Lisa Anthony from University of Florida, which provides evidence that a blocks-based version of Java leads to more and deeper version on the same assessments as students learning with textual Java (see link here).
• Design Principles behind Beauty and Joy of Computing by Paul Goldenberg and others. I love design principles papers, because they explain why the authors and developers were doing what they were doing. I have been reading Paul since back in the Logo days. I’m eager to read his treatment of how BJC works (see link here).

Please share in the comments your favorite papers with links to them.