Subgoal labelling influences student success and retention in CS

I have written a lot about subgoal labeling in his blog. Probably the best way to find it all is to see the articles I wrote about Lauren’s graduation (link here) and Briana’s (link here). They have continued their terrific work, and have come out with their most impressive finding yet.

In work with Adrienne Decker, they have shown that subgoal labeling reduces the rate at which students fail or drop out of introductory computer science classes: “Reducing withdrawal and failure rates in introductory programming with subgoal labeled worked examples” (see link here). The abstract is below.

We now have evidence that subgoal labelling lead to better learning, better transfer, work in different languages, work in both text and block-based programming languages, and work in Parsons Problems. Now, we have evidence that their use leads to macro effects, like improved student success and retention. We also see a differential impact — the students most at risk of failing are the ones who gain the most.

This is a huge win.


Background: Programming a computer is an increasingly valuable skill, but dropout and failure rates in introductory programming courses are regularly as high as 50%. Like many fields, programming requires students to learn complex problem-solving procedures from instructors who tend to have tacit knowledge about low-level procedures that they have automatized. The subgoal learning framework has been used in programming and other fields to breakdown procedural problem solving into smaller pieces that novices can grasp more easily, but it has only been used in short term interventions. In this study, the subgoal learning framework was implemented throughout a semester-long introductory programming course to explore its longitudinal effects. Of 265 students in multiple sections of the course, half received subgoal-oriented instruction while the other half received typical instruction.

Results: Learning subgoals consistently improved performance on quizzes, which were formative and given within a week of learning a new procedure, but not on exams, which were summative. While exam performance was not statistically better, the subgoal group had lower variance in exam scores and fewer students dropped or failed the course than in the control group. To better understand the learning process, we examined students’ responses to open-ended questions that asked them to explain the problem-solving process. Furthermore, we explored characteristics of learners to determine how subgoal learning affected students at risk of dropout or failure.

Conclusions: Students in an introductory programming course performed better on initial assessments when they received instructions that used our intervention, subgoal labels. Though the students did not perform better than the control group on exams on average, they were less likely to get failing grades or to drop the course. Overall, subgoal labels seemed especially effective for students who might otherwise struggle to pass or complete the course.

Keywords: Worked examples, Subgoal learning, Programming education, Failure rates

June 29, 2020 at 7:00 am 7 comments

Managing CS major enrollment boom with a lottery: “A lottery, by definition, is fair.”

I am excited to see that the University of California, San Diego is now managing their over-enrollment in the computer science major with a lottery — see the article here.

Instead of enrolling students holistically or based on GPA, the department selects at random — assuming they exceed the 3.3 CSE GPA threshold. With the lottery system, all students are equally considered despite differences in their experience, drive, and ability.

When asked about the implications of the new system — and possible disadvantage to high-performing students — CSE Chair Dean Tullsen explained, “a lottery, by definition, is fair.”

“I think there’s this false assumption that the students who work harder are the ones who are getting the 4.0s, that hard work directly translates to a higher grade. [The lottery system will] admit a lot of hard-working students who weren’t getting in before,” CSE Vice-Chair for Undergraduate Education Christine Alvarado added.

This is a much more fair system than simply allowing in the top GPA students. It probably doesn’t make Tech companies happier, but it’s not clear that it makes them less happy. They will still get lots of potential employees who are above the bar. Those employees will likely be more diverse than the graduates being produced from CS programs today. The students getting the top grades in the early classes are typically those with more opportunity to learn CS, more wealth, and more privilege. A lottery says that anyone who is prepared for the courses can take them.

June 22, 2020 at 7:00 am 22 comments

Why do students study computing, especially programming

Alan Kay asked me in a comment on my blog post from Monday:

You and your colleagues have probably done a survey over the years, but it would be useful to see one or two examples, and especially one from the present time of “why are you currently studying computing, especially programming?”

It would be illuminating — and very important — to see the reasons, and especially the percentage who say: “to learn and understand and do computing and programming”.

A half-dozen papers sprang to mind. Rather than type them into a teeny tiny response box, I’m going to put them here. This is not a comprehensive survey. It’s the papers that occurred to me at 8:30 am EDT in response to Alan’s query.

The biggest recent study of this question is the Santo, Vogel, and Ching white paper CS for What? Diverse Visions of Computer Science Education in Practice (find it here). This paper is particularly valuable because it starts K-12 — what are the reasons for studying CS in school?

The most recent paper I know on this topic (and there were probably new ones at RESPECT and SIGCSE 2020 that I haven’t found yet) is this one from Koli, It’s like computers speak a different language: Beginning Students’ Conceptions of Computer Science (find it here). I liked this one because I most resonated with the “Creator” perspective, but I design today for those with the “Interpreter” perspective.

Alan particularly asked what we had done in our group. We started asking these questions when we were doing Media Computation (here’s a 2005 paper where we got those answers from Georgia Tech and Gainesville College students — GT students mostly wanted to know how to use a computer better and then get a good grade, while Gainesville students wanted to know what programming was). We got different answers from the follow-on course MediaComp Data Structures course where we started seeing a real split in views (see Lana Yarosh’s paper here). When we were doing “Georgia Computes!”, we did a statewide survey in 2010 to understand influences on students’ persistence in CS (see paper here). This is important to read, to realize that it’s not just about ability and desire. Women and BIPOC students are told that they don’t belong, and they need particular attention and encouragement to get them to go on, even if they believe they could and should. Probably the study from my group most explicitly on this question is Mike Hewner’s Undergraduate conceptions of the field of computer science (see paper here).

Two of my former students, but not with me, developed a validated computing attitudes survey (Tew, Dorn, and Schneider, paper here). Here, they ask experts what CS is, then use the same instrument to ask students what CS is, so that they can compare how “expert-like” the students answers are.

Not from my research group, but a really important paper in this space is Maureen Biggers et al’s Student perceptions of computer science: a retention study comparing graduating seniors with CS leavers (see link here). Most studies look at those who stay in CS. Maureen and her team also interviewed those who left, and how their perception of the field differed.

There are so many others, like the “Draw a Computer Scientist” task to elicit what the field is about (see example paper here and here). I particularly like Philip Guo’s groups paper on “conversational programmers” — people who study programming so that they can talk to programmers, not to ever program or to understand programming (see CHI’16 paper here).

Here’s what I think is the main answer to Alan’s question: Yes, it’s changed. But there are some specific answers that are consistent. Yes, some want to learn programming. Others want to learn programming as part of general computer skills.

I’m going to stop here and push this out before my meetings start for the day. I invite readers to add their favorite papers answering this question below.

June 17, 2020 at 9:03 am 12 comments

How do teachers teach recursion with embodiment, and why won’t students trace their programs: ICLS 2020 Preview

This coming week was supposed to be the International Conference of the Learning Sciences (ICLS) 2020 in Nashville (see conference website here). But like most conferences during the pandemic, the face-to-face meeting was cancelled (see announcement here). The on-line sessions are being announced on the ICLS2020 Twitter feed here.

I’m excited that two of my students had papers accepted at ICLS 2020. I haven’t published at ICLS since 2010. It’s nice to get back involved in the learning sciences community. Here’s a preview of their papers.

How do teachers teach recursion with embodiment

I’ve written here about Amber Solomon’s work on studying the role of space and embodiment in CS learning. This is an interesting question. We live in a physical world and think in terms of physical things, and we have to use that to understand the virtual, mostly invisible, mostly non-embodied world of computing. At ICER 2018, she used a taxonomy of gestures used in science learning to analyze the gestures she saw in a high school computer science classroom (see link here). Last summer at ITiCSE, she published a paper on how making CS visible in the classroom (through gesture and augmented reality) may reduce defensive climate (see link here). In her dissertation, she’s studying how teachers teach recursion and how learners learn recursion, with a focus on spatial symbol systems.

Her paper at ICLS 2020 is the first of these studies: Embodied Representations in Computing Education: How Gesture,Embodied Language, and Tool Use Support Teaching Recursion. She watched hours of video of teachers teaching recursion, and did a deep dive on two of them.

I’m fascinated by Amber’s findings. Looking at what teachers say and gesture about recursion from the perspective of physical embodiment, I’m amazed that students ever learn computer science. There are so many metaphors and assumptions that we make. One of the teachers says, when explaining a recursive function:

“Then it says “… “now I have to call.”

Let’s think about this from the perspective of the physical world (which is where we all start when trying to understand computing):

  • What does it mean for a function to “say” something?
  • The function “says” things, but I “call”? Who is the agent in this explanation, the function or me? It’s really the computer with the agency, but that doesn’t get referenced at all.
  • Recursion is typically explained as a function calling itself. We typically “call” something that is physically distant from us. If a function is re-invoking itself, why does it have to “call” as if at a distance?

For most computer scientists, this may seem like explaining that the sky is blue or that gravel exists. It’s obvious what all of this means, isn’t it? It is to us, but we had to learn it. Maybe not everyone does. Remember how very few students take or succeed at computer science (for example, see this blog post), and what enormously high failure and drop-out rates we have in CS. Maybe only the students who pick up on these metaphors are the ones succeeding?

Why won’t students trace their programs?

Katie Cunningham’s first publication as a PhD student was her replication and extension of the Leeds Working group study, showing that students who trace program code successfully line-by-line are able to answer more accurately questions about the code (see blog post here). But one of her surprising results was that students who start tracing and give up do worse on prediction questions than those students who never traced at all. In her ITICSE 2019 paper (see post here), she got the chance to ask those students who stopped tracing why they did. She was extending that with a think-aloud protocol, when something unusual happened. Two data science students, who were successful at programming, frankly refused to trace code.

Her paper “I’m not a computer”: How identity informs value and expectancy during a programming activity is an exploration of why students would flat out refuse to trace code — and yet successfully program. She uses Eccle’s Expectancy Value Theory (which comes up pretty often in our thinking, see this blog post) to describe why the cost of tracing outweighs the utility for these students, which is defined in terms of their sense of identity — what they see themselves doing in the future. Sure, there will be some programs that they won’t be able to debug or understand because they won’t trace line-by-line. But maybe they’ll never actually have to deal with code that complex. Is this so bad?

Katie’s live session is 2:00-2:40pm Eastern time on June 23. The video link will be available on the conference website to registered attendees. A pre-print version of her paper is available here.

Both of these papers give us new insight into the unexpected consequences of how we teach computing. We currently expect students to figure out how their teachers are relating physical space and computation, through metaphors that we don’t typically explain. We currently teach computing expecting students to be able to trace code line-by-line, though some students will not do it (and maybe don’t really need to). If we want to grow who can succeed at computing education, we need to think through who might be struggling with how we’re teaching now, and how we might do better.

June 15, 2020 at 7:00 am 50 comments

Becoming anti-racist: Learning about race in CS Education

I don’t usually invite external review on my blog posts for CACM, but I did this month because it’s such an important topic and I know too little about it — “CS Teachers, It’s (Past) Time To Learn About Race” (see link here). Many thanks to Melissa Perez, Carl Haynes, Leigh Ann DeLyser, Betsy DiSalvo, Leo Porter, Chad Jenkins, Wes Weimer, Barbara Ericson, Matthew Guzdial, Katie Guzdial, and Manuel Perez Quinones.

We have to change CS Education. We do not talk enough about BIPOC (Black, Indigenous, and People of Color) students and faculty in CS education. We have to reflect that Black Lives Matter in our teaching practice. We have to become explicitly anti-racist (a term I just learned this last week, see the book link here) — actively seeking to address historic and systemic inequities (see piece at CNN too).

One of the reviewer’s comments was that, by offering some small suggestions (like changing how we grade), I might dissuade people from bigger changes. It’s a valid concern. I’m hoping that people will take me at my word: I’m just learning here, and I hope that you will educate me (and each other) by sharing other ideas and resources. Please do share more ideas in the comments to this post.

Here are a few more that have come my way since that post that I wanted to share:

  • Ron Eglash has written up a terrific list of strategies for address issues of racism in technology — see link here.
  • Melissa Perez, a PhD student working with Barb Ericson, pointed out that it’s not enough to bring more people into CS education if we don’t change what we’re doing in CS. For example, we have to consider the problem of using biased training data for machine learning training. She recommends this article for considering the ethics of what we do in CS, besides how we teach CS. We need to integrate ethics across CS education.
  • Carl Haynes, also a PhD student working with Barb, recommends this book on intersectionality (see link here).
  • Manuel Perez Quinones recommends this Best Paper awardee from CHI 2020 on “Critical Race Theory for HCI” (see link here).
  • Kamau Bobb gave a talk at CornellTech in January “Unpacking Equity: To Code + Beyond” which is available on YouTube here. (Thanks to Leigh Ann DeLyser and Dianne Levitt for this.)
  • Patricia Garcia, who does terrific work on helping underserved students author their computational identities, recommends this video on Black Lives Matter myths debunked.
  • The University of Michigan’s School of Information has been having an amazing online discussion about how to make their education anti-racist. A book that stood out on the resources shared there was Stamped from the Beginning: The definitive history of racist ideas in America by Ibram Kendi (Amazon link here).
  • I had my children read the CACM blog post, and they gave me valuable comments on it. My daughter, Katie, a science teacher in Detroit Public Schools suggested these three books: The Color of Law: A forgotten history of how our government segregated America by Richard Rothstein (link), We Want to Do More Than Survive: Abolitionist Teaching and the Pursuit of Educational Freedom by Bettina Love (Amazon link), and Why Are All the Black Kids Sitting Together in the Cafeteria? by Beverly Daniel Tatum (Target link).

The issues of race, culture, and members of underserved groups are particularly critical for us to consider as we move into the 2020-2021 academic year during a worldwide pandemic. As we move classes on-line, we are putting at a greater disadvantage underserved populations. We have to be sensitive and thoughtful that our response to pandemic doesn’t exacerbate our existing structural inequities. Let’s worry less about cheating, and more about

  • how to help students taking our classes remotely who don’t have laptops, or who have to share a single laptop with a family, or who don’t have broadband Internet access;
  • how to help students who can’t come to class because they would be put at risk;
  • how to help students who have hearing disabilities and won’t be able to read lips if a teacher is wearing a mask (thanks to Bonnie MacKellar for pointing out that concern).

We have privilege and resources, and we should use them to address inequities.

TL;DR: I know too little about race, and I have not considered the historic and systemic inequities in CS education when I make my daily teaching decisions. I haven’t read all of the above, but I’m working on it daily. Please do share valuable resources you have found in the comments. Let’s learn about race in CS education and make change to improve learning for everyone.

June 8, 2020 at 7:00 am 13 comments

Goals for CS Education include Getting Students In the Door and Supporting Alternative Endpoints

ACM Inroads has published an essay by Scott Portnoff “A New Pedagogy to Address the Unacknowledged Failure of American Secondary CS Education” (see link here). The Inroads editors made a mistake in labeling this an “article.” It’s an opinion or editorial (op-ed) piece. Portnoff presents a single perspective with little support for his sometimes derogatory claims. I have signed a letter to the editors making this argument.

Portnoff is disparaging towards a group of scholars that I admire and learn from: Joanna Goode, Jane Margolis, and Gail Chapman. He makes comments about them like “had CSEA educators been familiar with both CS education and the literature.” Obviously, they are familiar with the research literature. They are leading scholars in the field. Portnoff chides the CSEA educators for not knowing about the “Novice Programmer Failure problem” — which is a term that I believe he invented. It does not appear in the research literature that I can find.

In this blog, I want to try to get past his bluster and aggressive rhetoric. Let’s consider his argument seriously.

In the first part, he suggests that current approaches to secondary school CS education in the United States are failing. His measure of success is success rates on the Advanced Placement Computer Science Principles exam. He also talks about going on to succeed in other CS courses and about succeeding at industry internships, but he only offers data about AP CSP.

He sees the reason for the failure of US CS education in high school is that we have de-emphasized programming. He sees programming as being critical to success in the AP exams, in future CS classes, and in industry jobs. Without an emphasis on programming, we will likely continue to see low pass rates on the AP CS Principles exam among female and under-represented minority students.

In the second part, Portnoff lays out his vision for a curriculum that would address these failings and prepare students for success. He talks about using tools like CodingBat (see link here) so that students get enough practice to develop proficiency. He wants a return to a focus on programming.

What Portnoff misses that there is not consensus around a single point of failure or a set of goals about CS Education. In general, I agree with his approach for what he’s trying to do. I value the work of the CSEA educators because the problems that they’re addressing are harder ones that need more attention.

The biggest problem in US high school CS education is that almost nobody takes it. Less than 5% of US high school students attend any CS classes (see this blog post for numbers), and the students we currently have are overwhelmingly male, white/Asian, and from wealthier schools. Of course, we want students to succeed at the Advanced Placement exams, at further CS courses, and at industry jobs. But if we can’t get students in the door, the rest of that barely matters. It’s not hard to create high-quality education only for the most prepared students. Getting diverse students in the door is a different problem than preparing students for later success.

CSEA knows more about serving students in under-served communities than I do. They know more about how to frame CS in such a way that principals will accept it and teachers will teach it. That’s a critical need. We need more of that, and we probably need a wide range of approaches that achieve those goals.

A focus on programming is critical for later success in the areas that Portnoff describes. The latest research supporting that argument comes from Joanna Goode (as I described in this blog post), one of the educators Portnoff critiques. Joanna was co-author on a paper showing that AP CS A success is more likely to predict continuation in CS than AP CSP success. I’m also swayed by the Weston et al. article showing that learning to program led to greater retention among female students in the NCWIT Aspirations awards programs (see link here).

I also agree with Portnoff that learning to program requires getting enough practice to achieve some level of automaticity. CodingBat is one good way to achieve that. But that takes a lot of motivation to keep practicing that long and hard. We achieve reading literacy because there are so many cultural incentives to read. What will it take to achieve broad-based programming literacy, and not just among the most privileged? Portnoff tells us that his experience suggests that his approach will work. I’m not convinced — I think it might work with the most motivated students. He teaches in the same school district where the ExploringCS class was born. But Portnoff teaches in one of LAUSD’s premier magnet schools, which may mean that he is seeing a different set of students.

An important goal for CS Education is to get students in the door. I’m not sure that Portnoff agrees with that goal, but I think that many involved in CS education would. There is less consensus about the desired outcomes from CS education. I don’t think that CSEA has the same definition of success that Portnoff does. They care about getting diverse students to have their first experience with computer science. They care about students developing an interest, even an affinity for computing. They care more about creating a technically-informed citizenry than producing more software developers. Portnoff doesn’t speak to whether CSEA is achieving their desired outcomes. He only compares them to his goals which are about continuing on in CS.

There is a tension between preparing students for more CS (e.g., success in advanced classes and in jobs) and engaging and recruiting students. In a National Academy study group I’m working in, we talk about the tension between professional authenticity (being true to the industry) and personal authenticity (being personally motivating). The fact that so few students enroll in CS, even when it’s available in their school, is evidence that our current approaches aren’t attractive. They are not personally authentic. We need to make progress on both fronts, but considering how over-full undergraduate CS classes are today, figuring out the recruitment problem is the greater challenge to giving everyone equitable access to CS education.

I just learned about a new paper in Constructionism 2020 from David Weintrop, Nathan Holbert, and Mike Tissenbaum (see link here) that makes this point well, better than I can here. “Considering Alternative Endpoints: An Exploration in the Space of Computing Educations” suggests that we need to think about multiple goals for computing education, and we too often focus just on the software development role:

While many national efforts tend to deploy rhetoric elevating economic concerns alongside statements about creativity and human flourishing, the programs, software, curricula, and infrastructure being designed and implemented focus heavily on providing learners with the skills, practices, and mindset of the professional software developer. We contend that computing for all efforts must take the “for all” seriously and recognize that preparing every learner for a career as a software developer is neither realistic nor desirable. Instead, those working towards the goal of universal computing education should begin to consider alternative endpoints for learners after completing computing curricula that better reflect the plurality of ways the computing is impacting their current lives and their futures.

June 1, 2020 at 7:00 am 18 comments

Measuring progress on CS learning trajectories at the earliest stages

I’ve written in this blog (and talked about many times) how I admire and build upon the work of Katie Rich, Diana Franklin, and colleagues in the Learning Trajectories for Everyday Computing project at the University of Chicago (see blog posts here and here). They define the sequence of concepts and goals that K-8 students need to be able to write programs consisting of sequential statements, to write programs that contain iteration, and to debug programs. While they ground their work in K-8 literature and empirical work, I believe that their trajectories apply to all students learning to program.

Here are some of the skills that appear in the early stages of their trajectories:

  • Precision and completeness are important when writing instructions in advance. 
  • Different sets of instructions can produce the same outcome. 
  • Programs are made by assembling instructions from a limited set. 
  • Some tasks involve repeating actions. 
  • Programs use conditions to end loops.  
  • Outcomes can be used to decide whether or not there are errors.
  • Reproducing a bug can help find and fix it.
  • Step-by-step execution of instructions can help find and fix errors.

These feel fundamental and necessary — that you have to learn all of these to progress in programming. But it’s pretty clear that that’s not true. As I describe in my SIGCSE keynote talk (the relevant 4 minute segment is here), there is lots of valuable programming that doesn’t require all of these. For example, most students programming in Scratch don’t use conditions to end loops — still, millions of students find expressive power in Scratch. The Bootstrap: Algebra curriculum doesn’t have students write their own iteration at all — but they learn algebra, which means that there is learning power in even a subset of this list.

What I find most fascinating about this list is the evidence that CS students older than K-8 do not have all these concepts. One of my favorite papers at Koli Calling last year was  It’s like computers speak a different language: Beginning Students’ Conceptions of Computer Science (see ACM DL link here — free downloads through June 30). They interviewed 14 University students about what they thought Computer Science was about. One of the explanations they labeled the “Interpreter.” Here’s an example quote exemplifying this perspective:

It’s like computers speak a different language. That’s how I always imagined it. Because I never understood exactly what was happening. I only saw what was happening. It’s like, for example, two people talking and suddenly one of them makes a somersault and the other doesn’t know why. And then I just learn the language to understand why he did the somersault. And so it was with the computers. 

This student finds the behavior of computers difficult to understand. They just do somersaults, and computer science is about coming to understand why they do somersaults? This doesn’t convey to me the belief that outcomes are completely and deterministically specified by the program.

I’ll write in June about Katie Cunningham’s paper to appear next month at the International Conference of the Learning Sciences. The short form is that she asked Data Science students at University to trace through a program. Two students refused, saying that they never traced code. They did not believe that “Step-by-step execution of instructions can help find and fix errors.” And yet, they were successful data science students.

You may not agree that these two examples (the Koli paper and Katie’s work) demonstrate that some University students do not have all the early concepts listed above, but that possibility brings us to the question that I’m really interested in: How would we know?

How can we assess whether students have these early concepts in the trajectories for learning programming? Just writing programs isn’t enough. 

  • How often do we ask students to write the same thing two ways? Do students realize that this is possible?
  • Students may realize that programming languages are “finicky” but may not realize that programming is about “precision and completeness.” 
  • Students re-run programs all the time (most often with no changes to the code in between!), but that’s not the same as seeing a value in reproducing a bug to help find and fix it. I have heard many students exclaim, “Okay, that bug went away — let’s turn it in.” (Or maybe that’s just a memory from when I said it as a student…)

These concepts really get at fundamental issues of transfer and plugged vs unplugged computing education. I bet that if students learn these concepts, they would transfer. They address what Roy Pea called “language-independent bugs” in programming. If a student understands these ideas about the nature of programs and programming, they will likely recognize that those are true in any programming language. That’s a testable hypothesis. Is it even possible to learn these concepts in unplugged forms? Will students believe you about the nature of programs and programming if they never program?

I find questions like these much more interesting than trying to assess computational thinking. We can’t agree on what computational thinking is. We can’t agree on the value of computational thinking. Programming is an important skill, and these are the concepts that lead to success in programming. Let’s figure out how to assess these.

May 25, 2020 at 7:00 am 14 comments

Thought Experiments on Why Face-to-Face Teaching Beats On-Line Teaching: We are Humans, not Econs

With everything moving on-line, I’m seeing more discussion about whether this on-line life might just be better. Amy Ko recently blogged (see post here) about how virtual conferences are cheaper, more accessible, and lower carbon footprint than face-to-face conferences, ending with the conclusion for her “it is hard to make the case to continue meeting in person.” My colleague, Sarita Yardi, has been tweeting about her exploration of “medium-independent classes” where she considers (see tweet here), “Trying to use the block of class time just because that’s how we’ve always taught seems like something to revisit. Less synchronous time; support short, frequent individual/small group interaction, less class time.”

It’s hard to do on-line education well. I used to study this kind of learning a lot (see post on “What I have learned about on-line collaborative learning”). I recently wrote about how we’re mostly doing emergency remote teaching, not effective on-line learning (see post here). I am concerned that moving our classes on-line will hurt the most the students who most need our help (see post here).

It should come as no surprise then that I don’t think that we know how to do on-line teaching or on-line conferences in a way that is anywhere close to the effectiveness of face-to-face learning. I agree with both Amy and Sarita’s points. I’m only focusing on learning outcomes.

Let me offer a thought experiment on why face-to-face matters. How often do you…

  • Look at the movie trailer and not watch the movie.
  • Watch the first few minutes of a show on Netflix but never finish it.
  • Start a book and give up on it.
  • Start watching a YouTube video and immediately close it or click away.

Now contrast that with: How often do you…

  • Get up from a one-on-one meeting and walk out mid-discussion.
  • Get up in the middle of a small group discussion and leave.
  • Walk out of a class during a lecture.
  • Walk out of a conference session while the speaker is still presenting (not between talks or during Q&A).

For some people, the answers to the first set are like the answers for the second set. I tried this thought experiment on my family, and my wife pointed out that she finishes every book she starts. But for most people, the first set is much more likely to happen than the second set. This is particularly hard for professors and teachers to recognize. We are good at self-regulated learning. We liked school. We don’t understand as well the people who aren’t like us.

There are a lot of people who don’t really like school. There are good reasons for members of minority groups to distrust or dislike school. Most people engage in higher-education for the economic benefit. That means that they have a huge value for the reward at the end, but they don’t particularly want to go through the process. We have to incentivize them to be part of the process.

Yes, of course, many students skip classes. Some students skip many classes. But the odds are still in favor of the face-to-face classes. If you are signed up for a face-to-face class, you are much more likely to show up for that class compared to any totally free and absolutely relevant to your interests lecture, on-campus or on-line. Enrolling in a course is a nudge.

For most people, you are much more willing to walk away from an asynchronous, impersonal event than a face-to-face, personal event. The odds of you learning from face-to-face learning are much higher simply because you are more likely to show up and less likely to walk out. It’s a great design challenge to make on-line learning opportunities just as compelling and “sticky” as face-to-face learning. We’re not there yet.

I would be all in favor of efforts to teach people to be more self-regulated. It would be great if we all were better at learning from books, lectures, and on-line resources. But we’re not. The learners with the best preparation are likely the most privileged students. They were the ones who were taught how to learn well, how to learn from school, and how to enjoy school.

Here’s a second thought experiment, for people who work at Universities. At any University, there are many interesting talks happening every week. For me, at least a couple of those talks each week are faculty candidates, which I am highly encouraged to attend. Now, they’re all on-line. How many of those did you attend when they were face-to-face, and how many do you attend on-line? My guess is that both are small numbers, but I’ll bet that the face-to-face number is at least double the on-line number. Other people see that you’re there face-to-face. There are snacks and people to visit with face-to-face. The incentives are far fewer on-line.

On-line learning is unlikely to ever be as effective as face-to-face learning. Yes, we can design great on-line learning, but we do that fighting against how most humans learn most things. Studies that show on-line learning to be as effective (or even more effective) than face-to-face classes are holding all other variables equal. But holding all other variables equal takes real effort! To get people to show up just as much, to give people as much (or more) feedback, and to make sure that the demographics of the class stay the same on-line or face-to-face — that takes significant effort which is invisible in the studies that are trying to just ask face-to-face vs on-line. The reality is that education is an economic endeavor. Yes, you can get similar learning outcomes, at a pretty high cost. At exactly the same cost, you’re unlikely to get the same learning outcomes.

We are wired to show-up and learn from face-to-face events. I would love for all of us to be better self-regulated learners, to be better at learning from books and from lecture. But we’re not Econs, we’re Humans (to use the Richard Thaler distinction). We need incentives. We need prompts to reflect, like peer instruction. We need to see and be seen, and not just through a small box on a 2-D screen.

May 11, 2020 at 7:00 am 19 comments

SIGCSE 2020: Papers freely available, AP CSA over AP CSP for diversifying computing, and a tour of computing ed research in one hour

My Blog@CACM post for this month was about my first stop on my tour of SIGCSE 2020 papers (see link here). While the SIGCSE 2020 conference was cancelled, the papers are freely available now through the end of June — see all the proceedings here. I’ve started going through the proceedings myself. The obvious place to start such a tour is with the award-winning papers. My Blog@CACM post is on the paper from An Nguyen and Colleen M. Lewis of Harvey Mudd College on the negative impact of competitive enrollment policies (having students enroll to get into CS, or requiring a higher-than-just-passing GPA to get into the computing major) on students’ sense of belonging, self-efficacy, and perceptions of the department.

I said that this was the first stop on my tour, but that’s not really true. I’d already looked up the paper Does AP CS Principles Broaden Participation in Computing?: An Analysis of APCSA and APCSP Participants (see link here), because I’d heard about it from co-author Joanna Goode. I was eager to see the result. They show that AP CS Principles is effectively recruiting much more diverse students than the AP CS A course (which is mostly focused on Java programming). But, AP CS A students end up with more confidence in computing and much more interest in computing majors and tech careers. Maybe CSA students had more interest to begin with — there is likely some selection bias here. This result reminds me of the Weston et al result (see blog post here) showing that the female high school students they studied continued on to tech and computing majors and careers if they had programming classes.

I’ve been reading The Model of Domain Learning: Understanding the Development of Expertise (see Amazon link) which offers one explanation of what’s going on here. Pat Alexander’s Model of Domain Learning points out that domain knowledge is necessary to have sustained interest in a domain. You can draw students in with situational interest (having activities that are exciting and engage novices), but you only get sustained interest if they also learn enough about the domain. Maybe AP CSP has more situational interest, but doesn’t provide enough of the domain knowledge (like programming) that leads to continued success in computing.

In my SIGCSE 2020 Preview blog post (posted just two days before the conference was posted), I mentioned the cool session that Colleen Lewis was organizing where she was going to get 25 authors to present the entire 700+ page Cambridge Handbook of Computing Education Research in 75 minutes. Unfortunately, that display of organizational magic didn’t occur. However, in a demonstration of approximately the same level of organizational magic, Colleen got the authors to submit videos, and she compiled a 55 minute version (which is still shorter than reading the entire tome) — see it on YouTube here.

There are lots of other great papers in the proceedings that I’m eager to get into. A couple that are high on my list:

  • Dual-Modality Instruction and Learning: A Case Study in CS1 from Jeremiah Blanchard, Christina Gardner-McCune, and Lisa Anthony from University of Florida, which provides evidence that a blocks-based version of Java leads to more and deeper version on the same assessments as students learning with textual Java (see link here).
  • Design Principles behind Beauty and Joy of Computing by Paul Goldenberg and others. I love design principles papers, because they explain why the authors and developers were doing what they were doing. I have been reading Paul since back in the Logo days. I’m eager to read his treatment of how BJC works (see link here).

Please share in the comments your favorite papers with links to them.

May 4, 2020 at 7:00 am 4 comments

Data science as a path to integrate computing into K-12 schools and achieve CS for All

My colleague Betsy DiSalvo is part of the team that just released Beats Empire, an educational game for assessing what students understand about middle school computer science and data science The game was designed by researchers from Teachers College, Columbia University; Georgia Tech; University of Wisconsin, Madison; SRI International; Digital Promise; and Filament Games in concert with the NYC Dept. of Education. Beats Empire is totally free; it has already won game design awards, and it is currently in use by thousands of students. Jeremy Roschelle was a consultant on the game and he just wrote a CACM Blog post about the reasoning behind the game (see link here).

Beats Empire is an example of an important development in the effort to help more students get the opportunity to participate in computing education. Few students are taking CS classes, even when they’re offered — less than 5% in every state for whom I’ve seen data (see blog post here). If we want students to see and use computing, we’ll need to put them in other classes. Data science fits in well with other classes, especially social studies classes. Bootstrap: Data Science (see link here) is another example of a computing-rich data science curriculum that could fit into a social studies class.

Social studies is where we can reach the more diverse student populations who are not in our CS classes. I’ve written here about my work developing data visualization tools for history classes. For a recent NSF proposal, I looked up the exam participation in the two Advanced Placement exams in computer science (CS Principles and CS A) vs the two AP exams in history (US history and World history). AP CS Principles was 32% female, and AP CS A was 24% female in 2019. In contrast, AP US History was 55% female and AP World History was 56% female. Five times as many Black students took the AP US History exam as took the AP CS Principles exam. Fourteen times as many Hispanic students took the AP US History exam as took the AP CS Principles exam.

Data science may be key to providing CS for All in schools.

April 27, 2020 at 7:00 am Leave a comment

Active learning has differential benefits for underserved students

We have had evidence that active learning teaching methods have more benefit for underserved populations than for majority groups (for example, I discussed the differential impact of active learning here). Just published in March in the Proceedings of the National Academy of Science is a meta-analysis of over 40 studies giving us the strongest argument yet: “Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math” at I’ll remind everyone that a terrific resource for peer instruction in computer science is here:

Achievement gaps increase income inequality and decrease workplace diversity by contributing to the attrition of underrepresented students from science, technology, engineering, and mathematics (STEM) majors. We collected data on exam scores and failure rates in a wide array of STEM courses that had been taught by the same instructor via both traditional lecturing and active learning, and analyzed how the change in teaching approach impacted underrepresented minority and low-income students. On average, active learning reduced achievement gaps in exam scores and passing rates. Active learning benefits all students but offers disproportionate benefits for individuals from underrepresented groups. Widespread implementation of high-quality active learning can help reduce or eliminate achievement gaps in STEM courses and promote equity in higher education.

April 20, 2020 at 7:00 am 5 comments

Checking our hubris with checklists: Learning a lesson from the XO Laptop

My Blog@CACM blog post for February was on Morgan Ames’ book The Charisma Machine (see post here). The book is well-written, and I do recommend it. In the post, I say that the OLPC opposition to HCI design practices is one of the themes in her book that I found most interesting:

It takes humility to design software that humans will use successfully. The human-computer interaction (HCI) community has developed a rich set of methods for figuring out what users need and might use, and for evaluating the potential of a new interface. To use these methods requires us to recognize our limitations — that we are unlikely to get the design right the first time and that our users know things that we don’t.

How do we get developers to have that humility? There are a lot of rewards for hubris. Making big promises that you probably can’t keep is one way to get grant and VC funding.

I just finished Atul Gawande’s The Checklist Manifesto (which I already blogged about here, before I even read it). It’s a short book which I highly recommend. I hadn’t realized before how much Gawande’s story overlaps with the OLPC story — or rather, how much it doesn’t but should have. Gawande is a surgeon. His entry into the idea of checklists is because of the success of checklists in reducing costs and improving patient success rates in medicine. There, too, they had to deal with physician hubris. They saw the checklists as busywork. As one physician said in opposition to checklists, “Forget the paperwork. Take care of the patient.”

The OLPC project couldn’t be bothered with user studies or pilot studies. They wanted to airdrop tablets into Ethiopia. They were so confident that they were going to (in Negroponte’s words) “eliminate poverty, create peace, and work on the environment.” They couldn’t be bothered with the details. They were taking care of the patient!

Gawande points out that checklists aren’t needed because physicians are dumb, but because they know SO much. We’re humans and not Econs. Our attention gets drawn this way or that. We forget about or skip a detail. Our knowledge and systems are so complex. Checklists help us to manage all the details.

We need checklists to check our hubris. We have confidence that we can build technology that changes users lives. The reality is that the odds are slim that we can have impact without going through an HCI design process, e.g., know the user, test often, and iterate. The OLPC Project could have used an HCI checklist.

The second to last chapter in Gawande’s Checklist Manifesto captures the idea well that we need checklists:

We are all plagued by failures—by missed subtleties, overlooked knowledge, and outright errors. For the most part, we have imagined that little can be done beyond working harder and harder to catch the problems and clean up after them. We are not in the habit of thinking the way the army pilots did as they looked upon their shiny new Model 299 bomber—a machine so complex no one was sure human beings could fly it. They too could have decided just to “try harder” or to dismiss a crash as the failings of a “weak” pilot. Instead they chose to accept their fallibilities. They recognized the simplicity and power of using a checklist. And so can we. Indeed, against the complexity of the world, we must. There is no other choice. When we look closely, we recognize the same balls being dropped over and over, even by those of great ability and determination. We know the patterns. We see the costs. It’s time to try something else. Try a checklist.

April 13, 2020 at 7:00 am 14 comments

How I’m lecturing during emergency remote teaching

Alfred Thompson (whom most of my readers already know) has a recent blog post requesting: Please blog about your emergency remote teaching (see post here). Alfred is right. We ought to be talking about what we’re doing and sharing our practices, so we get better at it. Reflecting and sharing our teaching practices is a terrific way to improve CS teaching, which Josh Tenenberg and Sally Fincher told us about in their Disciplinary Commons

My CACM Blog Post this month is on our contingency plan that we created to give students an “out” in case they become ill or just can’t continue with the class — see post here. I encourage all my readers who are CS teachers to create such a contingency plan and make it explicit to your students.

I am writing to tell you what I’m doing in my lectures with my co-instructor Sai R. Gouravajhala. I can’t argue that this is a “best” practice. This stuff is hard. Eugene Wallingford has been blogging about his emergency remote teaching practice (see post here). The Chronicle of Higher Education recently ran an article about how difficult it is to teach via online video like Zoom or BlueJeans (see article here). We’re all being forced into this situation with little preparation. We just deal with it based on our goals for our teaching practice.

For me, keeping peer instruction was my top priority. I use the recommended peer instruction (PI) protocol from Eric Mazur’s group at Harvard, as was taught to me by Beth Simon, Leo Porter, and Cynthia Lee (see I pose a question for everybody, then I encourage class discussion, then I pose the question again and ask for consensus answers. I use participation in that second question (typically gathered via app or clicker device) towards a participation grade in the class — not correct/incorrect, just participating. 

My plan was to do all of this in a synchronous lecture with Google Forms, based on a great recommendation from Chinmay Kulkarni. I would have a Google Form that everyone answered, then I’d encourage discussion. Students are working on team projects, and we have a campus license for Microsoft Teams, so I encouraged students to set that up before lecture and discuss with their teams. On a second Google Form with the same question, I also collect their email addresses. I wrote a script to give them participation credit if I get their email address at least once during the class PI questions.

Then the day before my first lecture, I was convinced on Twitter by David Feldon and Justin Reich that I should provide an asynchronous option (see thread here). I know that I have students who are back home overseas and are not in my timezone. They need to be able to watch the video at another time. I now know that I have students with little Internet access. So, I do all the same things, but I record the lecture and I leave the Google Forms open for 24 hours after the last class. The links to the Google Forms are in the posted slides and in the recorded lectures. To fill out the PI questions for participation, they would have to at least look at that the lecture.

I’m so glad that I did. As I tweeted, I had 188 responses to the PI questions after the lectures ended. 24 hours later, I had 233 responses. About 20% of my students didn’t get the synchronous lecture, but still got some opportunity to learn through the asynchronous component. The numbers have been similar for every lecture since that first.

I lecture, but typically only for 10-15 minutes between questions. I have 4-5 questions in an 85 minute lecture. The questions take longer now. I can’t just move the lecture along when most of the students answer, as I could with clickers. I typically give the 130+ students 90 seconds to get the link entered and answer the question. 

I have wondered if I should just go to a fully asynchronous lecture, so I asked my students via a PI question. 85% say that they want to see the lecturer in the video. They like that I can respond to chat and to answers in Google Forms. (I appreciate how Google Forms lets me see a summary of answers in real-time, so that I can respond to answers.) I’d love to have a real, synchronous give-and-take discussion, but my class is just too big. I typically get 130+ students synchronously participating in a lecture. It’s hard to have that many students participate in the chat, let alone see video streams for all of them.

We’re down to the last week of lecture, then we’ll have presentations of their final projects. They will prepare videos of their presentations, and receive peer comments. Each student has been assigned four teams to provide peer feedback on. Each team has a Google Doc to collect feedback on their project.

So, that’s my practice. In the comments, I’d welcome advice on improving the practice (though I do hope not to have to do this again anytime soon!), and your description of your practice. Let’s share.

April 6, 2020 at 7:00 am 5 comments

So much to learn about emergency remote teaching, but so little to claim about online learning

The Chronicle of Higher Education published an article by Jonathan Zimmerman on March 10 arguing that we should use the dramatic shift to online classes due to Covid-19 pandemic as an opportunity to research online learning (see article here).

For the first time, entire student bodies have been compelled to take all of their classes online. So we can examine how they perform in these courses compared to the face-to-face kind, without worrying about the bias of self-selection.

It might be hard to get good data if the online instruction only lasts a few weeks. But at institutions that have moved to online-only for the rest of the semester, we should be able to measure how much students learn in that medium compared to the face-to-face instruction they received earlier.

To be sure, the abrupt and rushed shift to a new format might not make these courses representative of online instruction as a whole. And we also have to remember that many faculty members will be teaching online for the first time, so they’ll probably be less skilled than professors who have more experience with the medium. But these are the kinds of problems that a good social scientist can solve.

I strongly disagree with Zimmerman’s argument. There is a lot to study here. There is little to claim about online learning.

What we are doing right now is not even close to best practice for online learning. I recommend John Daniels’ book Mega-Universities (Amazon link). One of his analyses is a contrast with online learning structured as “correspondence school” (e.g., send out high-quality materials, require student work, provide structured feedback) or as a “remote classroom” (e.g., video record lectures, replicate in-classroom structures). Remote classrooms tend to have lower-retention and increase costs as the number of students scale. Correspondence school models are expensive (in money and time) to produce, but scales well and has low cost for large numbers. What we’re doing is much closer to remote classrooms than correspondence school. Experience with MOOCs supports this analysis. Doing it well takes time and is expensive, and is carefully-structured. It’s not thrown together with less than a week’s notice.

My first thought when I read Zimmerman’s essay was for the ethics of any experiment comparing to the enforced move to online classes versus face-to-face classe. Students and faculty did not choose to be part of this study. They are being forced into online classes. How can we possibly compare face-to-face classes that have been carefully designed, with hastily-assembled online versions that nobody wants at a time when the world is suffering a crisis. This isn’t a fair nor ethical comparison.

Ian Milligan recommends that we change our language to avoid these kinds of comparisons, and I agree. He writes (see link here) that we should stop calling this “online learning” and instead call it “emergency remote teaching.” Nobody would compare “business as usual” to an “emergency response” in terms of learning outcomes, efficiency, student satisfaction, and development of confidence and self-efficacy.

On the other hand, I do hope that education researchers, e.g., ethnographers, are tracking what happens. This is first-ever event, to move classes online with little notice. We should watch what happens. We should track, reflect, and learn about the experience.

But we shouldn’t make claims about online learning. There is no experiment here. There is a crisis, and we are all trying to do our best under the circumstances.

March 30, 2020 at 10:20 am 7 comments

Older Posts Newer Posts

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 8,422 other followers


Recent Posts

Blog Stats

  • 1,829,329 hits
January 2021

CS Teaching Tips