Purpose-first programming: A programming learning approach for learners who care most about what code achieves: Katie Cunningham’s Defense

On Wednesday, Katie Cunningham is defending her dissertation, “Purpose-First Programming: A Programming Learning Approach for Learners Who Care Most About What Code Achieves.” I’m proud of the work Katie has done with Barb and me over the years. Let me relate the story here, with links to the blog posts.

I first met Katie through through an on-line essay she wrote explaining the issues of gender and CS to her faculty (see my blog post referencing at link here). After she graduated, she worked on the CSin3 project at California State University at Monterey Bay which helped Latino and Latina students get undergraduate degrees in CS in three years. The paper she and the CSin3 team wrote won a Best Paper award at SIGCSE 2018 (see paper here).

Katie started her PhD research studying how students traced code when trying to understand and predict program behavior. She published her findings at ICER 2017 (see blog post). As you’d expect, students who traced programs line-by-line were more likely to get prediction problems (What is the output? What is this variable’s value?) correct. But not always. Most intriguing: Students who stopped mid-way through a trace were more likely to get the problems wrong than those who never traced at all.

In her next study, she replicated the original experiment and then brought into the lab those students who had stopped mid-way in order to ask them “why?” A common answer was that the students were trying to see the “pattern” of the program, and once they saw the pattern, they were able to predict the answer. The problem is that the students were novices. They didn’t know many patterns. They often guessed wrong. Katie presented this paper at ITiCSE 2019 (see blog post).

Katie did a think-aloud study where she could watch students tracing, and something unexpected and interesting happened — two participants refused to trace. These were data science students who did program successfully, but they were unwilling to trace code at the line-by-line level. She wrote an ICLS 2020 paper about their reasons (see blog post). She decided to study that population.

A 2018 CHI paper by another U-M student, April Wang, had talked about how computing education fails conversational programmers (see paper here). Katie decided to build a new kind of curriculum that addressed her data science students and April’s conversational programmers. How do you teach programming to students who (1) don’t want to become professional programmers and (2) are dissuaded from high cognitive load activities like tracing code? This is a very different problem than most of CS education at the undergraduate level where we have eager CS majors who want to get software development jobs. Katie was dealing with issues both of motivation and of cognitive load.

Katie invented purpose-first programming. I don’t want to say too much about it here — her dissertation and her future papers will go more into it. I’ll give you a sense for her process. She used Github repositories and expert interviews to identify a few programming plans (just like Elliot Soloway and Jim Spohrer studied years ago) that were in common use in a domain that her participants cared about. She then taught those plans. Students modified and combined the plans to create programs that the students found useful. Rather than start with syntax or semantics, she started with the program’s purpose. The results were very positive in terms of learning, performance, and affect. Rather than be turned away, they wanted more. One student asked if she could create a whole set of curricula like this, each for a different purpose. That’s the idea exactly. Katie may be on her way to inventing the Duolingo of programming.

Katie already has a post-doc lined up. She’ll be a CI Fellow with Nell O’Rourke at Northwestern. The defense will be on Zoom — feel free to come and cheer her on!

The School of Information is pleased to announce the oral defense of Kathryn Cunningham:

Title: Purpose-First Programming: A Programming Learning Approach for Learners Who Care Most About What Code Achieves

Date: Wednesday, December 2nd

Time: 10 am – 12 pm EST

Place: This defense will be held virtually for the public to attend. Please use this link.

Barbara Ericson and Mark Guzdial, serving as committee chairs, will preside over the oral defense.

All are welcome to (virtually) attend!

How do teachers teach recursion with embodiment, and why won’t students trace their programs: ICLS 2020 Preview

This coming week was supposed to be the International Conference of the Learning Sciences (ICLS) 2020 in Nashville (see conference website here). But like most conferences during the pandemic, the face-to-face meeting was cancelled (see announcement here). The on-line sessions are being announced on the ICLS2020 Twitter feed here.

I’m excited that two of my students had papers accepted at ICLS 2020. I haven’t published at ICLS since 2010. It’s nice to get back involved in the learning sciences community. Here’s a preview of their papers.

How do teachers teach recursion with embodiment

I’ve written here about Amber Solomon’s work on studying the role of space and embodiment in CS learning. This is an interesting question. We live in a physical world and think in terms of physical things, and we have to use that to understand the virtual, mostly invisible, mostly non-embodied world of computing. At ICER 2018, she used a taxonomy of gestures used in science learning to analyze the gestures she saw in a high school computer science classroom (see link here). Last summer at ITiCSE, she published a paper on how making CS visible in the classroom (through gesture and augmented reality) may reduce defensive climate (see link here). In her dissertation, she’s studying how teachers teach recursion and how learners learn recursion, with a focus on spatial symbol systems.

Her paper at ICLS 2020 is the first of these studies: Embodied Representations in Computing Education: How Gesture,Embodied Language, and Tool Use Support Teaching Recursion. She watched hours of video of teachers teaching recursion, and did a deep dive on two of them.

I’m fascinated by Amber’s findings. Looking at what teachers say and gesture about recursion from the perspective of physical embodiment, I’m amazed that students ever learn computer science. There are so many metaphors and assumptions that we make. One of the teachers says, when explaining a recursive function:

“Then it says “… “now I have to call.”

Let’s think about this from the perspective of the physical world (which is where we all start when trying to understand computing):

• What does it mean for a function to “say” something?
• The function “says” things, but I “call”? Who is the agent in this explanation, the function or me? It’s really the computer with the agency, but that doesn’t get referenced at all.
• Recursion is typically explained as a function calling itself. We typically “call” something that is physically distant from us. If a function is re-invoking itself, why does it have to “call” as if at a distance?

For most computer scientists, this may seem like explaining that the sky is blue or that gravel exists. It’s obvious what all of this means, isn’t it? It is to us, but we had to learn it. Maybe not everyone does. Remember how very few students take or succeed at computer science (for example, see this blog post), and what enormously high failure and drop-out rates we have in CS. Maybe only the students who pick up on these metaphors are the ones succeeding?

Why won’t students trace their programs?

Katie Cunningham’s first publication as a PhD student was her replication and extension of the Leeds Working group study, showing that students who trace program code successfully line-by-line are able to answer more accurately questions about the code (see blog post here). But one of her surprising results was that students who start tracing and give up do worse on prediction questions than those students who never traced at all. In her ITICSE 2019 paper (see post here), she got the chance to ask those students who stopped tracing why they did. She was extending that with a think-aloud protocol, when something unusual happened. Two data science students, who were successful at programming, frankly refused to trace code.

Her paper “I’m not a computer”: How identity informs value and expectancy during a programming activity is an exploration of why students would flat out refuse to trace code — and yet successfully program. She uses Eccle’s Expectancy Value Theory (which comes up pretty often in our thinking, see this blog post) to describe why the cost of tracing outweighs the utility for these students, which is defined in terms of their sense of identity — what they see themselves doing in the future. Sure, there will be some programs that they won’t be able to debug or understand because they won’t trace line-by-line. But maybe they’ll never actually have to deal with code that complex. Is this so bad?

Katie’s live session is 2:00-2:40pm Eastern time on June 23. The video link will be available on the conference website to registered attendees. A pre-print version of her paper is available here.

Both of these papers give us new insight into the unexpected consequences of how we teach computing. We currently expect students to figure out how their teachers are relating physical space and computation, through metaphors that we don’t typically explain. We currently teach computing expecting students to be able to trace code line-by-line, though some students will not do it (and maybe don’t really need to). If we want to grow who can succeed at computing education, we need to think through who might be struggling with how we’re teaching now, and how we might do better.

How to reduce the defensive climate, and what students really need to understand code: ITICSE 2019 Preview

This year, we’re presenting two papers at the 2019 ACM SIGCSE Innovation and Technology in CS Education (ITICSE) conference in Aberdeen, Scotland. I’ve been to ITICSE several times and enjoy it every time, but I can’t always justify the trip. This year, Barbara Ericson, Katie Cunningham, and I are all in Germany for a Dagstuhl Seminar on Notional Machines the week before, so we took the opportunity to submit and are fortunate to be invited to present.

Making CS Learning Visible: Case Studies on How Visibility of Student Work Supports a Community of Learners in CS Classrooms by Amber Solomon, Vanessa Oguamanam, Mark Guzdial, and Betsy DiSalvo

When I taught CS Ed Research this last semester (see the blog post about open questions from the class), the students so resonated with Lecia Barker’s papers about defensive climate (the classic paper is here). The story about how CS classes are “characterized by competitiveness rather cooperation, judgments about others, superiority, and neutrality rather than empathy” still rings true, 17 years after first written. Several of my students proposed research to follow-up on the original study.

Amber and Vanessa are also motivated by the concerns about defensive climate in CS classes, but they don’t want to measure it. They are suggesting an intervention.

They suggest that a community of learners approach would reduce defensive climate. Key to creating a community of CS learners, they propose, is making student work and process visible. Vanessa works in marker-oriented curricula, where student work is physical and the work process is visible. Amber did the evaluation of the AR design studio classroom that I’ve written about previously. In both of these case studies, they observed student communication patterns that were different from the defensive climate studies and more in keeping with a community of learners culture. Defensive climate is still a problem, and changing culture and community is the way to address it.

Novice Rationales for Sketching and Tracing, and How They Try to Avoid It by Katie Cunningham, Shannon Ke, Mark Guzdial, and Barbara Ericson

At ICER 2017, Katie presented a study of how students trace their programs on paper (see ICER 2017 paper here, and my blog post summary here). She had some fascinating and surprising results. For example, if students traced their programs only part way and then guessed at the final result, they were more likely to get the problem wrong than if they’d never traced before. But why? Why would students trace the program part way and then stop — only to get it wrong?

That’s what Katie explored in this follow-up paper. She had an innovative experimental design. She replicated her original tracing experiment, then pulled out about a dozen interesting participants, and invited them back for a retrospective interview. She could show them the original problem and what they traced — and then ask them why? Here’s one of the answers: They thought that they “got the hang of it.” They recognized a goal or pattern. They just recognized wrong.

One of my favorite parts of the paper is completely visual. Katie had a terrific idea — let’s ask the teacher of the class to trace the problems. Here’s one of the traces that the teacher did:

Here are some examples of what students did:

Notice a distinct lack of similarity? Why? Why don’t students trace like the instructor did?

This is a qualitative study, so it’s rich with interview data. I recommend reading the whole paper. There’s a neat part at the end where Katie points out, “Program visualizations do X. When students trace, they do Y. Why are these things so different?”

Teachers are not the same as students, and the role of tracing: ICER 2017 Preview

The International Computing Education Research conference starts today at the University of Washington in Tacoma. You can find the conference schedule here, and all the proceedings in the ACM Digital Library here. In past years, all the papers have been free for the first couple weeks after the conference, so grab them while they are outside the paywall.

Yesterday was the Doctoral Consortium, which had a significant Georgia Tech presence. My colleague Betsy DiSalvo was one of the discussants. Two of my PhD students were participants:

• Amber Solomon who is exploring the relationship between spatial reasoning and CS learning (see her position paper here). I blogged about Amber's work on our design studio using AR technology.
• Katie Cunningham who is exploring how instructors can use students' sketching and tracing to get greater insight into the students' mental models of the notional machine (see her position paper here). I blogged about Katie and her work when she won an NSF fellowship earlier this year.

We have two research papers being presented at ICER this year. Miranda Parker and Kantwon Rogers will be presenting Students and Teachers Use An Online AP CS Principles EBook Differently: Teacher Behavior Consistent with Expert Learners (see paper here) which is from Miranda C. Parker, Kantwon Rogers, Barbara J. Ericson, and me. Miranda and Kantwon studied the ebooks that we've been creating for AP CSP teachers and students (see links here). They're asking a big question: "Can we develop one set of material for both high school teachers and students, or do they need different kinds of materials?" First, they showed that there was statistically significantly different behaviors between teachers and students (e.g. different number of interactions with different types of activities). Then, they tried to explain why there were differences.

We develop a model of teachers as expert learners (e.g., they know more knowledge so they can create more linkages, they know how to learn, they know better how to monitor their learning) and high school students as more novice learners. They dig into the log file data to find evidence consistent with that explanation. For example, students repeatedly try to solve Parsons problems long after they are likely to get it right and learn from it, while teachers move along when they get stuck. Students are more likely to run code and then run it again (with no edits in between) than teachers. At the end of the paper, they offer design suggestions based on this model for how we might develop learning materials designed explicitly for teachers vs. students.

Katie Cunningham will be presenting Using Tracing and Sketching to Solve Programming Problems: Replicating and Extending an Analysis of What Students Draw (see paper here) which is from Kathryn Cunningham, Sarah Blanchard, Barbara Ericson, and me. The big question here is: "Of what use is paper-and-pen based sketching/tracing for CS students?" Several years ago, the Leeds' Working Group (at ITiCSE 2004) did a multi-national study of how students solved complicated problems with iteration, and they collected the students' scrap paper. (You can find a copy of the paper here.) They found (not surprisingly) that students who traced code were far more likely to get the problems right. Barb was doing an experiment for her study of Parsons Problems, and gave scrap paper to students, which Katie and Sarah analyzed.

First, they replicate the Leeds' Working Group study. Those who trace do better on problems where they have to predict the behavior of the code. Already, it's a good result. But then, Katie and Sarah go further. For example, they find it's not always true. If a problem is pretty easy, those who trace are actually more likely to get it wrong, so the correlation goes the other way. And those who start to trace but then give up are even more likely to get it wrong than those who never traced at all.

They also start to ask a tantalizing question: Where did these tracing methods come from? A method is only useful if it gets used — what leads to use? Katie interviewed the two teachers of the class (each taught about half of the 100+ students in the study). Both teachers did tracing in class. Teacher A's method gets used by some students. Teacher B's method gets used by no students! Instead, some students use the method taught by the head Teaching Assistant. Why do some students pick up a tracing method, and why do they adopt the one that they do? Because it's easier to remember? Because it's more likely to lead to a right answer? Because they trust the person who taught it? More to explore on that one.