The goal of the first CS course should be to promote confidence if we’re going to increase diversity in CS: Paying off on a bet

December 29, 2020 at 7:00 am 25 comments

This should be a thing: If you make a public bet on Twitter, and lose, you should have to write a blog post explaining how you got it wrong.

Let me set the stage for the bet. There are studies suggesting that the Advanced Placement (AP) Computer Science A exam has a significantly different impact on students’ majors than other AP exams. (For non-US readers: AP tests provide an opportunity for secondary school students to earn post-secondary school credit.) AP CS A exam-takers are more likely to go on to take more CS courses or become a CS major — more likely than, say, students taking AP Calculus or AP US History exams to become mathematics or history majors. But does that extend to the newer AP CS exam, AP CS Principles? AP CS Principles was designed to be less about the kinds of programming that CS majors do in their first year, and more about a broader understanding of computing and its effects (see College Board site here). There were several of us talking about this in the Spring. On April 1, 2019, I tweeted to Jeff Forbes (see link): “I bet that AP CS Principles has no impact on CS or STEM majors. It’s such a different course (eg doesn’t map to CS courses on most campuses).” He took that bet, and he was right. A study released by the College Board shows that there is a causal relationship between taking AP CS Principles and majoring in CS in undergraduate (see report link here). The impact is large. Overall, students who take AP CS Principles are three times more likely to major in computer science in college. AP CSP students who are female are twice as likely to major in CS.

I wasn’t crazy for expecting that AP CS Principles would not have such a big impact on recruitment and retention. At SIGCSE 2020, Joanna Goode and co-authors published a paper showing that (see blog post link here) AP CS Principles is effectively recruiting much more diverse students than the AP CS A course (which is mostly focused on Java programming). But, AP CS A students end up with more confidence in computing and much more interest in computing majors and tech careers. ACM TOCE in 2019 published a paper using NCWIT Aspirations award winners (see blog post link here) showing that taking the CS Advanced Placement A exam was one of the best predictors of persistence three years after the high school survey in both CS and other technology-related majors. The TOCE paper authors made a particular emphasis on the importance of programming: “It seems that involvement in general tech-related fields other than programming in high school does not transfer to entering and persisting in computer science in college for the girls in our sample.”

So I had good reason to believe that non-programming-intensive courses might not have a big impact on recruitment into the CS major and retention. But I accept the evidence that I was wrong. What else is going on?

Here’s another recent piece of evidence that supports Jeff’s belief that AP CS Principles (and classes like that) could be having a big impact. Philip Boda and Steve McGee have a paper coming out in SIGCSE 2021 showing that the Exploring CS course (see website here) is having a significant impact in driving up AP CS A participation and diversity (see paper here), which continues to have a large impact on majoring in CS. Exploring CS, like AP CS Principles, is de-emphasizes programming in favor of a broader understanding of computing and helping students to see themselves as successful at CS.

Neither of these papers offers an explanation for why AP CSP or ECS is having this positive impact. They’re both large scale quantitative stories. You’d think that I might have learned my lesson from this last failed bet. Nah. I’ve got guesses. My guesses might be wrong, as they were in this case. I’m a post-positivist. I don’t think we’ll ever get to the place where we know the complete truth, but we should keep trying, keeping making hypotheses, and we can keep getting closer.

Here’s my hypothesis for what’s going on, stated as a prediction:

A first course will be successful at promoting recruitment into CS as a major or career and at retaining students in CS if it increases students self-efficacy about programming tasks.

The critical part is for students to increase their confidence that they can be successful at programming tasks. AP CS A easily does this, which is why it has such great results in recruitment and retention. Not all classes or experiences do, as the NCWIT study suggests. AP CS Principles and ExploringCS are all about increasing student confidence, helping them to see themselves as successful at computing. I don’t know how little programming a student needs to do to increase their self-efficacy. Maybe it’s enough to see programs and what programming is about.

Recent research in computing education has been focusing on self-efficacy as one of the most important variables predicting student recruitment and retention in CS. Alex Lishinski and his co-authors showed that self-efficacy had different relationships for female and male CS students (see paper link here) and that programming projects influenced students’ sense of self-efficacy, which in turn influenced performance in the CS class (see paper link here). Jamie Gorson and Nell O’Rourke found (in an ICER 2020 paper that I blogged about here) that CS students had deflated self-efficacy, in part, because they had unreasonable expectations of what real programmers do. Dr. Katie Cunningham, soon to be a post-doc joining Nell’s lab, showed in her dissertation how students simply give up on programming tasks that they don’t think that they’ll be successful at (see blog post on Katie’s dissertation defense). Self-efficacy is likely an important variable in recruitment and retention, particularly of female students, and it’s one that we can manipulate with better designed education.

I’m not the first person to to suggest this relationship. In a study with over 5 million participants, Peter Kemp and colleagues suggest that female participation in secondary school computer science in England is being negatively impacted because of female students’ low self-efficacy in CS — and that this is because of the CS classes (see paper link here). In England, curriculum in Information and Communications Technology is being faded out in favor of a Computer Science focus. They write in their paper “Female Performance and Participation in Computer Science: A National Picture”:

The move to introduce CS into the English curriculum and the removal of the ICT qualifications look to be having a negative impact on female participation and attainment in computing. Using the theory of self-efficacy, we argue that the shift towards CS might decrease the number of girls choosing further computing qualifications or pursuing computing as a career. Computing curriculum designers and teachers need to carefully consider the inclusive nature of their computing courses.

I made my bet because I thought that the programming-light focus of AP CS Principles (or even ExploringCS) would have less of an impact on CS recruitment and retention than the programming-intensive focus of AP CS A. I now believe I was wrong. I would now bet that the amount of programming probably isn’t the critical variable at all. It’s whether students come out of these courses saying, “I can do this. I can program.” That’s the critical variable for recruitment and retention that I believe AP CS Principles and Exploring CS are influencing successfully.

Entry filed under: Uncategorized. Tags: , , .

Dijkstra’s Truths about Computing Education Aren’t: The many kinds of programming Promote diversity by teaching to many goals for computing

25 Comments Add your own

  • 1. chapter18  |  December 29, 2020 at 7:01 am

    Good analysis..

  • 2. MItch Resnick  |  December 29, 2020 at 7:55 am

    I agree that it is important for students to come away from their initial programming experiences with a feeling that “I can do this.” In addition, it is important for them to come away with a feeling of “I want to do this.” As we’ve developed Scratch (and associated resources and activities), our goal has been for young people to see programming not just as something that they are able to do, but something that is interesting, enjoyable, and personally meaningful.

    • 3. Mark Guzdial  |  December 29, 2020 at 9:15 am

      Hi Mitch,

      I agree. In Katie Cunningham’s dissertation work, she found that her data science students refused to trace program code because it was too hard, they expected to fail at it, and the code that they typically had to trace was toy-like, not doing anything interesting. By switching to tasks where they were doing real data science tasks, they engaged and were able to successfully debug their programs. Her dissertation drew on both cognitive theories (like cognitive load) and motivation theories (like expectancy-value theory).

  • 4. alfredtwo  |  December 29, 2020 at 9:26 am

    In the years before AP CS P I taught a one semester programming course that was all programming but which had a focus of helping students see that they could do programming. I was hoping it would feed more students into AP CS A and to some extent it did. On the other hand, I saw many promising female students ignore AP CS A in favor of yet another year of world language (I blame guidance). A number of these young women later became CS majors or minors. Because they gained self efficacy in my class? I’d like to think that was a factor.

    One thing that I think CS Principles does that helps move students to major in CS later is that it shows that CS is more than just programming. That means that a little confidence in programming goes a long way. It increases interest in the wider areas of CS with the knowledge that they can “read the language.”

  • 5. hodgeschuck  |  December 29, 2020 at 9:59 am

    This is music to this self-efficacy and ed tech researcher’s ears!

  • 6. orcmid  |  December 29, 2020 at 11:28 am

    This may correlate with ideas about progressive disclosure in programming systems used pedagogically and, my interest, in self-directed learning of software tools.

    I am reminded of the Heathkit slogan “We will not let you fail.” I relied on that to learn enough electronics, using Heathkit courses, to assemble my first personal computer, a Heathkit H8, in 1978. My first “bug” was a cold-solder joint in the fly-back of my H19 terminal. I also had a problem with stability of the Heathkit oscilloscope that I built and subsequently found not useful in maintaining the H8/H19, or H89 that I build later.

    My first mistake along these lines was purchasing something that was not owner-maintainable, a 70 lb unit containing a 5 MB (!) hard disk and an 8″ floppy drive. The bleeding edge is aptly named.

    Already being a programmer since 1958, my interests quickly moved from kit-bashing to application software and software tools. Assembly language was a lot easier in those days, Basic was handy, and then Turbo Pascal (and small C compilers) were also very useful and led many into personal software development and some small-business ventures too. Dial-up and text-based communication were also eye-openers. Pre-internet connectivity and protocols were also easier to grasp, in my experience.

  • 7. Mike Zamansky (@zamansky)  |  December 29, 2020 at 11:51 am

    The jury’s still out and will remain out (as you said, we may never know) – many schools now offer CSP where they offered nothing before and some (many?) require CSP where they required nothing before. It’s not really a surprise that this will result in more kids exploring CS in their futures.

    It’s a world away from history which is yet another history class or calc which is “the next math class.” Probably more like a school adding a music program where there was none before but CS until recently was even more fringe.

    I’d also guess that the self efficacy might have just a bit to do with the teacher and not just the class.

    Finally, I’d like to see retention. Are we encouraging all these students to go on to CS only to knock them out with gatekeeper courses at the next level?

    • 8. Mark Guzdial  |  December 29, 2020 at 5:06 pm

      The results from UCSD’s PI+PP+MediaComp course after 10 years are pretty convincing to me — see their ICER 2020 paper. They found that a class that encourages higher self-efficacy led to greater retention and no difference in grades through the upper-division CS classes.

  • 9. Grant Hutchison  |  December 29, 2020 at 5:34 pm

    I am curious how your statement might be translated from the view point of a high school CS teacher.

    Does …

    A first course will be successful at promoting recruitment into CS as a major or career and at retaining students in CS if it increases students self-efficacy about programming tasks.


    A first coding experience in an elementary school math course will be successful at promoting recruitment into CS electives in high school… if it increases students self-efficacy about programming tasks.

    I would guess this would also be true. In our province (Ontario) there are now specific coding expectations that were added to the math curriculum this past summer and high school CS teachers are trying to understand their impact over the next few years.

    Here are the Grade 3 expectations for example.

    C3.1 solve problems and create computational representations of mathematical situations by writing and executing code, including code that involves sequential, concurrent, and repeating events

    C3.2 read and alter existing code, including code that involves sequential, concurrent, and repeating events, and describe how changes to the code affect the outcomes

    • 10. Mark Guzdial  |  December 29, 2020 at 7:11 pm

      What a great question, Grant! I’ve been thinking about a much lower bar. I want any first coding experience in elementary or high school to lead to self-efficacy PERIOD. I’m not concerned about CS electives. I want all kids to come away with a sense that they can program, that programming isn’t scary. That some want to program and go on to CS electives is scary. Few students turn away from advanced History or Literature classes because they’re scary — not everyone wants more History or Literature. Students do turn away from Math and Science because it’s scary and they don’t think that they can be successful. I want students to not find CS scary.

      Those are really high expectations for 3rd graders. Do you know who wrote them? Do they know that these are achievable by most 3rd graders in Ontario?

      • 11. MItch Resnick  |  December 29, 2020 at 7:54 pm

        Mark: In your response to my comment above, you said you agreed that it is important for students to feel “I want to do this.” But in your response here, you again underplay the importance of engaging students in personally meaningful activities. I dislike the Ontario expectations (as presented above) not because they are “too high” but because they focus on the wrong things. Instead of asking students to “solve problems and create computational representations of mathematical situations,” how about asking them to create an interactive holiday card for a friend? That’s how to engage a broader diversity of students in coding and CS.

        • 12. Dinesh Ayyappan  |  December 30, 2020 at 6:32 am

          I don’t feel like these two ideas have to be at odds (“I want to do this” vs. “I can do this”), but in my classroom experience, I think I’ve found it easier to push students to push their technical limits after they’ve chosen a topic or context they care about.

          I agree that self-efficacy will be what students will have to rely on when they get to their road blocks in university (and this makes it necessary for increasing recruitment and retention in CS), but it’s a retrospective development. It’s good to have that conversation with learners after a project, quarter, or term, but it’s through building personally meaningful stuff, as Dr. Resnick said, that I’ve found it easier to nudge students to engage with, persist through, and succeed in the kinds of learning tasks that lead to self-efficacy.

        • 13. Mark Guzdial  |  December 30, 2020 at 9:45 am

          Guilty as charged! I agree that it’s important to engage students in activities that are personally meaningful, but that’s not the role of standards or expectations. (I like the Canadian’s optimistic tone with “expectations” over the American’s regulatory tone with “standards.”). Expectations are about what to teach, not how to teach it — Barb’s phrase, when I told her about this conversation. The latter is in the control of the teacher, who is much better prepared and situated to focus on personal meaning for individual students than are regulators.

          I like the first part of the first expectation “solve problems and create computational representations of mathematical situations by writing and executing code.” That creates value and encourages the use of computing in mathematics, which is so important. We live in Thorndike’s schools not Dewey’s. Teachers’ jobs are to teach to standards/objectives/expectations. Done well, standards can enable teachers, give them more freedom to explore and experiment. This first part of the standards gives the mathematics teacher the reason to integrate computing into mathematics. I fought that battle in Georgia, where mathematics regulators see no role for computing. I don’t understand why this phrase got added on, “including code that involves sequential, concurrent, and repeating events.” That requires 3rd grade math classes to use imperative programming, not Bootstrap’s functional programming nor Jens’ HyperBlocks, nor any of the task-specific programming languages that I’m working on. Because 3rd graders ought to learn industry-standard paradigms?

          Most of my research time is spent with social studies teachers these days. Social studies teachers do know how to teach for diversity — everyone has to take social studies. They are much more inventive than I am in figuring out how to find or create personally meaningful activities for students. My job is to give them the computational tools to do what they do best. I don’t want standards getting in their way. In Michigan, they have just created new social studies standards that say that students need to know how to construct, interpret, and use in arguments visualizations like timelines, map, and charts. The standards don’t say anything about how to do that, which is just right. I’m trying to give teachers the tools to meet those standards in the best way they know how.

          • 14. MItch Resnick  |  December 31, 2020 at 7:13 am

            A few final thoughts…

            I don’t accept that sharp division between “what” and “how” to teach. I think it’s incredibly important for elementary-school students to learn how to “design a project around an idea or interest” (much more important than learning how to multiply fractions, for instance). So it’s not just an approach to teaching: it is also what we should be teaching.
            It’s great that you are working with social studies teachers who support students working on personally meaningful activities. And I agree that many teachers know how to do that well. But many teachers don’t that in their classrooms. They should be explicitly encouraged to do so. That doesn’t mean we should tell them exactly how to implement activities in their classrooms. But we should encourage them to support students working on personally meaningful projects.
            I find it sad that you are so accepting of “Thorndike won, Dewey lost.” I understand that we sometimes need to work within the current constraints of schools. But we should never give up on trying to bring about change, to provide children with the type of education that they need and deserve.

            • 15. Mark Guzdial  |  December 31, 2020 at 11:25 am

              Hi Mitch,

              I strongly agree that we want to encourage teachers to find and create personally meaningful projects, and that we should be teaching teachers how to do that. Joanna Goode and I just wrote a chapter for a new National Academy report on authentic STEM learning where we made exactly that point. The what/how division is built in to the system. We’re trying to change the system.

              I haven’t given up, and I reject the suggestion that I have. I think we share a similar end goal. Your strategy has been to create a learning environment that both Dewey and Papert would recognize and value, and you’ve done a marvelous job of that. I’m pursuing a different strategy. I spent 12 years working around public policy (in Georgia Computes and then ECEP) with 16 states and Puerto Rico. I see terrific CS classes and learning environments in which single digit percentages of students participate. I see Seymour’s “schoolification” of CS across the US. That’s what Miranda Parker’s dissertation described.

              I want to see Seymour’s Mathland, and also Historyland and Languageland. But in Thorndike’s schools, those learning environments are going to first appear in Math, History, and ELA classes. They’re going to be domain-specific. That’s where I’m starting. I believe in incremental progress, not revolutionary. It’s not a popular strategy in educational reform — Alan Kay has told me that I’m wrong in this blog many times. But I believe that teachers are trying to do the best that they can, and they’re looking for good materials that make sense to them and their students. I’m trying to figure out what kind of computing will make sense to other-than-CS teachers and also to students who reject CS. That’s what Katie Cunningham was creating in her dissertation. I love the new paper by Breanne Litts, Kristin Searle, Bryan Brayboy, and Yasmin Kafai which tells us what kinds of story structures their Native American youth want to build, and I’m excited to build tools that support those structures.

              I see you creating great models of Dewey’s education, and I’m helping to change schools to value and embrace those models.

              Even if we disagree, I’m grateful that you visited, engaged, and commented, Mitch. I wish you a very happy new year.

              • 16. MItch Resnick  |  December 31, 2020 at 11:43 am

                Hi Mark. Thanks for your explanations. I have no problem with domain-specific approaches and incremental progress. There are many different strategies and pathways for making progress. What’s important is to have similar end goals. What triggered me was your seeming embrace of Thorndike, which seems like a very different end goal. But I probably misinterpreted. Thanks again for engaging, and best wishes to you, Barb, and family for a healthy, joyful, creative new year. — Mitch

        • 17. shriramkrishnamurthi  |  December 30, 2020 at 11:54 am

          But you can get personal meaning and engagement with a lot less “freedom”. This matters if the curriculum also has learning goals, teachers have only preliminary qualifications, the curriculum is integrated into another discipline, etc. It doesn’t have to be a free-for-all to achieve meaning and engagement.

          Achieving this is not easy: it requires careful design. But it can be done. This was the focus of our paper on Bootstrap’s approach to game design, which is highly restricted, yet achieves engagement and meaning while enabling other curricular goals as well:

      • 18. Grant Hutchison  |  December 30, 2020 at 8:15 am

        Mark, these are new expectations so this is the first year that teachers are being asked to officially integrate coding into math education.

        There have been many initiatives in Canada to encourage coding in the classroom over the past 4 years includeing a national program of $110m which was made available to non-profit organizations to design and deliver. This program is soon coming to an end so it will be interesting to see what the next phase might be.

        Each jurisdiction (province) has been revising their educational learning expectations and many have integrated coding into various subject areas and/or revising their existing high school elective offerings. Ontario is now working on updates to secondary CS and Math this year.

        I agree that these Grade 3 expectations are quite high, but teachers are encouraged to support students as they design personally meaningful activities as Mitch suggested.

        Hopefully we are able to support the elementary teachers with this goal in mind.

        • 19. Mark Guzdial  |  December 30, 2020 at 9:48 am

          I’m so glad that Canada is making this investment in integrating coding, but the first line of the press release increases my fears that it’s for the wrong reasons: “Canada needs a workforce savvy in science, technology, engineering and math (STEM), capable of taking on challenges to compete in the digital economy.” Really? We’re going to worry about the IT workforce in 3rd grade?

  • 20. Mimi Recker  |  January 4, 2021 at 5:33 pm

    Hi Mark, interesting post.

    Because I’d hate for you to lose a bet, here are some considerations:

    In 2018, 501,530 students (F=273,610, M=227,920) took the AP U.S. History test whereas 72,187 students took AP CSP (F=22,72, M=49,466) – big sample and sample size differences and thus hard to compare. Personally, I think it is too early to lay any bets (but then, I never do well at poker).

    Also, the College Board states in their report: “we are unable to control for all factors that may influence our outcomes of interest and the results do not support causal claims.”


    • 21. shriramkrishnamurthi  |  January 5, 2021 at 8:37 am

      Also: the study is of DECLARED majors. Declaration does not mean completion. Since there are real questions about the content and its quality of preparation for collegiate CS, declaration is still a pretty weak signal.

      Also: how to dissociate the overall growth of CS from this? CS has grown by 5-10%/year at most institutions and most of those students did not have CSP. How was that factored out?

      • 22. Mark Guzdial  |  January 5, 2021 at 9:48 am

        If students do not continue in the CS major, do we blame the introductory course, or do we blame the rest of the curriculum? Certainly, the introductory course plays a role in retention and completion. I was pleased and a little surprised that UCSD had such great longitudinal effects from their PI+PP+MC intro course, that there was no difference in upper-level course performance compared to those who took the traditional intro course (see ICER 2020 paper here). Students who opt-in to different courses (different paces, traditional, honors, advanced, whatever) likely have different prior preparation. There’s a biasing effect of why a student would take a particular course. If CSP students didn’t continue in the major, would that be because the really prepared students went straight into CSA? Or because CSP students were looking for a broader definition of CS, but they only found a job prep program for FAANG companies?

        There’s so much going on here, and longitudinal studies are so rare in CS Ed, that I can’t fault the authors for not telling us more.

        • 23. shriramkrishnamurthi  |  January 5, 2021 at 10:38 am

          Of course. But one concern that some of us had about CSP is that it presents a glossy, shallow view of CS, which could lead to student shock when they confronted the collegiate discipline (at least as currently taught). I certainly see some of that effect. If that view gets more students in the door but also causes (proportionally) more to exit, then it should get both praise and blame.

          Which is not at all a defense of current CS curricula. They sure have issues. We certainly are doing a ton of work on trying to address some of those issues, and there’s tons more to be done. But the success story here (and the bet) is not about some idealized notion of CS; it’s about CS in something fairly close to its current form.

          I want to be super-clear that I’m in no way “blaming” the authors at all. They have reported the data they have, and from first glance, accurately and dispassionately. I may actually slightly be blaming you for a poor bet or for conceding too quickly. (-:

          (I AM actually blaming you for not conveying what the report says accurately enough. The word DECLARED is really critical. You presented it as “majoring in CS”, but that phrase can mean a lot of things, and can easily be interpreted as “graduating with a CS degree”. Someone reading your blog post and not reading the AP posting clearly enough — and maybe doesn’t know the timeline well enough to understand those first students can’t even have graduated yet — could easily get entirely the wrong impression from your post. As actually I did, until I clicked through. To my mind, the jury is still very much out. This is useful, positive initial information, but not yet time for a declaration of victory.)

          I’m still curious how to dissociate CSP from the general phenomenon of rising CS enrollments. That seems very critical to me to understanding the role of CSP. Phrases like “would have an impact” are statements of causality. How does this report show causality?

    • 24. Mark Guzdial  |  January 5, 2021 at 9:41 am

      Hi Mimi, how nice of you to visit!

      This is a constant problem with CS Ed. In no state do more than 10% of high school students take a CS class. Any claims we make about performance in a CS class (high school or university) are being made with a highly-biased (mostly male, white or Asian, high SES) small sample. We can obviously be much more confident about claims in reading, mathematics, and even social studies where the majority of students take them.

      This is why I think recruitment into CS and integration of CS into other subjects is critical to reach the goal of CS for All. Students are turned off by CS and are turning away from our classes.

  • […] To see themselves as capable of doing computer science (the goal of self-efficacy), […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,186 other subscribers


Recent Posts

Blog Stats

  • 2,060,872 hits
December 2020

CS Teaching Tips

%d bloggers like this: