Results from Longitudinal Study of Female Persistence in CS: AP CS matters, After-school programs and Internships do not
October 14, 2019 at 7:00 am 12 comments
NCWIT has been tracking their Aspirations in Computing award applicants for several years. The Aspirations award is given to female students to recognize their success in computing. Tim Weston, Wendy DuBow, and Alexis Kaminsky have just published a paper in ACM TOCE (see link here) about their six year study with some 500 participants — and what they found led to persistence into CS in College. The results are fascinating and somewhat surprising — read all the way to the end of the abstract copied here:
While demand for computer science and information technology skills grows, the proportion of women entering computer science (CS) fields has declined. One critical juncture is the transition from high school to college. In our study, we examined factors predicting college persistence in computer science and technology related majors from data collected from female high school students. We fielded a survey that asked about students’ interest and confidence in computing as well as their intentions to learn programming, game design, or invent new technology. The survey also asked about perceived social support from friends and family for pursuing computing as well as experiences with computing, including the CS Advanced Placement (AP) exam, out-of-school time activities such as clubs, and internships. Multinomial regression was used to predict persistence in computing and tech majors in college. Programming during high school, taking the CS Advanced Placement exam, and participation in the Aspirations awards program were the best predictors of persistence three years after the high school survey in both CS and other technology-related majors. Participation in tech-related work, internships, or after-school programs was negatively associated with persistence, and involvement with computing sub-domains of game design and inventing new applications were not associated with persistence. Our results suggest that efforts to broaden participation in computing should emphasize education in computer programming.
There’s also an article at Forbes on the study which includes recommendations on what works for helping female students to persist in computing, informed by the study (see link here). I blogged on this article for CACM here.
That AP CS is linked to persistence is something we’ve seen before, in earlier studies without the size or length of this study. It’s nice to get that revisited here. I’ve not seen before that high school work experience, internships, and after-school programs did not work. The paper makes a particular emphasis on programming:
While we see some evidence for students’ involvement in computing diverging and stratifying after high school, it seems that involvement in general tech-related fields other than programming in high school does not transfer to entering and persisting in computer science in college for the girls in our sample. Understanding the centrality of programming is important to the field’s push to broaden participation in computing. (Italics in original.)
This is an important study for informing what we do in high school CS. Programming is front-and-center if we want girls to persist in computing. There are holes in the study. I keep thinking of factors that I wish that they’d explored, but they didn’t — nothing about whether the students did programming activities that were personally or socially meaningful, nothing about role models, and nothing about mentoring or tutoring. This paper makes a contribution in that we now know more than we did, but there’s still lots to figure out.
Entry filed under: Uncategorized. Tags: BPC, computing education, NCWIT, women in computing.
Computing Ed Research - Guzdial's Take 
1.
Raul Miller | October 14, 2019 at 11:56 am
I agree (gut feeling) with your closing remark here.
Where activities have not contributed, it’s worth understanding what distinguishes successes in those activities from the failures. (Value of this kind of study, I imagine, corresponds roughly to the amount of effort already going into those activities…)
2.
gasstationwithoutpumps | October 14, 2019 at 12:21 pm
I wonder how much of the effect here is selection—those who were most interested and most dedicated took AP exams in high school, so are most likely to persist. It is always difficult to distinguish between selection effects and effects of interventions when the interventions are elective.
I wonder a bit about the negative effect of internships—is it that high-school internships are mainly a way to get no-cost labor to do stuff way too boring for adults? or are they exposing students to a toxic work environment? or are the internships too soon—when students don’t have the skills to do anything useful for the company and so see themselves as not suited for the jobs?
3.
Mark Guzdial | October 14, 2019 at 12:30 pm
I share your interest in what’s negative about the internships. The paper expands on this a bit, though not enough to answer all these questions. The authors particularly highlight that internships and afterschool programs that emphasize “web design” tend not to give students technical skills.
4.
Raul Miller | October 14, 2019 at 12:35 pm
I suspect that there’s nothing wrong with self selection being relevant. We need people in a variety of roles, and personal interest can be a good mechanism for informing those choices.
But, more than that, I think we need to realize that there’s a self selection element out-of-school time activities, also. So if self-selection were the significant issue, why does it not play out for participation in clubs and internships?? (But, also, were some types of internships fruitful where others were not? For example, contrasting those at institutions with a solid research reputation with others, or maybe certain leaders or certain business practices have a positive or negative influence? Or was it all uniformly mediocre? Do we even have the kind of research discipline necessary to collect this kind of information? Or do administrative realities prevent us from seeing this kind of detail?)
That said… if we are casting wide, trying to understand what’s going on here, we might also want to investigate not just persistence into college but progress through and maybe beyond college (difficult to investigate, perhaps, but probably not irrelevant).
5.
gasstationwithoutpumps | October 14, 2019 at 1:32 pm
Self-selection is highly relevant—that’s the problem when trying to understand what interventions to try, as the selection bias wipes out any signal about what the intervention actually does.
The AP course was the most time-consuming and difficult of the interventions mentioned, so selection bias alone would explain the entire effect, leaving no evidence that teaching programming in high school actually increases persistence. But the data will be used to argue that teaching programming is the most important intervention. Then someone will put together an expensive program to require programming of everyone, which will fail to have much effect. (This cycle of misinterpreted research on education followed by failed interventions followed by new research has been seen repeatedly in math, reading, and other fields, so it will not be surprising to see it in CS education as well.)
6.
Bonnie | October 14, 2019 at 1:03 pm
What is a “tech related” internship at the high school level? Judging from the internships that my HS kids participated in (they are required by their HS), they are nothing more than installing PCs, helping a business person with spreadsheets, or doing a little web design. Nothing much that is useful for a CS major is learned in those internships. OTOH, the AP CS course is very strong. My oldest took it, is now a CS major, and boy did that course help. Taking AP CS is simply better preparation for college CS then plugging in PCs in a nursing home.
7.
gasstationwithoutpumps | October 14, 2019 at 1:40 pm
I agree that most high-school tech internships are not going to show students much about CS. The internship programs seem primarily aimed at getting kids into vocational positions that they might not have considered, rather than motivating them to go on to college.
For that matter, most colleges don’t have the capacity to handle any more CS students, so I’m not sure why we are trying to encourage more high school students to go into CS.
I respectfully disagree about AP CS A being “very strong”. It is better prep than setting up PCs, but it is a rather generic first-semester programming course (and CS Principles isn’t even that, being a non-major CS course at best). AP seems to be moving in the direction of mainly serving high-school-level courses (Physics 1 is high-school level, and only the much less common Physics C is college level; AP statistics is high-school level; CS Principles is high-school level; … ). I have no objection to AP being high-school level courses (after all, high-school students are the target market), but misleading claims about them being real replacements for college-level courses are not doing students a service.
Disclaimer: I have the same objection to colleges giving college-credit for high-school level courses, though this seems also to have become very common (even at R1 institutions like the University of California).
8.
Megan Lutz | October 15, 2019 at 1:02 pm
AP statistics is non-calc based, but it is more in-depth than many intro statistics courses at the college course. Similarly, AP physics is non-calc based but it covers physics at the depth and level that many non-engineering and non-technical majors require of a lab science. Let us not let our technical/mathematical snobbery blind us to the value of intro/methods courses.
That said, I have other concerns with regard to this study, its methods, and its reported outcomes. One, as was discussed in its limitations, is that its sample is non-representative of college-bound women, in that 42% of the sample took AP CS, whereas women are otherwise grossly underrepresented in AP CS. That AP CS is about the strongest predictor (i.e., with the largest effect) in all models reported is not that surprising, as the sample is a group of very special women. Other issues in effect size reporting (effects with “significant” p-values were discussed, while other larger effects with slightly higher p-values were ignored entirely, such as the underrepresented minorities cohort in the final model).
A second concern is that the study is misrepresented as longitudinal. While data were collected at two timepoints (two surveys in 2013 and 2016), a truly longitudinal study would compare the change in the responses over those time points, rather than using them as explanatory variables of a separate response. These data are instead cross-sectional measurements of the respondents at those time points, but no measurement of growth has been made or attempted.
9.
Importance of considering race in CS education research and discussion | Computing Education Research Blog | February 17, 2020 at 7:00 am
[…] article from Tim Weston and collaborators based on NCWIT Aspirations award applicants, which I blogged about here. I was telling him about the results — what correlated with women’s persistence in […]
10.
SIGCSE 2020: Papers freely available, AP CSA over AP CSP for diversifying computing, and a tour of computing ed research in one hour | Computing Education Research Blog | May 4, 2020 at 7:00 am
[…] there is likely some selection bias here. This result reminds me of the Weston et al result (see blog post here) showing that the female high school students they studied continued on to tech and computing […]
11.
Goals for CS Education include Getting Students In the Door and Supporting Alternative Endpoints | Computing Education Research Blog | June 1, 2020 at 7:00 am
[…] A focus on programming is critical for later success in the areas that Portnoff describes. The latest research supporting that argument comes from Joanna Goode (as I described in this blog post), one of the educators Portnoff critiques. Joanna was co-author on a paper showing that AP CS A success is more likely to predict continuation in CS than AP CSP success. I’m also swayed by the Weston et al. article showing that learning to program led to greater retention among female students in the NCWIT Aspirations awards programs (see link here). […]
12.
The goal of the first CS course should be to promote confidence if we’re going to increase diversity in CS: Paying off on a bet | Computing Education Research Blog | December 29, 2020 at 7:00 am
[…] and tech careers. ACM TOCE in 2019 published a paper using NCWIT Aspirations award winners (see blog post link here) showing that taking the CS Advanced Placement A exam was one of the best predictors of persistence […]