Tennessee allows CS to count: Exploration of an impact claim

The announcement is good news:

Congratulations Tennessee! This year, for the first time, the State of Tennessee Board of Education allows high school computer science courses to count towards graduation requirements. Now, Advanced Placement Computer Science A satisfies a math requirement for all high Tennessee high school students.

via Tennessee now allows computer science to count toward high school graduation | Code.org.

Then there’s a claim later on the same page, “In these states, enrollment in computer science is higher (particularly among women and students of color), compared to the other states.”  That claim is intriguing.  Where’d they get this data?  I’d love to get CS enrollment data by state!  So I followed the link to this PDF.

Where I found this graph:

I don’t know where one can get AP CS class size data.  I’ve not seen that from the College Board.  As far as I can tell from the AP Report to the Nation, the College Board doesn’t have enrollment data.  What could they be counting to get these results, using variables from the College Board?

The numbers looked close to something that I’d seen in Barb’s data. So, I tried an analysis with Barb’s spreadsheet of AP CS data.  I created a “CountsCS” variable (1=on the Code.org list, 0=everyone else), and looked at the number of AP CS test takers in a state divided by the number of schools passing AP CS audit in the state.  I think of this as the “yield” — the number of actual test-takers by teacher (assuming one teacher per school, which is pretty much the rule for AP CS).  Below are the yield distributions for 2012 (with average and +/- standard deviation). These numbers look pretty close to the above, so I’m guessing that this is what they’re counting (for some year previous to 2012).  It is true that the average yield (not enrollment) for CountCS states is higher than for non-CountCS states. There isn’t a statistically significant difference, though (using t-test with a 95% confidence interval).

It could be that these distributions will become more distinct over time.  Some states (like Tennessee) have just made CS count.  It will take years to see an impact.

Digging deeper, I looked at the number of test-takers by state in terms of whether the state counts CS. Below is the distribution. There is on average more test-takers in states that count CS, but the distribution is broad.   There isn’t a statistically significant difference.

Given that the test-takers are not significantly different based on whether a state counts CS or not, I didn’t think that the minority or female numbers would either. It is true that there are on average more women test-takers from states that count CS, but the distribution is large. The difference is not statistically significant. The CountCS states include Vermont (which had 1 female AP CS test-taker in 2012), but does not include North and South Dakota, each of which had 2 female AP CS test-takers in 2012. (Alaska, Mississippi, Montana, and Wyoming all had zero female AP CS test-takers, and none of them count CS.)  I didn’t see significant differences based on under-represented minority groups.

If we really want to show that counting CS matters, we’d really want to do it a different way entirely.  We should compare the same state pre/post making the decision to count CS. Even then, we’d want to give it a few years to filter through the system (e.g., Juniors and Seniors in high school are unlikely to change their plans for graduation to take CS as soon as it counts).  I do believe that counting CS towards high school graduation will increase the number of students taking CS, but measuring that impact is challenging.