Tennessee allows CS to count: Exploration of an impact claim

The announcement is good news:

Congratulations Tennessee! This year, for the first time, the State of Tennessee Board of Education allows high school computer science courses to count towards graduation requirements. Now, Advanced Placement Computer Science A satisfies a math requirement for all high Tennessee high school students.

Then there’s a claim later on the same page, “In these states, enrollment in computer science is higher (particularly among women and students of color), compared to the other states.”  That claim is intriguing.  Where’d they get this data?  I’d love to get CS enrollment data by state!  So I followed the link to this PDF.

Where I found this graph:

I don’t know where one can get AP CS class size data.  I’ve not seen that from the College Board.  As far as I can tell from the AP Report to the Nation, the College Board doesn’t have enrollment data.  What could they be counting to get these results, using variables from the College Board?

The numbers looked close to something that I’d seen in Barb’s data. So, I tried an analysis with Barb’s spreadsheet of AP CS data.  I created a “CountsCS” variable (1=on the Code.org list, 0=everyone else), and looked at the number of AP CS test takers in a state divided by the number of schools passing AP CS audit in the state.  I think of this as the “yield” — the number of actual test-takers by teacher (assuming one teacher per school, which is pretty much the rule for AP CS).  Below are the yield distributions for 2012 (with average and +/- standard deviation). These numbers look pretty close to the above, so I’m guessing that this is what they’re counting (for some year previous to 2012).  It is true that the average yield (not enrollment) for CountCS states is higher than for non-CountCS states. There isn’t a statistically significant difference, though (using t-test with a 95% confidence interval).

It could be that these distributions will become more distinct over time.  Some states (like Tennessee) have just made CS count.  It will take years to see an impact.

Digging deeper, I looked at the number of test-takers by state in terms of whether the state counts CS. Below is the distribution. There is on average more test-takers in states that count CS, but the distribution is broad.   There isn’t a statistically significant difference.

Given that the test-takers are not significantly different based on whether a state counts CS or not, I didn’t think that the minority or female numbers would either. It is true that there are on average more women test-takers from states that count CS, but the distribution is large. The difference is not statistically significant. The CountCS states include Vermont (which had 1 female AP CS test-taker in 2012), but does not include North and South Dakota, each of which had 2 female AP CS test-takers in 2012. (Alaska, Mississippi, Montana, and Wyoming all had zero female AP CS test-takers, and none of them count CS.)  I didn’t see significant differences based on under-represented minority groups.

If we really want to show that counting CS matters, we’d really want to do it a different way entirely.  We should compare the same state pre/post making the decision to count CS. Even then, we’d want to give it a few years to filter through the system (e.g., Juniors and Seniors in high school are unlikely to change their plans for graduation to take CS as soon as it counts).  I do believe that counting CS towards high school graduation will increase the number of students taking CS, but measuring that impact is challenging.

Entry filed under: Uncategorized. Tags: , , , , , .

• 1. geekymom/Laura  |  October 9, 2013 at 6:18 am

You’re also looking at just ap. Plenty of schools offer CS at the none AP level. We don’t have AP courses in any subjects, which is true of quite a few independent schools. I believe there is some data on CS courses more generally.

• 2. Mark Guzdial  |  October 9, 2013 at 7:13 am

Such data can be found, state-by-state. There is no national data source that I have found on CS enrollment. FERPA makes it difficult to collect and share these data.

• 3. gasstationwithoutpumps  |  October 9, 2013 at 11:27 am

Um, Mark, what sense does it make to compare raw numbers of test takers by state? States vary in size far more than any effect you might be interested in. At least normalize by the population, teenage population, number of high school graduates, number of AP tests of any sort, or some other measure that should correct for state size.

• 4. Mark Guzdial  |  October 9, 2013 at 11:57 am

Yeah, I did. It’s not statistically significant either, and I decided that I was going overboard in the analysis already. I’ve got to wait on a publication under review before I say much more, but I hope it is sufficient to say: I agree, and I have more results that I hope to share.