A careful model-based analysis of AP CS 2013 exam data: CS commenters need to learn statistics

January 18, 2014 at 4:12 pm 8 comments

There has been a lot of media attention to Barb’s analysis of AP CS 2013 exam data, but not all of it has been well-informed.  We both really enjoyed reading the Gas Station Without Pumps analysis, quoted and linked below.  Not only is it a careful, model-based analysis, but it’s a nice explanation too.  I learned more about how to measure under-representation — recommended reading.

There are states that do have significant under-representation of women: for example, Utah had 103 test takers, only 4 of whom were women. With an expected number of about 51.5, this is p<1.4E-16. Even with 51× multiple hypothesis correction, this under-representation is hugely significant.  Looking nationwide, total counts were 5485 female test takers out of 29555 total test takers.  That’s p< 1.4E-1677. The highest percentage of female test takers was in Tennessee, with 73 out of 251, which is  p< 2.6E-7, again highly significant.

Tennessee also had a high proportion of black test takers with 25 out of 251.  With an expected number of 42.12, this is p<0.003 (still significantly under-represented).  To see if black students were under-represented nationwide, one would have to add up the expected numbers for each state and see how the actual number compared with the expected number.

via CS commenters need to learn statistics | Gas station without pumps.

Entry filed under: Uncategorized. Tags: , , .

Data Mining Exposes Embarrassing Problems For Massive Open Online Courses: There is no dialogue Taking on the challenge of creating digital technologies with an online course for teachers

8 Comments Add your own

  • 1. Mike Zamansky  |  January 18, 2014 at 4:35 pm

    This is a nice write up.

    I can’t help but wonder if the popular media headlines might have been better and thus made a more solid case had Barbara not, in her first and third bullet points used the exact language lifted by the press and refuted by Gasstationswithoutpumps as “interesting findings.”

    • 2. Mark Guzdial  |  January 18, 2014 at 4:43 pm

      You might be right, Mike. If I’d known how to do the analysis that Kevin did, I would have. In any case, Barb didn’t lie. All the bullets were true. It’s just that some of the observations weren’t significant.

      • 3. Mike Zamansky  |  January 18, 2014 at 4:53 pm

        Mark – I’m certainly not saying that there’s any lying going on but I do think that we all have to be very careful about how we represent things given the media’s and various interests ability and desire to spin.

        When someone like Barb or yourself, both well respected voices as far as I’m concerned say “no females in…” we can count on the media’s response to be tl;dr and run with the bullet point.

    • 4. gasstationwithoutpumps  |  January 18, 2014 at 5:03 pm

      The problem is that the media coverage is largely driven by the sensational “no women” headlines, which are a consequence mainly of “almost no CS test takers in these states”. The message being conveyed (that women and blacks are way under-represented) is true, but not on the basis of the states with no female or no black test takers. It is in the states that have moderate numbers of CS test takers that the under-representation is statistically most significant. But zero is emotionally a special number, so getting the publicity based on the emotional appeal may not have been a bad thing.

  • 5. Cecily Heiner  |  January 21, 2014 at 11:36 am

    So, clearly, based on both Barbara’s analysis and Gas Station Without Pumps analysis, Utah is WAY under-represented in terms of AP CS test takers. This is pretty easy for me to explain; here in Utah, it is well known than the college credit you get for AP CS will not help you graduate. You are better off taking calculus or physics. My friends and I all knew this when we started our senior year of high school and it was a significant factor in our decision NOT to take AP CS, even though several of us were planning on a CS major in college. My question for bloggers/readers in other states; does AP CS help you graduate from college faster in other states– does it fill at least one non-elective requirement for at least one college major in at least one or more colleges or universities? If yes, which one?

    • 6. Mark Guzdial  |  January 21, 2014 at 11:41 am

      I don’t understand that, Cecily. The decision of how to interpret the AP CS exam results is the College’s, not the high school’s nor the state’s. So, if a Utah student gets a 5 on AP CS and applies to Georgia Tech, she WILL get advanced placement and will graduate sooner. Are you saying that NO Colleges and Universities in Utah give any credit to AP CS? Or that Utah high school students only apply to Colleges and Universities in Utah that don’t give any credit for passing AP CS?

      • 7. Mike Zamansky  |  January 21, 2014 at 1:35 pm

        As far as my school goes – our kids are finding that while there’s more and more pressure to take AP classes, the kids are getting less and less credit.

        For example, my daughter’s college will give placement but no credit for a 5 on APCS and she could just as easily gotten placement by taking an internal placement exam (or in her case, showing her GitHub account to the professor showing she knew her stuff).

        It seems that Calc, AP Science classes, and foreign language APs are more likely to do something for at least our kids than the APCS exam will.

      • 8. Cecily  |  January 21, 2014 at 4:14 pm

        Georgia Tech currently has 0 undergraduate students enrolled from Utah(or Wyoming), so I am not sure I understand the relevance of the argument that Georgia Tech gives advanced placement, so students in UT(or WY) should take AP CS. Realistically, the VAST majority of students in Utah will go to BYU-Provo or the University of Utah if they can. If you don’t there, you might go to BYU-Idaho, BYU- Hawaii, Utah Valley University, Utah State University, Southern Utah University, or a regional . At BYU, AP CS gives you credit for CS 103. However, to my knowledge, no BYU majors including computer science requires CS103. Instead, the first required class is CS142. At the University of Utah, you can get out of CS1 with AP CS, but you have to have a 5 on the test, not the typical passing score of 3. Since only a quarter of students who actually take the test hit that bar; it seems a little high to me. At Utah State University, AP CS A gives you 3 elective credits. SUU is actually pretty generous with AP credit, giving them credit for the first year, but a large number of our students come from rural high schools where there is not enough enrollment to offer AP CS. Thus, most college bound students are not motivated or able to take AP CS for college credit or “advanced placement” as you call it. By contrast, if you take AP Calculus, you partially fulfill your GE math requirement, you partially fulfill your major math requirement, and you reduce the number of credits you must finish in college to complete your degree, and even the small rural and charter schools are VERY motivated to offer AP calculus.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,005 other followers


Recent Posts

Blog Stats

  • 1,879,118 hits
January 2014

CS Teaching Tips

%d bloggers like this: