Posts tagged ‘CS1’

A biased attempt at measuring failure rates in introductory programming

Do students fail intro CS at higher rates than in comparable classes (e.g., intro Physics, or Calculus, or History)?  We’ve been trying to answer that question for years.  I studied that question here at Georgia Tech (see my Media Computation retrospective paper at last year’s ICER).  Jens Bennedsen and Michael Caspersen answered that question with a big international survey (see paper here).  They recognized the limitations of their study — it was surveying on the SIGCSE member’s list and similar email lists (i.e., to teachers biased toward being informed about the latest in computing education), and they got few responses.

This last year’s ITiCSE best paper awardee tried to measure failure rates again (see link below), by studying published accounts of pass rates.  While they got a larger sample size this way, it’s even more limited than the Bennedsen and Caspersen study:

  1. Nobody publishes a paper saying, “Hey, we’ve had lousy retention rates for 10 years running!”  Analyzing publications means that you’re biasing your sample to teachers and researchers who are trying to improve those retention rates, and they’re probably publishing positive results.  You’re not really getting the large numbers of classes whose results aren’t published and whose teachers aren’t on the SIGCSE members list.
  2. I recognized many of the papers in the meta-analysis.  I was co-author on several of them.  The same class retention data appeared in several of those papers.  There was no funny business going on.  We reported on retention data from our baseline classes.  We then tried a variety of interventions, e.g., with Media Computation and with Robotics.  The baseline then appears in both papers.  The authors say that they made sure that that didn’t double count any classes that appeared in two papers, but I can’t see how they could possibly tell.
  3. Finally, the authors do not explicitly cite the papers used in their meta-analysis.  Instead, they’re included on a separate page (see here).  SIGCSE shouldn’t publish papers that do this.  Meta-analyses should be given enough pages to list all their sources, or they shouldn’t be published.   Including them on a separate page makes it much harder to check the work, to see what data got used in the analysis.  Second, they are referencing work that won’t appear in any reverse citation indices or in the authors’ H-index calculations.  I know some of the authors of those papers who are up for promotion or tenure decisions this coming year.  Those authors are having impact through this secondary publication, but they are receiving no credit for it.

This paper is exploring an important question, and does make a contribution.  But it’s a much more limited study than what has come before.

Whilst working on an upcoming meta-analysis that synthesized fifty years of research on predictors of programming performance, we made an interesting discovery. Despite several studies citing a motivation for research as the high failure rates of introductory programming courses, to date, the majority of available evidence on this phenomenon is at best anecdotal in nature, and only a single study by Bennedsen and Caspersen has attempted to determine a worldwide pass rate of introductory programming courses.In this paper, we answer the call for further substantial evidence on the CS1 failure rate phenomenon, by performing a systematic review of introductory programming literature, and a statistical analysis on pass rate data extracted from relevant articles. Pass rates describing the outcomes of 161 CS1 courses that ran in 15 different countries, across 51 institutions were extracted and analysed. An almost identical mean worldwide pass rate of 67.7% was found. Moderator analysis revealed significant, but perhaps not substantial differences in pass rates based upon: grade level, country, and class size. However, pass rates were found not to have significantly differed over time, or based upon the programming language taught in the course. This paper serves as a motivation for researchers of introductory programming education, and provides much needed quantitative evidence on the potential difficulties and failure rates of this course.

via Failure rates in introductory programming revisited.

September 30, 2014 at 8:41 am Leave a comment

Enhancing syntax error messages appears ineffectual — if you enhance the error messages poorly

The ITICSE’14 paper referenced below is getting discussed a good bit in the CS Education community.  Is it really the case that enhancing error messages doesn’t help students?

Yes, if you do an ineffective job of enhancing the error messages.  I’m disappointed that the paper doesn’t even consider the prior work on how to enhance error messages in a useful way — and more importantly, what has been established as a better process.  To start, the best paper award at SIGCSE’11 was on an empirical process for analyzing the effectiveness of error messages and a rubric for understanding student problems with them — a paper that isn’t even referenced in the ITICSE paper, let alone applying the rubric.  That work and the work of Lewis Johnson in Proust point to the importance of bringing more knowledge to bear in creating useful error messages–by studying student intentionality, by figuring out what information they need to be successful.  Andy Ko got it right when he said “Programming languages are the least usable, but most powerful human-computer interfaces ever invented.”  We make them more usable by doing careful empirical work, not just tossing a bunch of data into a machine learning clustering algorithm.

I worry that titles like “Enhancing syntax error messages appears ineffectual” can stifle useful research.  I already spoke to one researcher working on error messages who asked if new work is even useful, given this result.  The result just comes from a bad job at enhancing error messages. Perhaps a better title would have been “An approach to enhancing syntax error messages that isn’t effective.”

Debugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including concrete examples that illustrate the kind of error that has occurred and how that kind of error could be corrected. We evaluate the effectiveness of the enhanced error messages with a controlled empirical study and find no significant effect.

via Enhancing syntax error messages appears ineffectual.

July 29, 2014 at 8:40 am 5 comments

Thoughts from Toronto on teaching code to librarians

The blog post linked below felt close to home, though I measure it differently than lines of code.  The base point is that we tend to start introductory programming courses assuming way more knowledge than is already there.  My experience this semester is that we tend to expect students to gain more knowledge more quickly than they do (and maybe, than they can).

I’m teaching Python Media Computation this semester, on campus (for the first time in 7 years).  As readers know, I’ve become fascinated with worked examples as a way of learning programming, so I’m using a lot of those in this class.  In Ray Lister terms, I’m teaching program reading more than program writing.  In Bloom’s taxonomy terms, I’m teaching comprehension before synthesis.

As is common in our large courses at Georgia Tech (I’m teaching in a lecture of 155 students, and there’s another parallel section of just over 100), the course is run by a group of undergraduate TA’s.  Our head TA took the course, and has been TA-ing it for six semesters.  The TA’s create all homeworks and quizzes.  I get to critique (which I do), and they do respond reasonably.  I realize that all the TA’s expect that the first thing to measure in programming is writing code.  All the homeworks are programming from a blank sheet of paper.  Even the first quiz is “Write a function to…”.  The TA’s aren’t trying to be difficult.  They’re doing as they were taught.

One of the big focal research areas in the new NSF STEM-C solicitation is “learning progressions.”  Where can we reasonably expect students to start in learning computer science?  How fast can we reasonably expect them to learn?  What is a reasonable order of topics and events?  We clearly need to learn a lot more about these to construct effective CS education.

I’m not going to articulate the next few orders of magnitude, both because they are not relevant to beginner or intermediate programmers, and because I’m climbing the 1K → 10K transition myself, so I’m not able to articulate it well. But they have to do with elegance, abstraction, performance, scalability, collaboration, best practices, code as craft.

The 3am realization is that many, many “introduction” to programming materials start at the 1 → 10 transition. But learners start at the 0 → 1 transition — and a 10-line program has the approachability of Everest at that point.

via thoughts from Toronto on teaching code to librarians.

January 29, 2014 at 1:49 am 26 comments

On-Line Course in Educational Robotics

This is from Jennie Kay, who was one of the organizers of the SIGCSE Robot Rodeo a few years ago, and is a leader in the use of robotics in CS education in the SIGCSE community.

Educational Robots for Absolute Beginners:

A Free On-Line Course that teaches the basics of LEGO NXT Robot Programming

Got a LEGO NXT robot kit but don’t know where to begin? Come learn the basics of LEGO NXT Robot Programming and discover a new way to bring math, science, and computer science content to your students both in and out of the classroom. By the end of this class, you (YES YOU!) will have built your own robot and programmed it to dance around the room.

This course, developed by the Rowan University Laboratory for Educational Robotics and supported by a generous grant from Google CS4HS, is specifically designed for K-12 teachers, but is free and open to anyone who is interested in learning about LEGO NXT robotics. The course will be starting at the end of October. Preregister now and we’ll send you an email when we open up the course. To preregister, as well as to see our video “trailer” and get the answers to frequently asked questions please visit: http://cs4hsrobots.appspot.com/

October 22, 2013 at 1:56 am 1 comment

What will programmers have to know by 2040?

Interesting claim below.  Do we believe that being able to build a JIT compiler will be a critical threshold for programming in 2040?  Or will programming become so much a literacy, that there will be people who can just write grocery lists and letters to Grandma and there will be Shakespeares?  I’m predicting a broader spread, not a higher bar.

The FizzBuzz problem described below is pretty interesting, a modern day version of the Rainfall problem.  I will bet that the results claimed for FizzBuzz are true, but I haven’t seen any actual studies of it yet.

While that may be true today, what will matter far more in the future is the quality of programmers, not the quantity. Any programmer who can’t hack together a JIT compiler in 2040 will be as useless as a programmer who can’t solve FizzBuzz today.

via Neil Fraser: News: Programming TNG.

September 27, 2013 at 1:38 am 5 comments

Success in Introductory Programming: What Works?

Leo Porter, Charlie McDowell, Beth Simon, and I collaborated on a paper on how to make introductory programming work, now available in CACM. It’s a shorter, more accessible version of Leo and Beth’s best-paper-award winning SIGCSE 2013 paper, with history and kibitzing from Charlie and me :

Many Communications readers have been in faculty meetings where we have reviewed and bemoaned statistics about how bad attrition is in our introductory programming courses for computer science majors (CS1). Failure rates of 30%–50% are not uncommon worldwide. There are usually as many suggestions for how to improve the course as there are faculty in the meeting. But do we know anything that really works?

We do, and we have research evidence to back it up. Pair programming, peer instruction, and media computation are three approaches to reforming CS1 that have shown positive, measurable impacts. Each of them is successful separately at improving retention or helping students learn, and combined, they have a dramatic effect.

via Success in Introductory Programming: What Works? | August 2013 | Communications of the ACM.

August 5, 2013 at 1:40 am 13 comments

edX offers a CS1 MOOC via Massachusetts community colleges

Definitely the most interesting MOOC experiment I’ve seen in the latest batches — an edX CS1 aimed at community college students, and offered in a blended format.  I very much hope that they do good assessment here.  If MOOCs are going to serve as an alternative to face-to-face classes for the majority of students, they have to work at the community college level and have better than face-to-face retention rates.  Retention (and completion) rates are too low already in community colleges.  If MOOCs are going to be part of a solution, part of making education better, then they need to have high completion rates.

The fast-moving world of online education, where anyone can take classes at a world-famous university, is making a new foray into the community college system, with a personal twist.

In a partnership billed as the first of its kind, the online education provider edX plans to announce Monday that it has teamed up with two Massachusetts community colleges to offer computer science classes that will combine virtual and classroom instruction.

Beginning next term, Bunker Hill and MassBay community colleges will offer versions of an online MIT course that will be supplemented with on-campus classes. Those classes, to be taught by instructors at the two-year schools, will give students a chance to review the online material and receive personal help.

“This allows for more one-to-one faculty mentoring” than exclusively online courses, said John O’Donnell, president of MassBay Community College in Wellesley. O’Donnell added that the schools’ involvement allows edX “to test its course content on a broader range of students.”

Students will pay the same amount they would for a standard class.

via edX expands offerings to Mass. community colleges – Metro – The Boston Globe.

November 23, 2012 at 8:05 am 6 comments

Older Posts


Recent Posts

October 2014
M T W T F S S
« Sep    
 12345
6789101112
13141516171819
20212223242526
2728293031  

Feeds

Blog Stats

  • 965,682 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 3,134 other followers

CS Teaching Tips


Follow

Get every new post delivered to your Inbox.

Join 3,134 other followers