Time spent is about expertise developed: Programmer productivity is an education problem

November 28, 2014 at 8:40 am 8 comments

Joel Spolsky argues that the time spent on an assignment by students is not correlated with the results.  This is part of his argument that there is a 5x to 10x productivity gap between the best and average programmers.

There’s just nothing to see here, and that’s the point. The quality of the work and the amount of time spent are simply uncorrelated. I asked Professor Eisenstat about this, and he pointed out one more thing: because assignments are due at a fixed time usually midnight and the penalties for being late are significant, a lot of students stop before the project is done. In other words, the maximum time spent on these assignments is as low as it is partially because there just aren’t enough hours between the time the assignment is handed out and the time it is due. If students had unlimited time to work on the projects which  would correspond a little better to the working world, the spread could only be higher.

via Hitting the High Notes – Joel on Software.

I did my Blog@CACM post in November on this argument (see the post here).  Joel is measuring the wrong thing in his experiment.  Time spent doesn’t correlate with quality.  Time on task correlates with learning.  More knowledge correlates with greater quality, but you don’t know much about how much is already known by the students in the class that he’s studying.  Therefore, you can’t possibly make an argument about productivity when the critical variable is unknown.

There may be a productivity gap between the best and average programmers.  There’s a productivity gap between the best and average practitioners in any field.  The issue is whether we can generate the performance at the top end.  Can we make someone highly productive?  It’s a design problem.   It’s an education problem.

Entry filed under: Uncategorized. Tags: , , , .

2014 CSEdWeek, Hour of Code, and (new!) Georgia Day of Code The dark at the end of the funnel: The pipeline in computing education leads to a sewage plant

8 Comments Add your own

  • 1. Neil Brown  |  November 28, 2014 at 8:53 am

    A colleague asked me for help last week with a problem that they had spent hours trying to solve and got nowhere. I solved it in under a minute. That’s at least a 100x difference, measured on the day! How did I solve it so quickly? Because I had spent a day last month with exactly the same issue until I figured it out.

    Because programming involves reasoning, I think we often lose sight of how much of it is based on recall or pattern recognition (both for code writing and for debugging). Thus we tend to equate intelligence with reasoning with programming ability. Not to mention that everyone considers themselves on the left-hand side of the 10:1 ratio, not the right-hand side.

  • 2. Bonnie  |  November 28, 2014 at 9:01 am

    I disagree with the assertion that students hand in incomplete work because ” there just aren’t enough hours between the time the assignment is handed out and the time it is due”. My students also tend to hand in incomplete assignments, but I give them the assignment several weeks before the due date. Because we use an online discussion board for project questions, I can tell pretty well when the students start working on the project – usually the day before the due date. I have not been able to change this behavior over several years of trying things like requiring initial designs, requiring initial attempts, or test cases. There are plenty of hours between the time the assignment is handed out and the time it is due. The students just choose not to use those hours.

  • 3. rademi  |  November 28, 2014 at 9:03 am

    In that particular dataset, Joel claimed: “No points are deducted for … lateness.” This directly contradicts Professor Eisenstat’s statement “the penalties for being late are significant”. So something is not making sense here.

    That said, personally, I do not know how to quantify prior knowledge.

    That said, if I read between the lines, the issue Joel is trying to select for is probably prior knowledge.

  • 4. Steve Tate  |  November 28, 2014 at 12:12 pm

    Mark says: “The issue is whether we can generate the performance at the top end.”

    I don’t think that’s the right issue. I know you want to believe that high quality teaching can overcome all, but I simply don’t believe that’s true. That belief is not based on any research and on anything more concrete than years of teaching experience, but frankly I think it’s obvious. Nothing in the world would be able to get ME to “performance at the top end” in writing poetry, playing basketball, or working in politics.

    I think the right question is whether we can make people BETTER at what they do. Even if the odds of me winning the Nobel Prize in Literature will never exceed zero, I could certainly improve my writing if I chose to spend time working on it, and a good teacher could be a huge help.

    This is my goal when I teach: To make every student (who is willing to make the effort) better at whatever that class is. It’s a question of gain. Not everyone will be a star, but there are plenty of people in plenty of areas that are effective in the minor leagues (or niche authors or community theater or local politics or…), and there’s nothing wrong with that.

    • 5. Mark Guzdial  |  November 28, 2014 at 12:27 pm

      I used too few words in that statement to convey my intent. This might be closer: “The research issue is to determine (a) what led to the performance at the top end and (b) whether we can engineer those experiences. If we can create those experiences explicitly (rather than as an accident or as a result of privilege), then we provide access to high levels of performance to those who are motivated to achieve those levels of performance but might not have the opportunity without the engineering provided by good schools with good teachers.” I would also hope that achieving (a) and (b) would give us insight to improve performance for students who aim to do better, even if they don’t want to achieve the highest level of performance. I see students in my teaching who want to get better, but don’t know how. No, I don’t think everyone has to be a star.

      • 6. Steve Tate  |  November 28, 2014 at 1:10 pm

        Yes! This is the part that I think is key: “I see students in my teaching who want to get better, but don’t know how.”

        I see an unfortunate number of faculty who don’t appreciate the power of that statement. Faculty are in general a bunch of over-achievers who probably never had to struggle with the issue of not knowing how to improve. To be truly effective, it’s not enough just to teach as you wish you had been taught, because that’s not going to be effective with a large number of students. But unfortunately, that’s what we have – faculty receive basically no training in how to teach, just a “go my child and do as I have done” approach that seems to suggest that because people have TAKEN classes and succeeded then they should be able to TEACH those classes.

        Most faculty honestly aren’t going to care about this, and it’s questionable whether it is even in their best career interests to care about it (particularly at research universities). That makes your job (and others who really care about this stuff) all the more important. If people who are dedicated to this can produce pedagogical materials and get them used so that good teaching practices are easier to adopt than bad ones, then faculty who don’t want to spend time thinking about education research will be pulled along even if they themselves don’t realize it.

        So go ahead Mark and change the way computer science is taught for the better – it’s just a little task, and hurry up with it will you? Is that too much to ask? 🙂

  • 7. gasstationwithoutpumps  |  November 28, 2014 at 3:47 pm

    Depending on the assignment, I would expect to see one of three patterns:

    1) top students take the least time, bottom students the most. Assignments that are easy for those who already know the material should see an inverse correlation between time spent and performance. These are good as warmup exercises, but are only really useful for the bottom half of the class. I give few of this sort of assignment.

    2) top students take the most time, bottom the least. Assignments that are extremely difficult and interesting will cause the top students to spend a lot of time on them, but weaker students to get lost and give up. These are the most interesting assignments to grade, but are are really only useful for the top half of the class. Some of my assignments end up in this category.

    3) top and bottom students take more time than students in the middle. Moderately challenging assignments with “bonus” parts will cause the top students to attempt the bonus parts, middle students to do the minimum required, and bottom students to struggle for a long time. This seems to be where a lot of my assignments end up, and I’m ok with that.

  • 8. Andy "SuperFly" Rundquist  |  November 28, 2014 at 8:19 pm

    This reminds me a little of a conversation I’ve been having with people in the Standards-Based Grading camp. I’ve been arguing that if you want all the standards to be worth the same, a good guide is how much time you spend on it in class. Others have said that for some not-so-important ideas we spend a lot of time in class, maybe because the math is hard or whatever.

    I connect that conversation to this one because I agree that time spent can correlate with gain. If, over the years, an instructor has found that some not-so-important idea needs a whole class day, it’s likely that for some students it’s quite a gain to learn that topic.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,184 other followers


Recent Posts

Blog Stats

  • 2,039,343 hits
November 2014

CS Teaching Tips

%d bloggers like this: