Enhancing syntax error messages appears ineffectual — if you enhance the error messages poorly

July 29, 2014 at 8:40 am 6 comments

The ITICSE’14 paper referenced below is getting discussed a good bit in the CS Education community.  Is it really the case that enhancing error messages doesn’t help students?

Yes, if you do an ineffective job of enhancing the error messages.  I’m disappointed that the paper doesn’t even consider the prior work on how to enhance error messages in a useful way — and more importantly, what has been established as a better process.  To start, the best paper award at SIGCSE’11 was on an empirical process for analyzing the effectiveness of error messages and a rubric for understanding student problems with them — a paper that isn’t even referenced in the ITICSE paper, let alone applying the rubric.  That work and the work of Lewis Johnson in Proust point to the importance of bringing more knowledge to bear in creating useful error messages–by studying student intentionality, by figuring out what information they need to be successful.  Amy Ko got it right when she said “Programming languages are the least usable, but most powerful human-computer interfaces ever invented.”  We make them more usable by doing careful empirical work, not just tossing a bunch of data into a machine learning clustering algorithm.

I worry that titles like “Enhancing syntax error messages appears ineffectual” can stifle useful research.  I already spoke to one researcher working on error messages who asked if new work is even useful, given this result.  The result just comes from a bad job at enhancing error messages. Perhaps a better title would have been “An approach to enhancing syntax error messages that isn’t effective.”

Debugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including concrete examples that illustrate the kind of error that has occurred and how that kind of error could be corrected. We evaluate the effectiveness of the enhanced error messages with a controlled empirical study and find no significant effect.

via Enhancing syntax error messages appears ineffectual.

Entry filed under: Uncategorized. Tags: , , .

New ExploringCS Working Paper: How do we avoid CS10K going to CS5K only five years later? Why MOOCs don’t help CS Education: Learning to lighten up

6 Comments Add your own

  • 1. Leigh Ann DeLyser  |  July 29, 2014 at 2:21 pm

    I also think we should distinguish between “error” and “syntax” messages. Error messages can be very useful especially, if like you say, they bring more information to the table.

    • 2. Kathi Fisler  |  July 29, 2014 at 2:27 pm

      I’ve seen three broad classes of error messages: syntax, static, and run-time. Not surprisingly, each seems to require different explanatory mechanisms (and different kinds of supporting information). Designing that additional information for usability is another question entirely.


  • 3. Kathi Fisler  |  July 29, 2014 at 2:24 pm

    This paper illustrates one of the “trends” that worries me: the tendency to do whatever analysis we can do automatically, rather than the analysis that actually gets at the issue we want to understand.

    The coding scheme used in this paper appears to not even distinguish between students getting the same error twice in a row from getting two different error messages in successive runs. That’s a critical difference for evaluating the effectiveness of error messages.

    Ultimately, this study needed either more manual coding of the data or a talk-aloud component (both of which we had to use to make sense of the data in our SIGCSE ’11 paper that Mark referenced — thanks!). I’d hate to think that the CSEd community is “evolving” to adore automated analysis so much that we reward work that misuses it.


  • 4. Andrew Luxton-Reilly  |  July 30, 2014 at 12:31 am

    Firstly, thanks for reading the paper. It is always good to know that someone is reading the research, even if you didn’t like it!

    I think the main consideration here is the transferability of research findings. The paper makes no claims that *all* error message are ineffectual – only that *this* attempt to provide additional information to students about their errors didn’t seem to help. In this case, as in most other educational research, the burden is upon the reader to interpret the results and decide for themselves the degree to which this result might transfer to their situation.

    I would love to see more research in this area that focused on student understanding of error messages, and what kinds of information *would* be effective, particularly in “live” environments rather than in artificial research settings. I would certainly be saddened to think that any researcher who has actually read the paper would think that it is now pointless to pursue research in this area.


    • 5. Michael S. Kirkpatrick  |  July 30, 2014 at 10:03 am

      Having looked at the paper, I can see where you’re coming from. I think the problem (and objection) is that certain elements of your writing blur this fact. As Mark points out, the title and the abstract seem to suggest that all attempts at enhancing error messages would be ineffective. In the paper, you make several statements like, “Does the enhanced feedback add value?” A clearer word choice may have been, “Does this approach to enhanced feedback…”

      Honestly, if I had read the paper without paying close attention to this subtle linguistic twist, I would have left with the conclusion that all attempts will be unsuccessful. While there may be problems with the research itself (see Kathi’s comments on coding…I don’t have enough experience there to comment), it seems that the main objection is the use of imprecise language that implies an overstatement of your thesis.

  • […] Carpenter published a study providing evidence that enhancing CEMs was not effective, generating a bit of discussion on Mark Guzdial’s Blog. Although these papers came up with opposing conclusions, there are a ton of variables involved in […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,038 other followers


Recent Posts

Blog Stats

  • 2,012,742 hits
July 2014

CS Teaching Tips

%d bloggers like this: