Enhancing syntax error messages appears ineffectual — if you enhance the error messages poorly

The ITICSE’14 paper referenced below is getting discussed a good bit in the CS Education community.  Is it really the case that enhancing error messages doesn’t help students?

Yes, if you do an ineffective job of enhancing the error messages.  I’m disappointed that the paper doesn’t even consider the prior work on how to enhance error messages in a useful way — and more importantly, what has been established as a better process.  To start, the best paper award at SIGCSE’11 was on an empirical process for analyzing the effectiveness of error messages and a rubric for understanding student problems with them — a paper that isn’t even referenced in the ITICSE paper, let alone applying the rubric.  That work and the work of Lewis Johnson in Proust point to the importance of bringing more knowledge to bear in creating useful error messages–by studying student intentionality, by figuring out what information they need to be successful.  Amy Ko got it right when she said “Programming languages are the least usable, but most powerful human-computer interfaces ever invented.”  We make them more usable by doing careful empirical work, not just tossing a bunch of data into a machine learning clustering algorithm.

I worry that titles like “Enhancing syntax error messages appears ineffectual” can stifle useful research.  I already spoke to one researcher working on error messages who asked if new work is even useful, given this result.  The result just comes from a bad job at enhancing error messages. Perhaps a better title would have been “An approach to enhancing syntax error messages that isn’t effective.”

Debugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including concrete examples that illustrate the kind of error that has occurred and how that kind of error could be corrected. We evaluate the effectiveness of the enhanced error messages with a controlled empirical study and find no significant effect.

via Enhancing syntax error messages appears ineffectual.