Creating (and improving) options for CS practice: Practice-It! and beyond

One of the (several!) pedagogical methods that I learned about at the DCCE meeting a couple weeks ago was Practice-It!, a new (to me) website from the University of Washington.  Practice-It! provides a variety of practice activities for students, from multiple choice questions, to predict-the-output problems, to exercises where students write a single method to solve a problem.  These help to fill the huge gap between reading the book and attending lecture on one side, and facing a full IDE (“a speeding compiler”) on the other side.  It joins pedagogical tools like CodingBat and Problets in an important, but surprisingly sparse area of tools for computing students.

I really like these tools and think that they fill an important role.  However, given that there is more than one tool in this space now, I have a criticism of all the existing tools, that I mean to be constructive.  Currently, the coding problems in these tools invoke a compiler or interpreter and return the error generated.  We can do better, and need to because the errors messages of virtually any interpreter or compiler presume a knowledgeable, professional programmer.  They are unclear, often useless, and always infuriating for a novice.

Here are a couple examples.  In CodingBat, I tried the Python problem where I have to write a function to determine if I can sleep in, depending on whether it’s a weekday or during vacation.  The inputs are booleans, but I tried (like many students) to write the function without reading the description of the inputs. I made assumptions about the inputs being objects and collections.  I compounded the error by writing the code “weekday is in vacation” as opposed to the correct “weekday in vacation.”  The error message isn’t useful.  My semantic error (of ignoring that the inputs are booleans) is hard (but doable!) to catch and address.  My syntax error is not helped by this message.

Practice-It! works similarly.  Here, I’m asked to write a Java method, and (again, as a student might) I decide to just get the basic method declaration in first — but get it wrong because I forget to deal with input parameters.

The error messages from the Java compiler are worse than useless.  They mention things like “enums” that I (as a student working on such a simple problem) have never seen.  To the credit of Practice-It!, they are collecting these awful error messages and trying to generate student “friendly” versions.

I wonder, though, if we can do even better than rewriting Java’s error messages. In each of these situations, we have a lot of knowledge about what code the student should be writing, what errors we might expect, and what the student knows already.  We should be able to tune the error messages to the problem.

Here’s the radical idea I’ve been exploring:  What about building our own parsers?  A parser for any of these problems does not have to be a parser for all of Java or Python. In fact, it shouldn’t be.  We know from lots of research (e.g., Lewis Johnson’s PROUST and Anderson’s Cognitive Tutors) that student answers to coding problems mostly fall in a small range of options, and those with radically different answers are far more likely to be radically wrong than brilliant-and-different-thinking — and letting beginning students flail with the all the flexibility of the full language is simply wasted time.  Only let the students type in a subset of the language, but provide understandable, informed error messages for that subset that are tuned to the problem.

As computer scientists, we might blanch at the complexity of writing parsers, remembering hours spent battling Lex and YACC.  Lex and YACC were written before 1975. What’s the possibility that we could do better in 35 years of development and Moore’s Law increases? I’ve been exploring OMeta lately for just this purpose — it makes it possible to build parsers for rich languages in surprisingly few lines of code.

The constructive recommendation that I have for computing educators building (or like me, considering to build) tools to fill the book-to-IDE gap: Be computer scientists, who build their own programming languages tools and can use these tools for improving education.  We don’t need to use a compiler as a monolithic piece of software.  We know the techniques used in building those compilers, and we can mix-and-match the techniques and components in our tools to help our students learn.