How do we make programming languages more usable and learnable?
April 1, 2014 at 9:43 am 25 comments
Amy Ko made a fascinating claim recently, “Programming languages are the least usable, but most powerful human-computer interfaces ever invented” which she explained in a blog post. It’s a great argument, and I followed it up with a Blog@CACM post, “Programming languages are the most powerful, and least usable and learnable user interfaces.”
How would we make them better? I suggest at the end of the Blog@CACM post that the answer is to follow the HCI dictum, “Know thy users, for they are not you.”
We make programming languages today driven by theory — we aim to provide access to Turing/Von Neumann machines with a notation that has various features, e.g., type safety, security, provability, and so on. Usability is one of the goals, but typically, in a theoretical sense. Quorum is the only programming language that I know of that tested usability as part of the design process.
But what if we took Amy Ko’s argument seriously? What if we designed programming languages like we defined good user interfaces — working with specific users on their tasks? Value would become more obvious. It would be more easily adopted by a community. The languages might not be anything that the existing software development community even likes — I’ve noted before that the LiveCoders seem to really like Lisp-like languages, and as we all know, Lisp is dead.
What would our design process be? How much more usable and learnable could our programming languages become? How much easier would computing education be if the languages were more usable and learnable? I’d love it if programming language designers could put me out of a job.
Entry filed under: Uncategorized. Tags: computing education research, HCI, Lisp, live coding, programming languages.
Computing Ed Research - Guzdial's Take 
1.
nickfalkner | April 1, 2014 at 11:18 am
I would be fascinated to see if such new languages could evolve as human languages normally do to allow numerous forms of expression that convey far more meaning, even where there is the risk of ambiguity.
I realise it’s impractical but there is a part of me that quite likes the idea of a program that compiles differently because of who wrote it and their grasp of idiom. The first and most useful aspect of this morass of difficulty grammar would be the removal of features that the writer didn’t need – hence replacing visual idioms for people who have no sight – or even potentially changing definition depending on the maturity of the writer.
The human communication languages we build are vast, byzantine, rich and compelling. Sometimes this leads to confusion. Sometimes this is deliberate. I suspect that this might be the path that growing programming languages might take once much smarter people than I sort out some minor issues with the Chomsky hierarchy. 🙂
Regards,
Nick.
2.
Doug Blank | April 1, 2014 at 11:20 am
I think that the idea that “Quorum is the only programming language … that tested usability as part of the design process” undervalues the manner that many languages are designed these days. Take, for example, Python. Every change, syntactic and semantic, is discussed in the open, and commented on by a fairly large group of people. And, the language evolves over time to address problems, including those of the API.
I wish there were more educators (and a more diverse group overall) that participated in the development discussions of these languages, but I think Python has turned out pretty well. (These are sometimes rough conversations for anyone to participate in—the aggressive, male-dominated, free-for-all culture can often make these discussions be arguments. I’ve sometimes succeeded and sometimes failed in affecting educational-based changes in language and library APIs.)
More importantly, languages are used by a diverse group of people. The language that may be most suitable for beginners may not be the same language suitable for experts. DrScheme/Rachet has configurations that you can indicate the language “level”, and even the language, the user wishes to use (A “level” might turn on/off different syntactic options). Calico allows one to switch languages while keeping the language API static.
One of the scientific contributions of Lisp/Scheme is that it drives home the point that “languages are data.” Scheme is not dead, and I am actively working to bring it back to being as useful as Python. To make it in today’s world though, a language is more about the “batteries included” (eg, libraries) rather than a particular syntax.
Human languages have evolved to be a wonderfully expressive API. Perhaps that is why we need grammar and English teachers. If programming language designers succeed, they may give us permanent job security.
3.
Mark Guzdial | April 1, 2014 at 12:43 pm
“Discussed in the open, and commented on by a fairly large group of people” is not the same as (a) empirical evidence of usability or (b) a design process akin to an HCI design process. Absolutely, a well-designed usable and learnable language would be unlikely to be used by a diverse group of people. It would be targeted for a set of users and their uses. What kind of a language design process do we need to create a targeted language? That’s the question I’m asking.
4.
shriramkrishnamurthi | April 1, 2014 at 11:24 am
On it: http://blog.brownplt.org/2014/04/01/var-vs-yar.html
5.
Mark Guzdial | April 1, 2014 at 12:43 pm
Very cute!
6.
gflint | April 1, 2014 at 1:39 pm
That looks fun.
7.
Steven Pemberton | April 2, 2014 at 6:16 am
Might I point out that Python is descended from ABC [1], which was designed (in the 80s) using classical usability techniques.
[1] ABC: A Simple but Powerful Interactive Programming Language and Environment. We did requirements and task analysis, iterative design, and user testing. You’d almost think programming languages were an interface between people and computers. Now famous because Python was strongly influenced by it. http://www.cwi.nl/~steven/abc/
8.
Mark Guzdial | April 2, 2014 at 9:15 am
That’s interesting and good to know, Steven. Can you say something about why a similar process wasn’t used with Python? Was it considered too expensive? Not useful enough?
9.
Greg Wilson | April 4, 2014 at 5:54 pm
Attempts were made (http://mail.python.org/pipermail/python-dev/2000-July/006427.html, http://mail.python.org/pipermail/python-dev/2000-July/006098.html); response was lukewarm.
10.
shriramkrishnamurthi | April 4, 2014 at 7:31 pm
Nobody seems to have pointed out the Natural Programming project, so I will: http://www.cs.cmu.edu/~NatProg/
11.
John Maloney | April 5, 2014 at 6:38 am
I’m not sure if you’d consider Scratch (scratch.mit.edu) a programming language for the purpose of this discussion. Scratch was heavily user tested with young people (the target audience) for several years before it was released, and that experience drove many of the key design decisions. During part of that testing, the entire Scratch team went to a local after-school computer center and worked one-on-with with youth using with the current version of Scratch. As the software lead on the project, I can attest that nothing is more compelling (and humbling) than watching a user struggling with software that you *thought* was great.
All that time with users paid off; the Scratch website has close to three million users who have shared over five million Scratch projects.
It is not necessary to collect and analyze data; just spending time with users is sufficient to find the problem areas and to motivate the team to fix them.
12.
shriramkrishnamurthi | April 5, 2014 at 9:23 am
John, I don’t count that as the same sort of thing. DrRacket also evolved over twenty years of similar user testing: hours and hours in labs, watching what people do, etc. We had the same humbling feeling as we did this. And based on all this, as the system evolved, we took especially great pride in our error messages.
Yet, when we sat down to do rigorous studies of those very error messages, the outcomes were dismal (some having nothing to do with the messages themselves but their educational context, but that was part of what we had missed). Eg:
http://cs.brown.edu/~sk/Publications/Papers/Published/mfk-measur-effect-error-msg-novice-sigcse/
Measuring the Effectiveness of Error Messages Designed for Novice Programmers
http://cs.brown.edu/~sk/Publications/Papers/Published/mfk-mind-lang-novice-inter-error-msg/
Mind Your Language: On Novices’ Interactions with Error Messages
and the story on this is far from over.
It was, for me, an even more humbling, and terrifying, lesson in the effectiveness of various UI evaluation techniques. Which isn’t to say Scratch isn’t a wonderful system, but so is DrRacket, and I’ve come to believe that there are layers and layers of awfulness under all of them, each just waiting for the right study to uncover. <-;
13.
Mark Guzdial | April 5, 2014 at 2:36 pm
I agree with Shriram that there are “layers and layers of awfulness under all of them” (which is a great tweet all by itself :-), but I do see Scratch and its evaluation as an example of making a programming language usable for a particular set of uses and users. Racket is used by lots of people for lots of different purposes, which makes the usability challenge enormous. Scratch won’t be used by Brian Dorn’s graphic designers or Michael Horn’s astrophysicists (most likely). It’s certainly a programming language, John, but it has been tuned for a target audience, which makes it easier to make usable.
14.
shriramkrishnamurthi | April 5, 2014 at 4:58 pm
True, Racket-the-language has to cater to everyone from middle-schoolers to professionals. Racket gets over 100,000 downloads every year, some of which are site downloads, and this has been going on for over a decade, so we’re in a similar ballpark in numbers, but much more broadly spread, than Scratch.
However, the _student sub-languages (dialects)_, and their support in DrRacket, are very specifically designed for intro college students, which is indeed the audiences we observed. (In contrast to widespread observation of professionals, which we have not done.)
Therefore, I think the two are quite comparable in that regard (just a difference of age/maturity/etc.). Which is why I think what we learned with detailed studies may be a harbinger of what others will find, too. [Just so it’s clear, I too absolutely do consider Scratch a “language” in this context.]
As an aside, today I had a wonderfully frustrating experience trying to program a robot with the Lego Mindstorms language, which is block-based and yet at the opposite end of the usability spectrum from languages like Scratch. (In fact, one of my faculty colleagues and I struggled with this, at a robot “block party”, so count two computer scientists down for the count.) Which just goes to show how critical the fine-grained details are (and how hard it was to do what Scratch did).
15.
gasstationwithoutpumps | April 6, 2014 at 12:47 am
I was surprised to see that someone who cared about user interface had their paper download as “paper.pdf” rather than something mnemonic and unique (like “DrRacket-error-analysis.pdf”).
16.
John Maloney | April 6, 2014 at 6:53 am
Thanks for the links about error messages; very interesting.
Scratch avoids error messages, not because we didn’t think they could be useful but because we worried that some of the audience we hoped to reach would find them discouraging, or might misinterpret them as an indication that they couldn’t become programmers. For Scratch, which is often used in informal educational settings and by self-guided learners at home, I think that makes sense. For DrRacket, having informative error messages make sense, but it’s sobering that even with a great deal thought and careful testing, it is *still* challenging to create useful error messages.
You’re certainly right that user testing only identifies *some* usability problems. But every problem we find and fix makes life better for our users. One of the things that I find so rewarding about this field is you can always learn more from your users and use those insights to improve your programming language and environment.
17.
Mark Guzdial | April 6, 2014 at 1:46 pm
The focus on iterative development and usability testing is great, but I read Andy’s blog post as also pointing to design. Scratch had a group of excellent designers involved (John, Mitchel, Yasmin, Brian, etc.), in part because it would be hard to involve the children in designing their own language. (Though one might design a programming language for children using a process like Allison Druin’s, where the children are co-designers.) It’s easier to imagine involving adult professionals in a participatory design process. How would physicists design a programming language for themselves? Graphic designers? Designers of pharmaceuticals? I’ve seen the programming languages that GT architects and industrial designers use — I wonder what might be created if those languages were co-designed with computer scientists and architects?
18.
Saad Farooq | April 6, 2014 at 9:08 pm
Does usability testing and iterative design make the language more ‘learnable’ though ?
I guess it depends on what you are designing and testing for. In human interface design, this goal is almost always user’s intuition. In some cases, this is natural intuition, for example the human tendency to interact via touch but in other cases, it’s ‘learned’ intuition. For example, being a long time Windows and then Linux user, I find small things on a Mac really annoying (swiping up versus down to scroll a page for instance). It’s the reverse for long time Mac users. An appropriate design for me would probably take my Windows roots into consideration or at least provide me a clean transfer path.
When it comes to programming I am not aware of any claims of natural intuition thus far (maybe John Pane’s work??) but I do find that programming often goes against my ‘learned’ intuitions. Up until the time I was first introduced to programming (this would be grade 9 for me), everything flowed from one stage to another and seemed to build up on the previous. Arithmetic progressed into algebra into calculus (it didn’t all make sense in terms of forward compatibility but backward compatibility wasn’t bad). Science classes used mathematical representations. All was well.
But then programming came along and it did everything differently. After struggling to make sense of the unknown ‘x’ variables in algebra, there was now a new concept of variables. Even after having used functions forever, programming functions were something totally unrelated. I wasn’t able to make a lot of those connections for a long time (some possibly I still haven’t).
Sherin,2001 showed that there are parallels between algebraic and computational notations for solving physics problems but on the other hand, there is some evidence of epistemic tensions between the scientific way of thinking and the computational way of thinking (Cabellaro work).
The question is, do we accept that programming is a unique way of thinking and try to teach that to everyone so those tensions never arise ? or do we evolve programming so that it is a medium of expression (not just a human computer interface) that is more integrate with other forms of expression ? Or is there a feasible middle route ?
19.
shriramkrishnamurthi | April 6, 2014 at 9:42 pm
Saad, it’s unfortunate that you had a computer science curriculum that decided to do violence, without so much as a by-your-leave, to your hard-won math intuitions (e.g., about what a “variable” or “function” is). Program by Design, and now Bootstrap, have been trying to fight this battle for ages. We believe this is an important _transition_ that should be _managed_, not a minor difference into which a learner should be dropped. (But I believe I’m very much of the minority here.)
20.
Mark Guzdial | April 6, 2014 at 10:29 pm
Hi Saad — completely agreed. Usability does not imply learnability, but usability is a pre-requisite for learnability. If things break or disappear in ways that you can’t figure out, then it’s not usable. I agree that we need a more nuanced notion of “usability” than simply “intuitive.” Andy diSessa in “Changing Minds” makes an excellent case about literacy and usability for a literate audience. (Wikipedia isn’t very “usable” to illiterate users.)
In the Blog@CACM piece, I argue for two senses of learnability: Value (you have to be able to see why it’s worth learning) and social relevance (this will help me move into my community of practice). We can add others, like low extrinsic cognitive load. I suspect that a participatory design process with the actual community of practice would help with both value and social relevance, because taking the practitioners seriously would mean that you make visible and evident the elements of the computing that connect to the practice — which is one of the heuristics that Andy Ko mentions in his blog post. Usability and learnability are connected in interesting ways.
BTW, yes, John Pane clearly explored intuitive notions of computing, as did L.A. Miller in the 80’s, and the Commonsense Computing group in the last decade.
21.
shriramkrishnamurthi | April 6, 2014 at 9:39 pm
John,
First I must congratulate you on navigating a little maze of twisty little papers, all named paper.pdf (-:
Now, to the point:
I dragged a move block into place, and then dragged in a division block. I put 100 in the numerator, 2 in the denominator, clicked on the move block, and the cat moved 50 paces.
Then I changed the denominator to 0, and the cat _disappeared_.
Now, even when I change the denominator back to 2 (or make it 1), there’s no cat sprite. Draggling the cat from the sprite library into the playback window (sorry, I don’t know Scratch-speak) doesn’t give me a cat: the window remains blank.
Simulating a simple-minded user (which comes easy to me), all I can see is:
– I had a working program
– I changed something (sometimes some subtle calculation)
– the system didn’t say anything
– my cat disappeared
I’m curious why this was considered less “discouraging” than an indication of _what happened_. In particular, in the very subtle interplay between statics (code) and dynamics (behavior), you decided it’s best to act maximally mysterious. I’m not arguing that being maximally _informative_ is better (it’s not), but this seems extreme.
I assume you have user studies that show at least that this particular behavior on the part of the system is a wise one?
22. Interesting Links 7 April 2014 | Dot Net RSS | May 11, 2014 at 11:48 am
[…] How do we make programming languages more usable and learnable? A lot of discussion in the comments as well. Another great post by Mark Guzdial. […]
23.
Those Who Say Code Does Not Matter are Wrong | Computing Education Blog | May 16, 2014 at 9:09 am
[…] Meyer is making a similar point to Andy Ko’s argument about programming languages. Programming does matter, and the language we use also matters. Meyer’s goes on to […]
24.
Guest Post from Shriram Krishnamurthi: Growing respect for Research around Computational Learning and Thinking | Computing Education Blog | June 12, 2014 at 8:37 am
[…] capable of holding my own in the open competition of conferences you respect”. People like Andy Ko—who published his work in conferences like ICSE—or my colleague Jeff Huang—an HCI person with […]
25.
Guest Post from Shriram Krishnamurthi: Growing respect for Research around Computational Learning and Thinking | Computing Education Research Blog | September 24, 2019 at 9:40 am
[…] capable of holding my own in the open competition of conferences you respect”. People like Amy Ko—who published her work in conferences like ICSE—or my colleague Jeff Huang—an HCI person with […]