Learning to code is really learning to code something: One doesn’t just “learn programming” nor “learn tracing”

May 20, 2019 at 7:00 am 7 comments

I asked a group of social studies educators what programming language(s) they might want to use in their classes. One of the interesting themes in the responses was “the same as what’s in math and science classes.” One teacher said that she didn’t want a “weird hierarchy” where there’s one programming language in STEM and another in “history and English” for fear they’d be seen as “dumbed down.” Another said that maybe teaching JavaScript in history class “would make history cool.”

There’s a belief in this theme that I think is wrong. Learning to program in science class probably won’t transfer without a bunch of work to programming in mathematics class, and programming STEM classes will probably be a very different thing than programming in the humanities classes. Even expert programmers learn to program in a domain, and have a hard time transferring that knowledge of programming between domains. Expertise is expertise in a domain.

My advisor, Elliot Soloway, was involved in some of the early studies that supported this claim. The first paper was “The role of domain experience in software design” by Beth Adelson and Elliot Soloway from 1985. I quote from the abstract:

A designer’s expertise rests on the knowledge and skills which develop with experience in a domain. As a result, when a designer is designing an object in an unfamiliar domain he will not have the same knowledge and skills available to him as when he is designing an object in a familiar domain.

In this study, they took expert software designers in various fields, and have them design systems in other fields. They also asked novice designers to do some of the same tasks. For example, maybe we have a software designer who has been building banking software, and another who has been designing real-time control systems. Now, let’s ask both designers to design an elevator control system.

What they found was that the designers in the new domain struggled. They stopped planning (e.g., making notes). When they were in the familiar domain, they would often visualize the working system (“simulation” in the paper). Novices didn’t. The experts didn’t when they were faced with a new domain. Experts in an unfamiliar domain looked much like novices. Now, experts in an unfamiliar domain were better than the novices at noticing constraints on the design, so something transferred.

The second paper is even more striking. “Empirical Studies of Programming Knowledge” (1984) by Elliot Soloway and Kate Ehrlich. From the abstract:

We suggest that expert programmers have and use two types of programming knowledge: (1) programming plans, which are generic program fragments that represent stereotypic action sequences in programming, and (2) rules of programming discourse, which capture the conventions in programming and govern the composition of the plans into programs.

When we teach programming, we tend to focus on the syntax and semantics of the language. We don’t explicitly teach plans — chunks of code that do something useful. But we expect students to figure them out. We rarely teach discourse rules. The domain-specific knowledge lies in both plans and discourse rules.

To test the claim about the importance of these discourse rules, they produce sets of two programs: Alpha and Beta. Alpha is a perfectly fine program. Beta breaks the rules. For example, if you see a variable initialized n := 0;, you would find it weird to later see read(n); (to input a new value for n). It’s not wrong. The code might work just fine — in fact, it does work just fine in the experimental construction of Beta. But the program breaks the rules of discourse. They write:

Notice that both Alpha version and the Beta version are runnable programs that in almost all cases compute the same values. Moreover, to an untrained eye their differences may even not be apparent; they always only differ by a very few textual elements.

Here’s an example of one Alpha and Beta — these both work. In this case, they do not do the same thing:

Beta isn’t wrong. It successfully computes minimum. However, it uses the variable max which is confusing. It breaks our discourse rule. The program does work.

Different domains use different standards and different styles of programming. Engineers using MATLAB rarely use FOR or WHILE loops, for example. Graphic designers writing JavaScript code use far more exception handling than we ever expected.

Soloway and Ehrlich showed these programs to “experts” (undergraduate juniors to graduate students) and novices (students in their first programming course). When asked questions about Alpha (e.g., “What goes in this missing line in the code?” Or “Do you remember that code that I showed you?”), experts do far better than novices. When asked questions about Beta, experts do essentially the same as novices (no statistically significant differences).

I find it particularly notable that the expert drop is steeper.  Experts rely heavily on cues like variable names, even more than novices. CS expertise is really expertise in the discourse rules.

If expert programmers “knew programming,” they should be able to just trace the code (“be a computer”) and answer the questions correctly. Instead, they struggle to understand what’s going on. They’re pretty much like a student in their first semester of programming. The experts know Alpha well because it’s just like all the other programs they’ve ever seen — they can pattern match, rather than reason about the code itself. The experts struggle with Beta. It’s kind of like the difference between humans and Econs. Econs can reason through code rationally. Humans rely on expectations.

These results also suggest that the question of “Does tracing come before writing?” is moot.  Tracing what?  The program matters.  Some programs are harder to trace than others — for everyone, and particularly, with expertise.  There is no generic “tracing skill.”

Conclusion: People don’t just learn “coding.” Programmers in general know plans and discourse rules. Break the rules and you just have the programming language — and even experts aren’t really good at just applying the syntax and semantics rules. No better than a novice. If you have enough expertise in different domains, then you can work in different domains. When you start programming in a new domain, you’re not that much different than a new programmer.

The social studies teachers I’m working with have a sense that students can just
know JavaScript.” I don’t think that’s true. I think if I taught students to write JavaScript code to use Google’s Charts service for making data visualizations, it wouldn’t be much easier to teach them Web programming with React, to write scripts for Adobe Photoshop, nor to build simulations in Lively Web. It’s all JavaScript, and the syntax and semantics are the same in each — but in terms of what people really know and use (i.e., plans and discourse rules), it’s completely different.

Entry filed under: Uncategorized. Tags: , , .

Seeking Data: What’s happening at your school as you cap CS major enrollment? Why I say task-specific programming languages instead of domain-specific programming languages

7 Comments Add your own

  • 1. alfredtwo  |  May 20, 2019 at 8:13 am

    We old timers often joke that we can write a good FORTRAN program in any language. With programming languages I see a similar problem to what you are talking about with domains. Maybe its related? In any case when I learn a new programming language the struggle is not the syntax (I’ve learned a lot of languages) but trying to understand the idiom of the new language. What structures are natural to it and how are they different from what I am used to using.

  • 2. Raul Miller  |  May 20, 2019 at 11:51 am

    I am reminded of two things here:

    (1) Working in code bases where the naming had nothing to do with the concepts which I considered important, and

    (2) Writing code where I was at a loss for coming up with good names to use.

  • 3. Quinn Burke  |  May 20, 2019 at 1:53 pm

    Interesting post which reinforces the case for integrating CS into other disciplinary content, though in a more deliberate way. I am less worried however about social studies teachers’ misguided enthusiasm for programming but rather the fact that many K-12 schools still attempt to bring “coding” as a singular stand-alone course. No bueno. Yes, asking “coding what?” is a logical question. The various colleges in which CS exists as a post-secondary course of study reinforces what Mark articulates here.

    • 4. Mark Guzdial  |  May 21, 2019 at 9:45 am

      Hi Quinn,

      I don’t think of the teachers I’m working with as “misguided.” They have reasonable beliefs. Given that we call it a programming “language,” and language skills do transfer between disciplines, I get where they’re coming from.

      This is input to a design process. I see several possible responses:

      Show them these data and try to convince them that PL skills will likely be domain-specific. That’s unlikely to work.
      Develop task-specific PLs so that each language is easy and useful enough that teachers are less worried about transfer. Maybe that’ll work.
      Explain to teachers using PL in domain/task-specific ways which parts will likely transfer, and how to teach for that transfer. Another possible strategy.

  • 5. Josh Sheldon  |  May 20, 2019 at 3:46 pm

    This makes a lot of sense – I’ve definitely noticed similar patterns, anecdotally. It raised a couple of questions for me:
    1) Is there another level of “expertise” w.r.t. programming/computing beyond what was investigated in the 2nd Soloway paper? Certainly some grad students are quite good, but I’d argue there are levels far above that. Though they are rare, I’ve had the good fortune to work with a few individuals that do seem to be able to work effectively across domains, or at least it looks that way. They are able to read almost any code and make sense of it, and they make code contributions in areas that they have little or no experience in. However, each of these folks have worked across a good number of other domains in the past – they have been novices multiple times before, and my hunch is that they have learned techniques that can be applied to learning and problem solving in other new domains.

    2) I also wonder what instructions the programmers in the 2nd study were given. Could the investigators given different instructions that would have encouraged the “expert” programmers to behave more like Econs? Again, this is simply a hunch, but is it possible that the expert programmers were using heuristics for reading code that have served them well to that point, but failed them in this case? If they were given explicit instructions to read “assuming first principles,” or warned that the code was non-standard, would they have fared better?

    3) It would be interesting to do an experiment that’s an adaptation of the 2nd paper: recruit expert coders who work in one domain, and then ask them to answer questions about code written for another domain. One could do a couple iterations of this – looking at domains closer to the original and farther…

    • 6. Mark Guzdial  |  May 21, 2019 at 9:41 am

      Hi Josh,

      1) I’ll bet that you’re right, that there are people with a great deal of expertise who can carry computing between domains, because they’re practiced in computing in multiple domains. We might want that for our students, but the cost in time to develop such expertise is enormous. Can we agree that we’re unlikely to see elementary and secondary school students with that level of expertise? They’re even unlikely to develop the expertise seen in Elliot’s study.

      2) In Daniel Kahneman’s terrific “Thinking Fast and Slow,” he says that he still demonstrates cognitive biases — even biases that he discovered and named, even when he knows he’s making a biased decision. I don’t think we can successfully instruct Humans to be Econs.

      3) Agreed that this is a great thread of research that I wish there was more replication and exploration of.

      • 7. Josh Sheldon  |  May 21, 2019 at 10:46 am

        Oh, I was definitely not suggesting this ability to work across domains semi-fluently was something to aim for with students, of any sort, in any reasonable time frame. I’ve probably met 5 people like this, that I know of, in a place where you might expect to find a density of them.

        I’ll check out that article. Certainly humans will maintain some biases, and would never be “full econs.” I don’t know if they could be a closer approximation – would be an interesting exploration (for someone who isn’t me).


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,186 other subscribers


Recent Posts

Blog Stats

  • 2,060,647 hits
May 2019

CS Teaching Tips

%d bloggers like this: