Posts tagged ‘computing education research’

Where a Notional Machine Doesn’t Help: JavaScript and the DOM

At the Dagstuhl Seminar on Notional Machines and Programming Language Semantics, I came to a new understanding about where notional machines are not useful and another kind of support is needed.

I joined a breakout group on “Notional Machines for everything else.” There had been discussions about notional machines for popular programming languages for Scratch and Python, and one breakout group was formed around that goal. This group was about exploring where else notional machines could be useful, like trying to understand machine learning, generating proofs, and JavaScript for Web development. Joe Politz was our group leader.

After the first round of discussion, we all decided to focus on JavaScript. We had some serious experts on JavaScript and the DOM in our group, like Shriram Krishnamurthi and Titus Barik. The discussion was amazing! I was learning so much, and I took pages and pages of notes. Everyone noticed my feverish note-taking, so I got elected to report back to the group. That’s this blog post — it’s the work of the whole group (not just me). I just happened to be the guy who made the slides.

We had already come to the realization that there isn’t just one notional machine per language. A notional machine is about helping students understand a computational process, and there can be lots of processes in a given language. So we picked a specific scenario that we were aiming to explain: You (as a student) want to turn the background yellow when the button is clicked.

But the student makes a mistake.

And I came to realize that, without a lot of support:

The problem is that window.bg = "yellow" isn’t wrong. Because there isn’t a previously defined bg property, this assignment simply creates a property bg for the window object. No error. It just doesn’t do what the students wanted. How does the student figure out that the desired property is backgroundColor? Get a list of all bindings on window? There are hundreds or even thousands of them. How do you find backgroundColor among all of those?

The breakout group started listing on the blackboard the things we might need to explain to students to help them understand when went wrong or what might go wrong with clicking on a button to turn the background yellow. It was a long list.

You probably can’t read all of those, so I’ll list a few of them here:

  • What happens if you have two DOM elements with the same ID?
  • Where did window as an object get defined at all?
  • If you have event handlers defined on an object, and you delete the object, what happens to the event handler?
  • What happens if objects higher up in the DOM modify the event triggered by the event click?
  • Something I heard students ask: If my JavaScript changed the DOM, how come my HTML file didn’t change?
  • And Many More

I really had no idea just how complicated JavaScript and the DOM were! Amy Ko looked up the JavaScript definition of what == means (see link here). This isn’t the formal semantics. This is meant to be understandable. It’s insanely complicated.

At this point, Ben Shapiro raised a really interesting side question: What’s the cost of JavaScript’s overly-complicated rules? Is there a way of measuring the lost productivity of bad programming language design?

So, what’s the answer?

I realized that the answer is not a notional machine. The problem is that long list Amy found for us. I can teach part of that list as a notional machine, but I can’t teach all of it with a simplified model. Any simplification I create would be insufficient for the complexity of the reality. And even having a notional machine wouldn’t help if a student typed window.bg = "yellow". The student needs IDE’s and other supports to figure out errors that never trigger an error.

The solution is to reduce complexity to make it teachable.  In an earlier talk at Dagstuhl, Ben Shapiro explained how Shriram and Joe and their collaborators did this with Pyret.  In Pyret, they explicitly disallow some things that Python allows but are way too complicated to explain.

Probably the best idea for JavaScript, following Racket’s lead, is to have language levels. We should teach students a strict subset of JavaScript, where the really complicated things are simply disallowed. The goal is to help students to learn a real subset, then grow the subset.  TypeScript offers an alternative model, because it offers a more sane way of doing JavaScript.  For example, TypeScript’s type checking might help figure out the window.bg bug. There are lots of other languages that are more reasonable and compile to JavaScript — but those are avoiding JavaScript entirely.

The very best idea would be to fix JavaScript and the DOM, but it’s probably too late for that.

This working group was useful to me for two reasons. First, I really do have to teach JavaScript and the DOM (again) in January 2020, and now I have a new sense of the challenges and my options. Second, this was a great example of where a notional machine is not the answer to a pedagogical problem.

Thanks to members of the group who reviewed an earlier draft of this summary.  They’re not responsible for where I still didn’t get the details right.

 

September 9, 2019 at 7:55 am 13 comments

Holding ourselves to a higher standard: “Language-independent” just doesn’t cut it

My CACM blog post this month (see link here) is a retraction of the term “language-independent” in our work on the FCS1 and SCS1:

There is no language independence here. The FCS1 and SCS1 are multi-lingual which is a remarkable achievement. We might also call them pseudcode-based assessments, which is how they can be multi-lingual, but since a pseudocode-based test isn’t necessarily validated across other languages, “multi-lingual” is a stronger claim than “pseudocode-based.” We do not cover all of any of those languages (Java, MATLAB, or Python), but we do cover the subset most often appearing in an introductory CS course.

They are clearly not language independent. In the great design space of programming languages, Java, MATLAB, and Python cluster together pretty closely. There are much more different programming languages than these — I’m sure it’ll take any reader here just a few moments to generate a half-dozen candidates whose learners would score poorly on the FCS1 and SCS1, from Scratch to Haskell to Prolog.

I only vaguely remember the discussion about using the term “language independence” with Allison many years ago.  I remember her asking me if we should worry about the (relatively few) classes that used languages other than Python, MATLAB, and Java.  I think I told her she needed to graduate. I judged from the perspective of what was being published at the SIGCSE Symposium — Python, MATLAB, and Java was “language independent” enough for the paper to be seen as valuable to SIGCSE reviewers. I don’t remember the details, but I’ll accept the blame for the decision to call FCS1 (and SCS1 later) language independent.

That was a long time ago, before the International Computing Education Research Conference (ICER) was invented.  Since then, we have a computing education research community that aims to answer questions about how people learn computing — period. We’re not just about undergraduate introductory computer science classes. Even at the undergraduate level, we should study the classes (no matter how few) doing something different to see what’s powerful and interesting about them. We should explicitly be exploring unusual (even purpose-invented) languages to understand more of the interaction between programming languages and human cognition.  An insightful PPIG paper from Clayton Lewis (see link here) was recently circulated on Twitter (see tweet) that makes great points about the complexity of measuring that interaction:

The PPIG community should be proud that cognitive dimensions analysis emerged from the work of people in its ranks, Thomas Green, Marian Petre, Alan Blackwell, and others. We should be skeptical of calls to replace its use with A-B trials or other quantitative methods that cannot cope with the complexity of the language design landscape. When results of A-B trials and similar studies are presented, we should diplomatically ask for the mechanisms that are involved to be described. Colleagues who present the results of such trials should be prepared to respond to this request, so that the generalizability of their results can be assessed.

We should not be driven by what’s in classrooms today (see previous post making that argument). We should hold ourselves to a higher standard. Our goal is to create a lasting record of exploration and research for a research community.

That’s why it’s past time for this retraction.

 

August 26, 2019 at 7:00 am Leave a comment

Summarizing findings about block-based programming in computing education

As readers of my blog know, I’m interested in alternative modalities and representations for programming. I’m an avid follower of David Weintrop’s work, especially the work comparing blocks and text for programming (e.g., as discussed in this blog post).

David wrote a piece for CACM summarizing some of his studies on block-based programming in computing education. It has just been published in the August issue.  Here’s the link to the piece — I recommend it.

To understand how learners make sense of the block-based modality and understand the scaffolds that novice programmers find useful, I conducted a series of studies in high-school computer science classrooms. As part of this work, I observed novices writing programs in block-based tools and interviewed them about the experience. Through these interviews and a series of surveys, a picture emerged of what the learners themselves identified as being useful about the block-based approach to programming. Students cited features discussed here such as the shape and visual layout of blocks, the ability to browse available commands, and the ease of the drag-and-drop composition interaction. They also cited the language of the blocks themselves, with one student saying “Java is not in English it’s in Java language, and the blocks are in English, it’s easier to understand.” I also surveyed students after working in both block-based and text-based programming environment and they overwhelmingly reported block-based tools as being easier. These findings show that students themselves see block-based tools as useful and shed light as to why this is the case.

August 19, 2019 at 7:00 am Leave a comment

Social studies teachers programming, when high schools choose to teach CS, and new models of cognition and intelligence in programming: An ICER 2019 Preview

My group will be presenting two posters at ICER this year.

  • Bahare Naimipour (Engineering Education Research PhD student at U-Michigan) will be presenting our participatory design session with social studies educators, Helping Social Studies Teachers to Design Learning Experiences Around Data–Participatory design for new teacher-centric programming languages. We had 18 history and economics teachers building data visualizations in either Vega-Lite or JavaScript with Google Charts. Everyone got the starter visualization running and made changes that they wanted in less than 20 minutes. Those who started in Vega-Lite also tried out the JavaScript code, but only about 1/4 of the JS groups moved to Vega-Lite successfully.
  • Miranda Parker (Human-Centered Computing PhD student at Georgia Tech) will be presenting her quantitative model explaining about half of the variance in whether Georgia high schools taught CS in 2016, A Statewide Quantitative Analysis of Computer Science: What Predicts CS in Georgia Public High School. The most important factor was whether the school taught CS the year before, suggesting that overcoming inertia is a big deal — it’s easier to sustain a CS program than start one. She may talk a little about her new qualitative work, where she’s studying four schools as case studies about their factors in choosing to teach CS, or not.

Barbara is co-author on a paper, A Spaced, Interleaved Retrieval Practice Tool that is Motivating and Effective, with Iman Yeckehzaare and Paul Resnick . This is about a spaced practice tool that 32% of the students in an introductory programming course used more than they needed to, and the number of hours of use had a measurable positive effect on the final exam grade.

All of our other papers were rejected this year, but we’re in good company — the accept rate was around 18%. But I do want to talk about a set of papers that will be presented by others at ICER 2019. These are papers that I heard about, then I asked the authors for copies. I’m excited about all three of them.

How Do Students Talk About Intelligence? An Investigation of Motivation, Self-efficacy, and Mindsets in Computer Science by Jamie Gorson and Eleanor O’Rourke (see released version of the paper here)

One of the persistent questions in computing education research is why growth mindset interventions are not always effective (see blog post here). We get hard-to-interpret results. I met Jamie and Nell at the Northwestern Symposium on Computer Science and the Learning Sciences in April (amazing event, see here for more details). Nell worked with Carol Dweck during her graduate studies.

Jamie and Nell found mixed mindsets among the CS students that they studied. Some of the students they studied had growth mindsets about intelligence, but their talk about programming practices showed more fixed mindset characteristics. Other students self-identified as having some of both growth and fixed mindset beliefs.

In particular, some students talked about intelligence in CS in ways that are unproductive when it came to the practice of programming. For example, some students talked about the best programmers as being able to write the whole code in one sitting, or never getting any errors. A more growth mindset approach to programming would be evidenced by talking about building programs in pieces, expecting errors, and improving through effort over time.

This is a really helpful finding. It gives us new hypotheses to explore about why growth mindset interventions haven’t been as successful in CS as in other disciplines. Few disciplines have this strong distinction between their knowledge and their practice as acutely as we do in CS. It’s no wonder that we see these mixed mindsets.

Toward Context-Dependent Models of Productive Knowledge in Programming Cognition, by Brian A. Danielak

I’ve known Brian since he was a PhD student, and have been hoping that he’d start to publish some of his dissertation work. I got to read one chapter of it, and found it amazingly insightful. Brian explained how what we might see as a “random walk” of syntax was actually purposeful and rational behavior. I was excited to hear about this paper, and I enjoyed reading it.

It’s such an unusual paper for ICER! It’s empirical, but has no methods section. A big part of it is connecting to prior literature, but it’s not about a formal literature review.

Brian is making an argument about how we characterize knowledge and student success in CS. He points out that we often talk about students being wrong and having misconceptions, which is less productive than figuring out what they understand and where their alternative conceptions work or fail. I see his work following on to the work of Rich et al. (mentioned in this blog post) on CS learning trajectories. There are so many things to learn in CS, and sometimes, just getting started on the trajectory is a big step.

Spatial Encoding Strategy Theory: The Relationship between Spatial Skill and STEM Achievement by Lauren Margulieux.

Lauren is doing some impressive theoretical work here. She’s considering the work exploring the relationship between spatial reasoning and CS learning/performance, then constructs a theory explaining the observed results. Since it’s Lauren, the theory is thorough and covers well the known results in this space. I wrote her that I didn’t think that theory explains things that we expect are related to spatial reasoning, but we don’t yet have empirical evidence to support it. For example, when programmers simulate a program in their mind, their mental models may have a spatial component to them, but I don’t know of empirical work that explores that dimension of CS performance. But again, since it’s Lauren, I wouldn’t be surprised if her presentation addresses this point, beyond what was in the paper. (Also, read Lauren’s own summary of the paper here.)

I am looking forward to the discussion of these papers at ICER!

August 12, 2019 at 7:00 am Leave a comment

Let’s think more broadly about computing education research: Questions about alternative futures

At the Dagstuhl Seminar in July, we spent the last morning going broad.  We posed three questions for the participants.

Imagine that Gates funds a CS teacher in every secondary school in the world, but requires all new languages to be taught (not Java, not Python, not R, not even Racket). “They’re all cultural colonialism! We have to start over!” Says Bill. We have five years to get ready for this. What should we do?

Imagine that Oracle has been found guilty of some heinous crime, that they stole some critical part of the JVM, whatever. The company goes bankrupt, and installation of Java on publicly-owned computers is outlawed in most countries. How do we recover CS Ed?

Five years from now, we’ll discover that Google has secretly been moving all of their infrastructure to Racket, Microsoft to Scala, and Amazon to Haskell (or swap those around). The CS Ed world is shocked — they have been preparing students for the wrong languages for these plum jobs! What do we do now? How do you redesign undergrad ed when it’s not about C++/#/Java/Python?

We got some pushback.  “That’s ridiculous. That’s not at all possible.” (I found amusing the description of we organizers as “Willy Wonka.”) Or, “Our goal should be to produce good programmers for industry — PERIOD!”

Those are reasonable positions, but they should be explicitly selected positions. The point of these questions is to consider our preconceptions, values, and goals. All computing education researchers (strike that: all researchers) should be thinking about alternative futures. What are we trying to change and why? In the end, our goal is to have impact. We have to think about what we are trying to preserve (and it’s okay for “producing industry programmers” to be a preserved goal) and what we are trying to change.

August 5, 2019 at 7:00 am 13 comments

So what’s a notional machine anyway? A guest blog post from Ben Shapiro

Last week, we had a Dagstuhl Seminar about the concept of notional machines, as I mentioned in an earlier blog post about the work of Ben Shapiro and his student Abbie Zimmermann-Niefield. There is an amazing amount being written about the seminar already (see the Twitter stream here), with a detailed description from Amy Ko here in her blog and several posts from Felienne on her blog. I have written my own summary statement on the CACM Blog (see post here). It seems appropriate to let Ben have the summary word here, since I started the seminar with a reference to his work.

I’m heading back to Boulder from a Dagstuhl seminar on Notional Machines and Programming Language Semantics in Education. The natural question to ask is: what is a notional machine?

I don’t think we converged on an answer, but here’s my take: A notional machine is an explanation of the rules of a programmable system. The rules account for what makes a program a valid one and how a system will execute it.

Why this definition? Well, for one, it’s consistent with how du Boulay, coiner of the term notional machine, defined it at the workshop (“the best lie that explains what the computer does”). Two, it has discriminant utility (i.e. precision): the definition allows us to say that some things are notional machines and some are not. Three, it is consistent with a reasonable definition of formal semantics, and thus lets us imagine a continuum of notional machines that include descriptions of formal semantics, but also descriptions that are too imprecise — too informal — to be formal semantics but that still have explanatory value.

The first affordance is desirable because it allows us to avoid a breaking change in nomenclature. It would be good if people reading research papers about notional machines (see Juha Sorva’s nice review), including work on how people understand them, how teachers generate or select them, etc., don’t need to wrestle with what contemporary uses of the term mean in comparison to how du Boulay used the term thirty years ago. It may make it easier for the research community to converge on a shared sense of notional machine, unlike, say, computational thinking, where this has not been possible.

The second affordance, discriminant utility, is useful because it gives us a reason to want to have a term like notional machine in our vocabulary when we already have other useful and related terms like explanation and model and pedagogical content knowledge. Why popularize a new term when you already have perfectly good ones? A good reason to do so is because you’d like to refer to a distinct set of things than those terms refer to.

The scope of our workshop was explicitly pedagogical: it was about notional machines “in education.” It was common within the workshop for people to refer to notional machines as pedagogical devices. It is often the case that notional machines are invented for pedagogical purposes, but other contexts may also give rise to them. Consider the case of Newtonian mechanics. Newton’s laws, and the representations that we construct around them (e.g. free body diagrams), were invented before Einstein described relativity. Newton’s laws weren’t intended as pedagogical tools but as tools to describe the laws of the universe, within the scales of size and velocity that were accessible to humans at the time. Today we sequence physics curriculum to offer up Newtonian physics before quantum because we believe it is easier to understand. But in many cases, even experts will continue to use it, even if they have studied (and hopefully understand) quantum physics. This is because in many cases, the additional complexity of working within a quantum model offers no additional utility over using the simpler abstractions that Newtonian physics provides. It doesn’t help one to predict the behavior of a system any better within the context of use, but likely does impose additional work on the system doing the calculation. So, while pedagogical contexts may be a primary locus for the generation, selection, and learning of notional machines, they are not solely of pedagogical value.

Within the workshop, I noticed that people often seemed to want their definitions, taxonomies, and examples of notional machines to include entities and details beyond those encompassed by the definition I have provided above. For example, some participants suggested that action rules can be, or be part of, notional machines. An example of an action rule might be “use descriptive variable names” or “make sure to check for None when programming in Python.” While both of these practices can be quite helpful, my definition of notional machines accepts neither of them. It rejects them because they aren’t about the rules by which a computer executes a program. In most languages, what one names variables does not matter, so long as one uses a name consistently within the appropriate scope. “Make sure to check for None” is a good heuristic for writing a correct program, but not an account of the rules a programming environment uses to run a program. In contrast, “dereferencing a null pointer causes a crash” is a valid notional machine, or at least a fragment of one.

Why do I want to exclude these things? Because a) I think it’s valuable to have a term that refers to the ways we communicate about what programming languages are and how the programs written in them will behave. And b) a broader definition will refer to just about everything that has anything to do with the practice of programming. That doesn’t seem worth having another term in our lexicon, and it would be less helpful for designing and interpreting research studies for computing education.

The third affordance is desirable because it may allow us to form stronger bridges to the programming languages research world. It allows us to examine — and value — the kinds of artifacts that they produce (programming languages and semantics for those languages) while also studying the contradictions between the values embedded in the production of those artifacts and the values that drive our own work. Programming languages (PL) researchers are generally quite focused on demonstrating the soundness of designs they create, but typically pay little attention to the usability of the artifacts they produce. Research languages and written (with Greek) semantics have difficult user interfaces, at least to those of us sitting on the outside of that community. How can we create a research community that includes the people, practices, and artifacts of PL and that conducts research on learning? One way is to decide to treat the practices and artifacts of PL researchers, such as writing down formal semantics, an instance of something that computing education researchers care about: producing explanations of how programming systems work. PL researchers describing languages’ semantics aren’t doing something that is very different in kind than what educators do when they explain how programming languages work. But (I think) they usually do so with greater precision and less abstraction than educators do. Educators’ abstractions may be metaphorical (e.g. “There’s a little man inside the box that reads what you wrote, and follows your instructions, line by line…”) but at least if we use my definition, they are of the same category as the descriptions that semanticists write down. As such, the range of things that can be notional machines, in addition to the programming languages they describe, may serve as boundary objects to link our communities together. I think we can learn a lot from each other.

That overlap presents opportunities. It’s an opportunity for us to learn from each other and an opportunity to conduct new lines of research. Imagine that we are faced with the desire to explain a programming system. How would a semanticist explain this system? How would an experienced teacher? An inexperienced teacher? What do the teachers’ explanations tell us about what’s important? What does a semanticist’s explanation tell us about what’s the kernel of truth that must be conveyed? How do these overlap? How do they diverge? What actually works for students? Can pedagogical explanations be more precise (and less metaphorical) and still be as helpful to students? Are more precise definitions actually more helpful to students than less precise ones? If so, what does one need to know to write a formal semantics? How does one learn to do that? How does one teach educators to do that? How can we design better programming languages, where better is defined as being easier to understand or use? How can we design better programming languages when we have different theories of what it means to program well? How do we support and assess learning of programming, and design programming languages and notional machines to explain them, when we have different goals for what’s important to accomplish with programming?

There are many other questions we could ask too. Several groups at the workshop held breakout sessions to brainstorm these, but I think it’s best to let them tell their own stories.

In summary, I think the term notional machines has value to computing education research, but only if we can come to a consensus about what the term means, and what it doesn’t. That’s my definition and why I’ve scoped it how I have. What’s your take?

If you’d like to read more (including viewpoints different than mine), make sure to check out Felienne’s and Amy’s blog posts on this same topic.

Thank you to Shriram, Mark, Jan, and Juha for organizing the workshop, and to the other participants in the workshop for many lively and generous conversations. Thanks as well to the wonderful Dagstuhl staff.

 

July 15, 2019 at 12:00 pm 14 comments

How to reduce the defensive climate, and what students really need to understand code: ITICSE 2019 Preview

This year, we’re presenting two papers at the 2019 ACM SIGCSE Innovation and Technology in CS Education (ITICSE) conference in Aberdeen, Scotland. I’ve been to ITICSE several times and enjoy it every time, but I can’t always justify the trip. This year, Barbara Ericson, Katie Cunningham, and I are all in Germany for a Dagstuhl Seminar on Notional Machines the week before, so we took the opportunity to submit and are fortunate to be invited to present.

Making CS Learning Visible: Case Studies on How Visibility of Student Work Supports a Community of Learners in CS Classrooms by Amber Solomon, Vanessa Oguamanam, Mark Guzdial, and Betsy DiSalvo

When I taught CS Ed Research this last semester (see the blog post about open questions from the class), the students so resonated with Lecia Barker’s papers about defensive climate (the classic paper is here). The story about how CS classes are “characterized by competitiveness rather cooperation, judgments about others, superiority, and neutrality rather than empathy” still rings true, 17 years after first written. Several of my students proposed research to follow-up on the original study.

Amber and Vanessa are also motivated by the concerns about defensive climate in CS classes, but they don’t want to measure it. They are suggesting an intervention.

They suggest that a community of learners approach would reduce defensive climate. Key to creating a community of CS learners, they propose, is making student work and process visible. Vanessa works in marker-oriented curricula, where student work is physical and the work process is visible. Amber did the evaluation of the AR design studio classroom that I’ve written about previously. In both of these case studies, they observed student communication patterns that were different from the defensive climate studies and more in keeping with a community of learners culture. Defensive climate is still a problem, and changing culture and community is the way to address it.

Novice Rationales for Sketching and Tracing, and How They Try to Avoid It by Katie Cunningham, Shannon Ke, Mark Guzdial, and Barbara Ericson

At ICER 2017, Katie presented a study of how students trace their programs on paper (see ICER 2017 paper here, and my blog post summary here). She had some fascinating and surprising results. For example, if students traced their programs only part way and then guessed at the final result, they were more likely to get the problem wrong than if they’d never traced before. But why? Why would students trace the program part way and then stop — only to get it wrong?

That’s what Katie explored in this follow-up paper. She had an innovative experimental design. She replicated her original tracing experiment, then pulled out about a dozen interesting participants, and invited them back for a retrospective interview. She could show them the original problem and what they traced — and then ask them why? Here’s one of the answers: They thought that they “got the hang of it.” They recognized a goal or pattern. They just recognized wrong.

One of my favorite parts of the paper is completely visual. Katie had a terrific idea — let’s ask the teacher of the class to trace the problems. Here’s one of the traces that the teacher did:

Here are some examples of what students did:

Notice a distinct lack of similarity? Why? Why don’t students trace like the instructor did?

This is a qualitative study, so it’s rich with interview data. I recommend reading the whole paper. There’s a neat part at the end where Katie points out, “Program visualizations do X. When students trace, they do Y. Why are these things so different?”

July 14, 2019 at 1:00 pm 5 comments

Older Posts


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 7,062 other followers

Feeds

Recent Posts

Blog Stats

  • 1,690,970 hits
October 2019
M T W T F S S
« Sep    
 123456
78910111213
14151617181920
21222324252627
28293031  

CS Teaching Tips