Media Computation today: Runestone, Snap!, Python 3, and a Teaspoon Language

I don’t get to teach Media Computation¹ since I moved to the University of Michigan, so I haven’t done as much development on the curriculum and infrastructure as I might like if I were teaching it today. I did get a new version of JES (Jython Environment for Students) released in March 2020 (blog post here), but have rarely even started JES since then.

But using Jython for Media Computation is so 2002. Where is Media Computation going today?

I’ve written a couple of blog posts about where Media Computation is showing up outside of JES and undergraduate CS. Jens Moenig has been doing amazing things with doing Media Computation in Snap! — see this blog post from last year on his Snap!Con keynote talk. SAP is now offering a course From Media Computation to Data Science using Snap! (see link here). Barbara Ericson’s work with Runestone ebooks (see an example blog post here) includes image manipulation in Python inside the browser at an AP CS Principles level (see example here). The amazing CS Awesome ebook that Beryl Hoffman and Jen Rosato have been doing with Barb for AP CS A includes in-browser coding of Java for the Picture Lab (see example here).

I was contacted this last January by Russ Tuck and Jonathan Senning. They’re at Gordon College where they teach Media Computation, but they wanted to do it in Python 3 instead of Jython. You can find it here. It works SO well! I miss having the image and sound explorers, but my basic demos with both images and sounds work exactly as-is, with no code changes. Bravo to the Gordon College team!

Most of my research these days is grounded in Task-Specific Programming languages, which I’ve blogged about here (here’s a thread of examples here and here’s an announcement of funding for the work in social studies). We now refer to the project as Teaspoon Computing or Teaspoon Languages — task-specific programming => TSP => Teaspoon. We’re adding a teaspoon of computing into other subjects. Tammy Shreiner and I have contributed a chapter on Teaspoon computing to a new book by Aman Yadav and Ulf Dalvad Berthelsen (see announcement of the book here).

We have a new Teaspoon language, Pixel Equations, that uses Media Computation to support an Engineering course in a Detroit Public School. Here, students choose a picture as input, then (1) enter the boolean equations for what pixels to select and (2) enter equations for new red, green, and blue values for those pixels. The conditionals and pixel loops are now implicit.

In several of our tools, we’re now exploring bilingual or multilingual interfaces, inspired by Sara Vogel’s work on translanguaging (see paper here) and Manuel Pérez-Quiñones’s recent work on providing interfaces for bilingual users (see his TED talk here and his ACM Interactions paper here). You can see in the screenshot below that colors can be referenced in either English or Spanish names. We’re now running participatory design sessions with teachers using Pixel Equations.

I’m planning a series of blog posts on all our Teaspoon languages work, but it’ll take a while until I get there.

---

1. For new readers, Media Computation is a way of introducing computing by focusing on data abstractions used in digital media. Students write programs to manipulate pixels of a picture (to create photo filters), samples of a sound (e.g., to reverse sounds), characters of a text, and frames of a video (for video special effects). More at http://mediacomputation.org ↩

Rules work as a way of communicating computation at a mechanistic level without teaching programming

Sometimes as a reviewer, you get to read a paper that you wish was published immediately. That’s how I felt when I got to review Eliane Wiese and Marcia Linn’s paper “It Must Include Rules”: Middle School Students’ Computational Thinking with Computer Models in Science. It was published in ACM TOCHI in April (see link here).

Eliane and Marcia offer a solution to a problem that teachers face when they want to teach about computational models, but they don’t want to teach programming. How do you get students to reason about the models underlying the simulations they’re exploring without talking about program code? And if you do talk about some notation, some representation of the model, what can you expect students to reason about without teaching them the notation or representation first?

Eliane and Marcia show that rules work. They have students interact with simulations, and then show them rules that might be in that model. Like in a simulation of light, photosynthesis, and glucose levels in plants, a rule might be: When light is on, total glucose made increases.. Eliane and Marcia show rules to students and ask “Are these in the model?” In their abstract, they write:

In our sample, 99% of students identified at least one key rule underlying a model, but only 14% identified all key rules; 65% believed that model rules can contradict; and 98% could not distinguish between emergent patterns and behaviors that directly resulted from model rules. Despite these misconceptions, compared to the “typical” questions about the science content alone, questions about model rules elicited deeper science thinking, with 2–10 times more responses including reasoning about scientific mechanisms. These results suggest that incorporating computational thinking instruction into middle school science courses might yield deeper learning and more precise assessments around scientific models.

The misconceptions don’t bother me. Students will have misconceptions about models — that’s part of teaching science with models. What’s fascinating to me is that the rules worked. Students reasoned mechanistically about the computational models.

My favorite result in this study was where they asked students to predict what would happen if they added a new rule to the model. Basically, “What happens if we change the program like this?” Students were way better at playing these what-if games if the question was posed as a rule. Quoting from the paper:

Asking students to make predictions about the implementation of incorrect rules led to more scientific reasoning about mechanisms than simply asking students about a causal relationship portrayed in a correct model. This pattern was evident for both model contexts, with twice as many workgroups proposing mechanisms with the New Rule question compared to the Typical question for Global Climate (29% vs. 14%) and ten times as many workgroups doing so for Chemical Reactions (53% vs. 5%).

Students can reason about computational models described as rules, even without instruction on rules. That’s a terrific result. It’s one that I’m thinking about how to use in my task-specific programming languages.

Now, this isn’t saying that students can’t reason with function or with imperative statements. Maybe functional or procedural programming paradigms would work, too. Eliane and Marcia have found one approach that does work. They offer us a way to integrate computational modeling into science education, with real discussion of the mechanism of the models, without teaching programming first.

Katie Cunningham’s Purpose-first Programming: Glass box scaffolding for learning to code for authentic contexts

Last month, Katie Cunningham presented her CHI 2021 paper “Avoiding the Turing Tarpit: Learning Conversational Programming by Starting from Code’s Purpose.” The video of her presentation is available here. This is the final study from her dissertation work, about which I blogged here.

Katie is trying to support the kinds of programming learners whom she discovered in her work on tracing — students who want to write programs, but have no interest in understanding the details of how programs work. As one said to her (which became the title of her ICLS 2020 paper), “I’m not a computer.” Block-based programming won’t work for her learners because, like most conversational programmers, the authenticity of the language they’re learning matters. They don’t want to use blocks. They want to see the code that developers see — a form of what Cindy Hmelo-Silver and I called “glass-box scaffolding.”

Katie focused on one particular purpose: writing Python code to scrape Web pages using Beautiful Soup. She and Rahul Bejarano dug into Beautiful Soup code on Github and identified a set of code chunks (“plans”) that were really used for this purpose and which could be recombined in useful ways. She then developed a curriculum as a Runestone ebook for teaching those plans where she taught students how to combine them (using Parsons Problems) and, importantly, how to tailor them for specific needs. Here’s a figure from her paper with an example plan with a description of the “slots” for tailoring.

My favorite part of this study is her analysis of how students debugged using these plans. They did make mistakes, and they fixed them. They reasoned about their programs in terms of the plans. In a think aloud, they talked about the names of the plans and the slots, and where they tailored the plan wrong. It’s not that they were just copying and pasting chunks of Python code. They were reasoning about the chunks — but they were not doing much reasoning about Python. In some sense, she defined a task-specific programming language whose components happened to be defined in terms of visible lines of Python code.

My favorite outcome of the study is that students came away excited and felt that they were doing something “realistic” — from a half hour lesson. One participant asked if she could do this kind of learning for different purposes every week, a kind of DuoLingo for programming. Those are strong results from a short intervention. It is a pretty amazing intervention.

I blogged for CACM this month on how we we predict about knowledge transferring between programming languages may be based on an assumption of mathematics background which might have been true in the 1970’s but is less likely to be true today (see post here). I suggest that we need to develop ways of teaching programming that doesn’t relate to mathematics, that instead connect to the programmer’s purpose and task. Katie’s work is what I had in mind as an example.

Award-winning papers at ICER 2020 explore new directions and point towards the next work to do

The 2020 ACM SIGCSE International Computing Education Research Conference was in August (see website here), hosted in Dunedin, New Zealand — but was unfortunately entirely virtual. I became so much more aware of the affordances of face-to-face conferences when attending one of my favorite conferences all through my screen. The upside of the all-virtual format is that all the talks are available on YouTube (see ICER 2020 channel here). Here are my comments on the three papers receiving awards — see them listed here.

What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming. (Paper link)

This is the paper that I have read and re-read the most since the conference. The authors review what the literature tells us about metacognition in programming. Metacognition is thinking about thinking, like “Did I really understand that? Maybe I should re-read this. Or maybe I should write down my thoughts so I can reflect on them I’m not sure that I’m making progress here. Taking a walk would probably help me clear my head and focus.”

One of their findings that is most intriguing is “Metacognitive knowledge is difficult to achieve in domains about which the learner has little content knowledge.” In other words, you can’t teach students metacognition and self-regulation first, and then teach them something using those new thinking skills. Learners have to know some of the domain first. Now why is that?

Here’s a hypothesis: Metacognition and self-regulation are hard. They take a lot of cognitive load. You have to pay attention to things that are invisible (your own memory and thoughts) and that’s hard. Trying to learn or problem-solve at the same time that you’re monitoring yourself and thinking about your own learning — super hard. Maybe you have to know enough about the domain for some of that activity to be automatized, so that you don’t have to pay as much attention to it in order to do it.

So the biggest hole I see in this paper (which given that it’s a review paper, probably means that the hole is in the literature) is that it does not consider enough factors like gender, race, disability, or SES (e.g., wealth). (Gender gets mentioned when reporting Alex Lishinski’s great work, but only with respect to self-efficacy.) My hypothesis is that the story is more complicated when you consider non-dominant groups. If you don’t think you belong, that takes more of your attention, which takes attention away from your learning — and leaves even less attention for metacognition and self-regulation. If you’re worrying about your screen reader working or where you’re going to get dinner tonight, how do you also have attention left over for monitoring your learning?

The biggest unique opportunity I see in thinking about metacognition and programming is in thinking about debugging. Like psychology or veterinarian science, but unlike most other fields, a lot of a computer scientists’ job is in understanding the “thinking” (behavior, processing, whatever) of another agent. When you’re debugging your program, isn’t that a kind of metacognition. “Okay, what is the computer doing here? How is it interpreting what I wrote? Oh wait, is that what I wanted to write? Is that what I wanted to happen?” The complexity of mapping your thoughts and intentions to what you wrote to what the computer did is enormous. Now, debug someone else’s code — you’ve got what you want in mind, you’re constructing a model of mind of whoever wrote the code before you (did they know what they were doing? is this code brilliant or broken?), and you’re trying to figure out how the computational agent is “thinking about” the code. There’s some seriously complex metacognition going on there.

Exploring Student Behavior Using the TIPP&SEE Learning Strategy. (See paper here.)

No surprise that Diana Franklin’s CANON Lab at U. Chicago continues to do terrific and award-winning work. I’m excited about the TIPP&SEE learning strategy. A commonly found problem in computer science education is that students are bad at Explain In Plain English (EIPE) problems (e.g., see this SIGCSE 2012 paper on the topic). EIPE problems are a measure of students being able to step back from the structure and behavior of code to describe its function or purpose. Katie Cunningham has been exploring how some students focus more on the purpose of the programming, and others get stuck in the code and can’t see the purpose of the program. The TIPP&SEE learning strategy explicitly addresses these problems. Students are guided through how to understand a programming project and relating code to purpose.

This award-winning paper (which follows on their SIGCSE 2020 paper) shows us that students using the TIPP&SEE approach perform better than students who don’t. They get more of their programs done. The SIGCSE 2020 paper shows that they learn more.

The papers totally convince me that this strategy works. The next question is that I want to know is how and why. The SIGCSE paper does some qualitative work, but it’s pretty big n — 184 students. With this kind of scale, the programs are given and the problems are given. There’s not as much opportunity for the detailed cognitive interviews to figure out how the students are thinking about interpreting programs. What happens when these students just go to the Scratch website to look at something that they want to reuse? Do they use TIPP&SEE? Do they understand the programs that they just happen to come across? What happens when they want to build something, where they provide their purpose? Can they draw on TIPP&SEE and succeed? This is not a critique of the papers — they’re great and make real contributions. I’m thinking about what I want to know next.

Hedy: A Gradual Language for Programming Education. (See program link here.)

Easily my favorite paper at ICER 2020 this year. Felienne is doing what I am trying to do. Let’s invent new more usable programming languages! I am happy that she got this paper published, because (selfishly) it gives me hope that I can get my new work published. I am thrilled that the ICER community valued this paper so highly that it received a John Henry award.

The basic idea is to create a sequence of programming languages, where advancing levels have most of the elements of the previous level but include new elements. Her earliest level has no punctuation — no quotes, no semi-colons, no curly-braces. I recently built a task-specific programming language that had the same attribute, and one of the students I’m working with looked at it and asked, “Wait — you can have programming languages without all that punctuation? Well, then, why do we have so much when it scares people off?” Great question! When do we need all that extra punctuation, and where can we avoid it?

The next stage is to explore how we design languages like these. (I’m biased since this is where I’m spending most of my research time on these days.) Why do we choose those language features? Why the keywords print, ask, echo, assign, if, else, and repeat? How do we design and iteratively develop the language? How do we know that people can do things that they want to do with this language? My answer to this is participatory design with teachers, but there are many other viable answers. Felienne provides good design rationale for Hedy’s language features, based in literature from computing education and natural language acquisition. In a process of user experience (UX) design, we’d also user iterative development including testing with real users. This paper shows us use at large scale, and a big chunk of her paper describes what people did with it. It’s fascinating work, but we don’t talk to any of them. We don’t know what they liked, what they disliked, what they found frustrating, and what they were able to do. We need to move programming language design closer to user experience design — UX for PX.

All three papers are terrific contributions to the research community, and I plan to cite and built on them all. I’m eager to see what comes next!

---

Sidebar: I am a member of the ICER Steering Committee (which has no role in reviewing papers or in picking awards), and I was a metareviewer for ICER 2020. I am speaking here just for myself as a reader and attendee.

Let’s program in social studies classes: NSF funding for our work in task-specific programming languages

If we want all students to learn computer science (CS for All), we have to go to where the students are. Unfortunately, that’s not computer science class. In most US states, less than 5% of high school students take a course in computer science.

Programming is applicable and useful in many domains today, so one answer is to use programming in science, mathematics, social studies, and other non-CS classes. We take programming to where the students are, and hope to increase their interest and knowledge about CS. I love that idea and have been working towards that goal for the last four years. But it’s a hard sell. I told the story in 2018 (see post here) about how the mathematics teachers rejected our pre-calculus course that integrated computing. How do we help non-CS teachers to see value in computing integrated into their classes?

That’s the question Tammy Shreiner at Grand Valley State and I get three years to explore, thanks to a new grant from the US National Science Foundation in the research strand of the “CS for All” Program. Tammy teaches a course on “Data Literacy for Social Studies Teachers” at GVSU, and she (with her colleague Bradford Dykes) have been building an open educational resource (OER) to support data literacy education in social studies classes. We have been working with her to build usable and useful data visualization tools for her curriculum. Through the grant, we’re going to follow her students for three years: From taking her pre-service class, out into their field experiences, and then into their first classes. At each stage, we’re going to offer mentoring and workshops to encourage teachers to use the things we’ve showed them. In addition, we’ll work on assessments to see if students are really developing skills and positive attitudes about data literacy and programming.

Just a quick glimpse into the possibilities here. AP CS Principles exam-takers are now about 25% female. AP US History is 56% female exam takers. There are fives times as many Black AP US History exam-takers as AP CSP exam-takers. It’s a factor of 14 for Hispanic students. Everyone takes history. Programming activities in a history class reach a far more diverse audience.

I have learned so much in the last couple of years about what prevents teachers from adopting curriculum and technology — it’s way more complicated than just including it in their pre-service classes. Context swamps pre-service teaching. The school the teacher goes to influences what they adopt more than what they learned pre-service. I’ve known Anne Ottenbreit-Leftwich for years for her work in growing CS education in Indiana, but just didn’t realize that she is an expert on technology adoption by teachers — I draw on her papers often now.

Here’s one early thread of this story. Bahare Naimipour, an EER PhD student working with me, is publishing a paper at FIE next month about our early participatory design sessions with pre-service social studies teachers. The two tools that teachers found most interesting were CODAP and Vega-Lite. Vega-Lite is interesting here because it really is programming, but it’s a declarative language with a JSON syntax. The teachers told us that it was powerful, flexible — and “overwhelming.” How could we create a scaffolded path into Vega-Lite?

We’ve been developing a data visualization tool explicitly designed for history inquiry (you may remember seeing it back here). We always show at least two visualizations, because historical problems start from two accounts or two pieces of data that conflict.

As you save graphs in your inquiry to the right, you’re likely going to lose track of what’s what. Click on one of them.

This is a little declarative script, in a Vega-lite-inspired JSON syntax. It’s in a task-specific programming language, but this isn’t a program you write. This is a program the describes the visualization — code as a concise way of describing process.

We now have a second version where you can edit the code, or use the pull-down menus. These are linked representations. Changing the menu changes the code and updates the graph. Changing the code updates the menu and the graph. Now the code is also malleable. Is this enough to draw students and teachers into programming? Does it make Vega-Lite less overwhelming? Does it lead to greater awareness of what programming is, and greater self-efficacy about programming tasks?

We just had our first in-service teacher workshop with these tools in August. One teacher just gushed over them. “These are so great! How did I not know that they existed before?” That’s easy — they didn’t exist six months ago! We’re building things and putting them in front of teachers for feedback as quickly as we can, in a participatory design process. We make lots of mistakes, and we’re trying to document those, too. We’re about applying an HCI process to programming experience design — UX for PX.

---

If you know a social studies teacher who would want to keep informed about our work and perhaps participate in our workshops, please have them sign up on our mailing list. Thank you!

Please forward to high school history teachers: Task-specific programming in social studies data viz

Hi,

We built this tool, DV4L (Data Visualization for Literacy), to help teachers quickly create data visualizations for social study classes. This is currently a minimum viable product (MVP) version of our tool, and we would like to collect your feedback so that we can improve! 

• Please take a few moments and try out our tool at http://b48ca06e.ngrok.io/index.html
• We intend for DV4L to be intuitive and easy to use, so you shouldn’t need any instructions to start. (If you ever see a screen with ‘TOO MANY CONNECTIONS’ on the top, please just wait for a few seconds and refresh the page.)
• After you have tried out the tool, please take a few minutes to fill out this survey at https://forms.gle/jASed4f7GfTZD9xa8. 
  

We would really appreciate it if you could do this by Sunday Dec 8th to meet our class requirements. 

Feel free to share this with any other teachers or students who would like to try this out. Thank you so much!

I have been working with a team of four University of Michigan Computer Science students (in a class with Elliot Soloway) to develop a data visualization tool for history classes. They have a prototype ready for testing, and they need user data for their class by Sunday, December 8. Could you please forward this to any high school (or middle school) history teachers you know? They would also love to get some feedback from high school history students, too.

Here are three reasons why I’m excited about this tool.

First, the team really listened to us, our history professor collaborator (Tammy Shreiner), and the social studies teachers who gave us feedback on different data visualization tools. One of the students drove 2.5 hours to attend a participatory design session with social studies teachers in October. As an example, there are driving questions for each database, to guide students in what they might inquire about — that’s a specific request from Tammy.

Second, this is a tool for history inquiry. Data visualization tools like Vega-Lite and CODAP are terrific. (Readers of this blog know how impressed I am by Vega-Lite.) But they’re not designed for inquiry. As Bob Bain has taught me, historical inquiry starts from two pieces of evidence or two accounts that disagree. This tool is designed to support comparing different pieces of evidence, and maintaining a trace of what you’ve explored. Inquiry is about comparison, not from building a single visualization.

Third, this is task-specific programming in a subtle and interesting way. You build visualizations by making choices from pull-down menus on the left. As you find interesting graphs, you save them to the right. When you want to remember what the graph is about, you hover over it and get a textual representation of what pull-down menus generated this graph. I’m arguing that hover text is a program —- it’s a representation of the process for generating that graph, and it serves as a reminder of where you’ve been. It’s a program whose value is in reading it, not executing it.

Amy Ko told me once (I’m paraphrasing) that a program is a description of a process for the future. Using a tool is for now. A program represents the future. This program could be executed by the user in the future — set the pop-up menus to the same values, and you’ll get the same visualization. More importantly, the program represents the past and serves as a reminder for what you can do next.

Please do pass this around so that our team can get a sense of what’s working and what’s not in this prototype. Thanks!

Making the Case for Adaptive Parsons problems and Task-Specific Programming: Koli Calling 2019 Preview

I am excited to be presenting at the 19th Koli Calling International Conference on Computing Education Research (see site here). Both Barbara Ericson and I have papers this year. This was my third submission to Koli, and my first acceptance. Both of us had multiple rejections from ICER this year (see my blog post on ICER), so we updated and revised based on reviews, and were thrilled to get papers into Koli.

Investigating the Affect and Effect of Adaptive Parsons Problems

By Barbara Ericson, Austin McCall, and Kathryn Cunningham.

Barb is presenting the capstone to her dissertation work on adaptive Parsons problems (see blog post on her dissertation work here). This paper captures the iterative nature of her study. Early on, she did detailed think-aloud/interview protocols with teachers to understand how people used her adaptive Parsons problems. At the end, she looked at log files to get a sense of use at scale.

Abstract: In a Parsons problem the learner places mixed-up code blocks in the correct order to solve a problem. Parsons problems can be used for both practice and assessment in programming courses. While most students correctly solve Parsons problems, some do not. Un- successful practice is not conducive to learning, leads to frustration, and lowers self-efficacy. Ericson invented two types of adaptation for Parsons problems, intra-problem and inter-problem, in order to decrease frustration and maximize learning gains. In intra-problem adaptation, if the learner is struggling, the problem can dynamically be made easier. In inter-problem adaptation, the next problem’s difficulty is modified based on the learner’s performance on the last problem. This paper reports on the first observational studies of five undergraduate students and 11 secondary teachers solving both intra-problem adaptive and non-adaptive Parsons problems. It also reports on a log file analysis with data from over 8,000 users solving non-adaptive and adaptive Parsons problems. The paper reports on teachers’ understanding of the intra-problem adaptation process, their preference for adaptive or non-adaptive Parsons problems, their perception of the usefulness of solving Parsons problems in helping them learn to fix and write similar code, and the effect of adaptation (both intra-problem and inter-problem) on problem correctness. Teachers understood most of the intra-problem adaptation process, but not all. Most teachers preferred adaptive Parsons problems and felt that solving Parsons problems helped them learn to fix and write similar code. Analysis of the log file data provided evidence that learners are nearly twice as likely to correctly solve adaptive Parsons problems than non-adaptive ones.

Task-Specific Programming Languages for Promoting Computing Integration: A Precalculus Example

By Mark Guzdial and Bahare Naimipour

This is my first paper on the work I’m publishing on the new work I’m doing in task-specific programming. I mostly discuss my first prototype (see link here) and some of what math teachers are telling me (see link here). We also include a report on Bahare’s and my work with social studies educators A good bit of this paper is putting task-specific programming in a computing education context. I see what I’m doing as pushing further microworlds.

Typically, a microworld is built on top of a general-purpose language, e.g., Logo for Papert and Boxer for diSessa. Thus, the de- signer of the microworld could assume familiarity with the syntax and semantics of the programming language, and perhaps some general programming concepts like mutable variables and control structures. The problem here is that Logo and Boxer, like any general-purpose programming language, take time to develop proficiency. A task-specific programming language (TSPL) aims to provide the same easy-to-understand operations for a microworld, but with a language designed for a particular purpose.

Here’s the abstract:

Abstract: A task-specific programming language (TSPL) is a domain-specific programming language (in programming languages terms) designed for a particular user task (in human-computer interaction terms). Users of task-specific programming are able to use the tool to complete useful tasks, without prior training, in a short enough period that one can imagine fitting it into a normal class (e.g., around 10 minutes). We are designing a set of task-specific programming languages for use in social studies and precalculus courses. Our goal is offer an alternative to more general purpose programming languages (such as Scratch or Python) for integrating computing into other disciplines. An example task-specific programming language for precalculus offers a concrete context: An image filter builder for learning basic matrix arithmetic (addition and subtraction) and matrix multiplication by a scalar. TSPLs allow us to imagine a research question which we couldn’t ask previously: How much computing might students learn if they used a multiple TSPLs in each subject in each primary and secondary school grade?

Eventually the papers are going to appear in the ACM Digital Library. I have a preprint version of Barb’s paper here, and a longer form (with bigger screenshots) of my paper here.

Task-specific programming for and about computing education (Precalculus TSP Part 5 of 5)

I am exploring task-specific programming as a direct outgrowth of my work on GaComputes, ECEP, and ebooks. I’ve worked hard at helping computing education to grow in the US, but it’s not growing much (see my September Blog@CACM post for stats on that). There are too few people learning with the power of computing. It’s because we make programming so hard. We need to make programming more accessible, and one way to do that is to make it easier.

Why do we need to make it more accessible? My answer is: in order for people to use computer science for learning everything else. In 2009, when Matthias and Shriram wrote “Why computer science doesn’t matter” (see paper here), I hated it. Of course, computer science matters! Now I realize that they’re right. Nobody gets turned away from college admissions because they didn’t have high school CS. MANY students get turned away because they can’t pass Algebra 1. Other students don’t finish their degrees because they can’t get past Calculus. This other stuff really matters. I believe that we can use programming to help learn the stuff that really matters.

A key insight for me is that what students really use in Bootstrap:Algebra or even in Scratch is a small piece of programming. (I talked about this a good bit in my SIGCSE keynote.) We can reduce how much of programming we teach and still get huge benefits. Essentially, the students in Scratch and Bootstrap:Algebra are doing task-specific programming. I’m just going one step further to strip away even the trappings of a more general programming language. I’m making it as small as I can (but large enough to cover a learner’s task), so that we can increase usability, and thus increase the probability that we can apply programming to improve learning outcomes in other disciplines.

But it’s still programming, so the insights and theories of computing education research (CER) deeply influence this work on task-specific programming. In turn, task-specific programming offers the opportunity to ask some of our CER questions in new contexts. Here are two examples.

Notional Machines

At the Dagstuhl Seminar on Notional Machines (see post by Ben Shapiro), there was a key moment for me. Someone said something there about “Part of the notional machine for algebra.” I stopped and went all academic on them. “Wait a minute — if that’s a real rule used for evaluating algebra, then it’s not a notional machine. Notional machines are simplifications. That’s real algebra, not a notional machine.” There was a bit of a fight after that. It’s kind of a blur now.

In my two prototypes, I want the mathematics to be the notional machine. The notional machine for the image filter builder is matrix arithmetic and scalar multiplication. Underneath, it’s doing something more complicated, but I want students to completely understand what’s going on in terms of matrices.

The notional machine for the texture wave builder is a bit more complicated. My goal is for the notional machine to be just the wave function, but it’s a bit more than that. It’s also how the wave function maps to RGB values in a picture, and I’m actually not sure I have it right yet. If you have a wave where you just manipulate red here, and a wave that manipulates gray there (where red=green=blue at all pixels), then how do I combine the red component with the gray component in some reasonable way? I’ve tried a few ways already. I’ve thought about adding another task-specific language, just to let the students specify the mapping.

Of course, these are really simple programming models (no variables, no user-defined functions), so the notional machines are really simple, too. As much as possible, the notional machine is the context itself — math, or math+graphics. When does learning this notional machine help you learn other notional machines later?

And what have you learned if you learn those? Does task-specific programming help you learn more within the task domain? I hope that learning the matrix notional machine for image filters helps you with matrix manipulation later. Do students really learn precalculus from these prototypes?

If you learn the notional machine for task-specific programming, does that help you learn other notional machines later? There still is a computational notional machine embedded in there, e.g., about controlling the computational agent, about order of execution, and so on. Does that knowledge transfer to other computational contexts?

Structure-behavior-function models

My student Katie Cunningham is studying the use of structure-behavior-function (SBF) models for understanding how students come to understand programs. (I defined SBF models here). In short, this theoretical framing helps us understand the relationships between students learning to read code, to write code, to trace code, and to explain code in plain English.

Task-specific programming doesn’t fit the same way in that model. There is no writing of code. There is structure to the programs, but more of it is embedded in the environment than in the textual language. One of the insights from the participatory design sessions that we’ve had with teachers is that the environment is so much more powerful than the language. Consider the statement in my wave texture generator Set Gray to 4sin(5(x-3))+0. That does completely define the structure and transformation. However, the below picture is is so much more powerful and is what students really see — multiple, linked representations that explain that one line of code:

Behavior is complicated. As I said above, I want the behavior to be the notional machine of the mathematics. To trace the operation above, a student might plug in values for X to see what Y is generated, and check the plot and the wave to see if it makes sense. But it’s not like variable tracing.

But the explain in plain English task of figuring out the function is still there. Check out this image filter program:

Readers who know Media Computation or graphics will likely recognize that as the program to compute the negation of an image. How do we help students to do that? How do we help students to see the program and figure out what it does at a macroscopic level? I built tools into the wave texture builder to make it possible to see the role of each wave in the overall texture, but if you were to describe a texture as a “tightly-woven red and green plaid,” I’m not sure how you’d get that purpose/function from the definition of the waves The problem of figuring out the function is pretty much the same in task-specific programming.

Where to go from here

So this is the end of the series. I’ve described two prototypes for task-specific programming in precalculus (matrix transformations and wave functions), and explored the implications of task-specific programming for research about programming, in education, and in relation to computing education research (this post).

I did these as blog posts, in part, because I’m not yet sure where I might publish and fund this work.

• Most learning sciences work focuses on students. I’m focusing on teachers first.
• Most CS education work focuses on learning about CS, especially programming. I’m focusing on using CS programming for learning something else.
• Most programming languages work focuses, well…on languages. I’m focusing on programming where languages are a second-class citizen.
• Most work on CS in K-12 is focused on either computational thinking or teaching standalone CS classes. I’m focusing on integrating computing into classes with a goal of reducing computational thinking as much as possible.
• Most NSF funding in CS Education is tied to work with schools and districts (RPP), and is about CS integration in elementary school and CS-specific classes in high school (STEM+C).  I’m doing design work with teachers for CS integration at the high school level.
• There is funding (like NSF DRK12 and Cyberlearning) for developing interesting technology for learning, but I’m at the design stage. DRK12 has exploratory grants, but the funding level is too low to pay for my collaborators and my students. How do you get something like this started?

I’m seeing more clearly the point that Greg Nelson and Amy Ko talked about at ICER last year (see paper here). This is design-first work. It’s hard to find a home for that.

I’d appreciate your advice. Where is there a research community that’s concerned about these kinds of things? Where should I be publishing this work? Where should I be looking for funding?

 

Task Specific Programming will only matter if it solves a user’s problem (Precalculus TSP Part 4 of 5)

This is the fourth in a five part series about Precalculus Task-Specific Programming. I presented two prototypes in Parts 1 and 2, and discussed what I’m exploring about programming in Part 3.

I’ve shown my prototypes to several teachers — some computer science (e.g., I presented the first prototype at the Work In Progress Workshop at ICER in Toronto) and a half-dozen math teachers. The computer science teachers have been pretty excited, and I have had several requests for the system to try out with various student groups.

Why am I looking at precalculus?  Because it’s what leads to success in college calculus. I’m influenced by Sadler and Sonnert’s work showing that high school calculus isn’t the critical thing to support success in undergraduate calculus (see article here). It’s precalculus. Undergraduate calculus is a gatekeeper. Students don’t get STEM undergraduate degrees because they can’t get through calculus. So if we want more students to succeed in STEM undergraduate, we want high school precalculus to get better, in terms of more inclusive success.

Precalculus is a pretty abstract set of topics (see an example list here).  For the most part, it’s foreword looking: “You’ll need this when you get to Calculus.”  My idea is to teach precalculus with concrete contexts made possible by computing, like image filters. I want more students to find precalculus more engaging and more personally meaningful, leading to more learning.

So, might my prototypes help students learn precalculus?

Math teachers have generally been, “Meh.”

I’ve had four teachers tell me that it’s “interesting*.” One math teacher was blunt and honest — neither of these tools solve a problem that students have. Basic matrix notation and element-by-element matrix operations are the easiest parts of matrices. Precalculus students can already (typically) figure out how to plot a given wave equation.

What’s hard? Students struggle with forms of matrix multiplication and determinants. They struggle with what each of the terms in the wave function do, and what influences periodicity. Seeing the graphed points is good, but having the values display in symbolic form like (3*pi)/2 would be more powerful for making a connection to a unit circle. I’m learning these in a participatory design context, so I actually pushed more on what would be useful and what I need to do better — I have a much longer list of things to do better than just these points.

The math teachers have generally liked that I am paying attention to disciplinary literacy. I’m using their notations to communicate in the ways that things are represented in their math textbooks. I am trying to communicate in the way that they want to communicate.

Here’s the big insight that I learned from the mathematics teachers with whom I’ve spoken: Teachers aren’t going to devote class time or their prep time to something that doesn’t solve their problems. Some teachers are willing to put time into additional, enrichment activities — if the teacher needs more of those. As one teacher told me, “Most math classes are less about more exploration, and more about less failure.” The point of precalculus is to prepare students to pass calculus.  If you want more diverse students to get into STEM careers, more diverse students have to get through calculus. Precalculus is important for that. The goal is less failure, more success, and more student understanding of mathematics.

This insight helps me understand why some computational tools just don’t get a foothold in schools. At the risk of critiquing a sacred cow, this helps to explain why Logo didn’t scale. Seymour Papert and crew developed turtle geometry, which Andrea diSessa and Hal Abelson showed was really deep. But did Logo actually solve a problem that students and teachers had? Turtle graphics is beautiful, and being body syntonic is great, but that’s not the students’ problem with math. Most of their real problems with mathematics had to do with the cartesian coordinate system, not with being able to play being a turtle. Every kid can walk a square and a triangle. Did students learn general problem-solving skills? Not really. So, why should teachers devote time to something that didn’t reduce student failure in mathematics?

It would be hard to be disciplinary literate when Logo and turtle geometry was invented. Logo was originally developed on teletype machines. (See Cynthia Solomon’s great keynote about this story.) The turtle was originally a robot. Even when they moved Logo to the Apple II, they could not match the representations in the kid’s textbooks, the representations that the teachers were most comfortable with. So instead, we asked student to think in terms of fd 200 rt 90 instead of (x,y). Basic usability principles tell us to use what the user is familiar with. Logo didn’t. It demanded more of the students and teachers, and maybe it was worthwhile in long run — but that tradeoff wasn’t obvious to the teachers.

I have a new goal:

I want to provide a programming experience that can be used in five minutes which can be integrated into a precalculus class to improve on student learning.

I want to use programming to solve a learning problem in another domain. Programming won’t enter the classroom if it doesn’t solve a teacher’s problem, which is what they perceive as the student’s problem. Improving student learning is my users’ (teachers’) goals. Good UI design is about helping the user achieve their goals.

I’ve started on designs for two more precalc prototypes based on the participatory design sessions I’ve had, and I’m working on improving the wave texture generator to better address real learning problems. The work I’m doing with social science educators is completely driven by teachers and student learning challenges. That’s one of the reasons why I don’t have working prototypes there yet — it’s harder to address real problems.  My precalc prototypes were based on my read of literature on precalculus. That’s never going to be as informed as the teacher-in-the-classroom.

Now, there’s another way in which we might argue that these prototypes help with education — maybe they help with learning something about computer science? Here’s a slide from my SIGCSE 2019 keynote, referencing the learning trajectories work by Katie Rich, Carla Strickland, T. Andrew Binkowski, Cheryl Moran, and Diana Franklin (see this blog post).

You’re not going to learn anything about #1 from my prototypes — you can’t write imprecise or incomplete code in these task-specific programming environments. You can learn about #2 — different sets of transformation can produce the same outcomes. You definitely learn about #3 — programs are made by assembling instructions from a (very) limited set. If I were to go on and look at the Rich et al. debugging learning trajectories (see paper here), there’s a lot of that in my prototypes, e.g., “Outcome can be used to decide whether or not there are errors.”

So here’s the big research question: Could students learn something about the nature of programming and programs from using task-specific programming? I predict yes. Will it be transferable? To text or blocks language? Maybe…

Here’s the bigger research question that these prototypes have me thinking about. For the moment, imagine that we had tools like these which could be used reasonably in less than five minutes of tool-specific learning, and could be dropped into classes as part of a one hour activity. Imagine we could have one or two per class (e.g., algebra, geometry, trigonometry, biology, chemistry, and physics), throughout middle and high school. Now: Does it transfer? If you saw a dozen little languages before your first traditional, general-purpose programming language, would you have a deeper sense of what programs did (e.g., would you know that there is no Pea-esque “super-bug” homunculus)? Would you have a new sense for what the activity of programming is about, including debugging?

I don’t know, but I think it’s worth exploring task-specific programming more to see if it works.

Request to the reader: I plan to continue working on precalculus task-specific programming (as well as in social studies).  If you know a precalculus teacher or mathematics education researcher who would be interested in collaborating with me (e.g., to be a design informant, to try out any of these tools, to collaborate on design or assessment or evaluation), please let me know. It’s been hard to find math ed people who are willing to work with me on something this weird. Thanks!

---

* In the South, if you hear “Bless your heart!” you should know that you’re likely being insulted.  It sort of means, “You are so incompetent that you’re pitiful.”  I’ve learned the equivalent from teachers now.  It’s “That would make a nice enhancement activity” or “We might use that after testing.”  In other words, “I’d never use this. It doesn’t solve any of my problems.”

 

Task Specific Programming vs Coding: Designing programming like we design other user interfaces (Precalculus TSP Part 3 of 5)

In the last two posts, I described a prototype task-specific programming tool for building image filters from matrix manipulations, and for building textures out of wave functions.

So, let’s tackle the big question here: Is this programming?

These are both programs. Both of them are specification of process. Both of them allow for an arbitrary number of operations. In the image filter example (second example), the order of operations can matter (e.g., if you change the red matrix through scalar multiplication, then add or subtract it with another matrix, versus the reverse order). The image filter is also defining a process which can be applied to an arbitrary input (picture).

Wil Doane pointed out that this is not coding. No program code is written — not text, not blocks, and not even by spreadsheet formula. This is programming by layering transformations which are selected through radio buttons, pull-down menus, and text areas. It’s really applying HCI techniques to constructing a program.

Probably the big sticking point here is just how limited the programming language is here. Notice that I’m using the term task-specific programming and not task-specific programming language in this series. The programming language is important here as a notation: What transformations am I using here, and what’s the order of execution? If you didn’t get the effect you wanted, the language is important for figuring out which card you want to change. I’m working on more prototypes where the language plays a similar role: A notation for explanation and rationalization. Amy Ko has a paper I like where she asks, “What is a programming language, really?” (See link here.) I’m not sure that the use of programming language here fits into her scheme. Overall, the focus in these two prototypes is on the programming not the programming language.

I spent several months in Spring and Summer reading in the programming language literature — some research papers, but more books like the Little series (e.g., The Little Prover), Shriram Krishnamurthi’s Programming Languages: Application and Interpretation, and Beautiful Racket. I started a Pharo MOOC because it covered how to build domain-specific programming languages. I plan to return to more programming languages research, techniques, and tools in the future. I have a lot to learn about what is possible in programming, about notations for programming, and about alternative models of programming. I need better tools, too, because I’m certainly going to need more horsepower than LiveCode.

But as I started designing the prototypes, I realized that my focus was on the user interface, not the language. Of course, text and blocks are user interfaces, but they use few user interface mechanisms. Even what I’m doing here is a far cry from what’s possible in user interfaces. I am using common UI widgets like buttons, menus, and text areas in order to specify a program. I realized that I didn’t need programming language techniques for these prototypes. By any measure of programming language quality, these are poorly-designed languages. My real focus is on the design process. This approach worked for getting prototypes built so that I could start running participatory design sessions with teachers.

The goal of task-specific programming: Usability for Integration

Here’s the goal I set out for myself:

I want to provide a programming experience that can be used in five minutes which can be integrated into a precalculus class.

In the small sessions I’ve had with 1 or 2 people, yeah, I think it worked.

• I believe that this user interface is easier to learn and use than the comparable user interface (to do the same tasks) in text-based languages or even a block-based programming language like Snap! or GP.
• I’d even bet that the language itself (seen in the above screenshots) would be less usable if it wasn’t embedded in the environment, e.g., those two examples above are not that understandable outside of the two prototypes. The environment (e.g., visualizations, UI elements, matrix displays, etc.) are providing more information than the language is.

There are some interesting usability experiments to explore in the future with task-specific programming.  The empirical evidence we have so far on task-specific programming is pretty impressive, as I described in my June 2019 Blog@CACM post (see link here) and which has been published in the September CACM on the topic of “designing away Computational Thinking” (see link here), which is what I was trying to do. Sara Chasins’ published work showed that she had people solving real programming problems in less than 10 minutes. Vega-Lite is amazing, and we’re finding social studies teachers building visualizations in it in less than 20 minutes.  I’m curious to explore other directions in improving usability through these tiny, task-specific languages. For example, it is very easy here to represent the language in something other than English words. Perhaps we might use task-specific programming to make programming more accessible to non-English speakers by targeting languages other than English as the notation.

There’s a cost to that ease of use. There’s a commensurate loss in generalizability. This is laser-focused on TASK-SPECIFIC programming. My prototypes are not Turing-complete. There are no loops, conditionals, or variables. There is decomposition (e.g., which matrix transformations and wave equations do I need for my desired goal), but no abstraction.

I’m inspired by Amy Ko’s blog post: Programming languages are the least usable, but most powerful human-computer interfaces ever invented (see post here).

• Most attempts to improve on the usability problem are about making programming languages more usable, e.g., less complicated text languages or shifting modality to block-based languages. In general, though, the languages are just as powerful — and are still harder to understand than most students and teachers are willing to bear.
• There have been attempts to embed programming languages within applications, e.g., VBA in Office applications or JavaScript to script Photoshop. Again, these languages tend to be just as powerful, and not much more usable.
• I’m explicitly starting from an application (e.g., an image filter) and adding a programming language that only provides a notation for the facilities. It’s less powerful. I hope it’s more usable. I’m designing to be more like Excel and less like C.

Here’s the big idea here: We should design programming languages as we design other user interfaces. We should involve the users. We should aim to achieve the users’ goals. We should empower users. We should reduce their cognitive load.

Task-specific languages certainly can include more programming languages features.  For example, including variables with types makes sense for data science tasks (as in Bootstrap:Data Science). Those are part of data science tasks, and can help students avoid errors.  The critical part is including the components that help the learners with their tasks. My use of task-specific programming is really about learner-centered design of programming languages.

I have submitted two NSF proposals about building task-specific programming for precalculus and for social studies, both rejected. I spoke to both program officers. One told me it was impractical to propose a new programming language, and especially multiple languages. The officer pointed out that a language is hard and requires all kinds of infrastructure around it, like a community and a literature. I built these prototypes to explain what I meant by a “programming language.” It doesn’t have to be big. It can be little, be easy to learn and use, and fit into a one hour lesson in a precalculus class.

Alan Kay has argued for using the “real thing” as much as possible, because students might imprint on the “toy” thing and use it for everything. The concern is that it’s then hard to move on to better notations. I hope to avoid that fate by providing a notation that is so task-specific that it can’t be used for anything else. There is no support for abstraction here, alternative decompositions, or any kind of system design at all. It’s at the very lowest end of the Rich et al. learning trajectories.

Talking about trajectories is the segue to the next post.

I welcome your comments. Do you buy task-specific programming as a kind of programming?

Building Textures from Waves: Debugging in Task-Specific Programming (Precalculus TSP Part 2 of 5)

The second prototype I built for Task Specific Programming in Precalculus is about using wave functions for building textures. As I mentioned in my last blog post, my goal is to use computing to make precalculus less abstract and more relevant. I want to provide a programming experience that can be used in five minutes which can be integrated into a precalculus class. In this prototype, I explicitly designed for inquiry and debugging. What would debugging look like in task-specific programming when the goal is learning precalculus and not programming or computer science?

I’m going to describe this one bottom-up. Wave functions are a common topic in a precalculus class (like in this YouTube lecture). These are typically taught in the form:

A sin(B(x-C)) + D

Or

Amplitude sin(frequency(x-horizontalShift)+ verticalShift

I built a tool where the student specifies a wave function, then that wave is used to define a texture in terms of gray, red, green, or blue components.

This equation is plotted (lower right hand corner) where the X-axis controls the darkness. Each pixel is evaluated left-to-right, and it’s X-position is plugged into this formula. The Y value mapped to a gray scale. The plot should (if it all worked right, and it doesn’t really in this example) line up with the texture so that darkness and lightness maps to rise and fall in the darkness level of the texture.

Let’s set the “Challenge” section aside for just a few minutes.

We can map equations to other colors, and swap sine for cosine, and use the X or Y components of the pixel to determine the color. Like in the first prototype, each wave function appears on a separate card in a deck.

You can have an arbitrary number of these cards. When you get to the front page of the deck, you have a program of all the wave functions. These are then composed to create a more complex texture.

What does debugging look like here?

With this prototype, I thought more about the student’s process than I did with the image filter tool. In my work with both precalculus and social studies teachers, I’m increasingly thinking about inquiry, and I see debugging as a form of inquiry. You have some output, and you’re trying to understand it — and by “understand,” I mean being able to explain how the parts constitute the whole. The impetus to inquire (i.e., to debug) is that the output you have is not the output you expect or want. So, if we want to support inquiry as debugging, we have to support starting from a goal — an example, a requirement, or a set of tests.

I’m inspired by Diana Franklin’s group’s work on La Playa, a Scratch-like programming environment for 4th-6th grade students (see paper here). In La Playa, the programming tool opens up onto a starter file. The student could ignore it and just do open-ended programming, but more likely, the starter file guides the student and serves as a scaffold.

I made a prominent Challenge texture on the front page (and the challenge image appears on every wave card afterward). Can you make this? If you need a Hint, press the Hint button for a suggestion about is needed to make this texture.

In general, my challenge textures don’t really work. I built a bunch of cool textures to use as hints, like below, but the composition of color waves is one of the more complicated parts here — I talk more about it in Part 5 of the series.  But on the other hand, I showed some of these textures to researchers in culturally-relevant curricula who said that they reminded them of woven cloths from the cultures that they study.  Could we use wave functions in a form of culturally-relevant curriculum that blends precalculus and computing?

Imagine that you have a texture that you generated on the first page, and now you want to understand how your existing waves contribute to this texture, e.g, to figure out which equations need to be changed. One tool is the Pop button on the front Texture Program page. This pops out the texture into a separate window.

Now, you can drag the texture over the individual wave textures to understand how the pieces compose into the whole.

Each wave has a checkbox, so that the wave can be removed from the texture (without deleting), then put back in.

The idea is that a student can apply basic debugging principles: Compare to the example, find how this statement/computational-unit impacts the final outcome (like a print statement), and remove the statement/unit (like with a comment) to see the effect on the final outcome.

Multiple-linked representations

As I’ve shown this to math teachers, one of the things they like here are the multiple linked representations. We’ve got the equation:

When the student hovers, they get an explanation for each of the terms:

The plot and the texture, and if you press the “Points” button, you see exactly which points were plotted — both x,y on the graph, but also the computed x and y before they were mapped to the discrete points:

When I showed this to teachers, they wanted it to support more exploration. For example, they told me that students struggle to understand the difference between 3sin(x) and sin(3x). I was thinking that a useful addition to this tool would be a Bret Victor inspired scrubbing gesture over each of the terms in the wave function (as in his fabulous Learnable Programming essay, or in the Scrubbing Calculator). I want to be able to click on the “3” above, drag up, then move the cursor left or right to make that number less or more. As I scrub, I want to see the plot change and the texture change — but with some kind of shadow/echo so that I can compare the “3” plot/texture to the new values for “C.” Students can’t generally keep the original curve “in their head” when seeing the new one, so they need both to be visible to understand the effect of the equation change.

I built these in Livecode. The source file is available here. I’m also making binaries available for Macintosh and for Windows. (Yes, I can provide Linux, if someone’s interested.) These are provided with no warranty or guarantee — these barely do what I describe, and probably have lots of other mistakes, holes, weaknesses, and vulnerabilities. They’re slow as molasses. I strongly discourage you from using my prototypes with students. As I describe in Part 4 of this series, this is nowhere near where it needs to be in order to be useful.

Precalculus Task-Specific Programming for Building Image Filters from Matrix Operations (Precalculus TSP Part 1 of 5)

This is the start of a five part series of blog posts on Task-Specific Programming in Precalculus, subtitled “What Mark Did This Summer.” I wrote several blog posts in the Spring (last in May, linked here) about “task-specific programming languages.” I’ve been thinking a lot about them over the summer, and built my first prototypes for two task-specific programming environments. In these blog posts, I’m describing the two prototypes, and then considering the interaction with research on programming, on education, and on computing education research.

Both of my prototypes are built around teaching precalculus. My goal is to use computing to make precalculus less abstract and more relevant. I am inspired by Phil Sadler and Gerhard Sonnert’s work supporting the argument that precalculus is the most important set of concepts for students to succeed at college calculus (even more than high school calculus).  I want to provide a programming experience that can be used in five minutes which can be integrated into a precalculus class.

The first prototype is for constructing image filters from simple matrix manipulations that are part of many precalculus texts. In a digitized picture, each pixel has a red, green, and blue component (or channel) for its color. All the red values from the pixels can be described as a matrix for the red channel in the picture.

This is the opening screen for the prototype.

The image being manipulated is on the left. The list of matrix manipulations appears in the upper left. The manipulated image is on the upper right.

Across the bottom appear the matrices for the red, blue, and green pixels of this picture. A text description explains where these matrices come from. I tried hard to get the matrix representation close to what students might see in a precalculus textbook. That’s part of being able to be “integrated into a precalculus course.” The teachers I’ve shown this to have appreciated that attention — they don’t want to have to explain differences between the user interface and the textbook.

Changing the picture (see the button under the source picture), updates the matrices, the description, and re-runs the matrix transformation steps on the new input picture.

The lower right hand corner is an inspector to check individual pixel values. This was a suggestion from Line Have Musaeus from Aarhus when I demonstrated this prototype at ICER in Toronto. Precalculus teachers want students to understand how the math is working at the matrix-element level, so I give them a way to check individual matrix values. That’s when I added the block-color picture into the picture set — it makes it easier to understand the four quadrants, what the base colors are, and how the transformations change them.

Each of the matrix transformations appears on a separate card in a deck. Here’s one:

This transformation does a scalar multiplication of 0.5 (which the user sets by changing the text field in the statement). A pull-down lets the user pick which matrix they want to modify. When the user picks a matrix, the display in the lower left updates to show the new matrix. Again, I’m trying to get the formatting for scalar multiplication right from the textbook. The text description on the right updates to describe the selected transformation, and the text and matrix displayed on the lower right updates to show how that matrix is being transformed by this operation.

Here’s the other possible transformation (selected by radio button):

Here, we’re changing one of the channel matrices (pull down for red, green, or blue) to basic element-by-element arithmetic (addition or subtraction) of two selected matrices: red, green, blue, or a matrix of all-255’s. (I’ve had several suggestions to also provide the option of having a matrix of all the same number — all 1’s, all 0’s, or something like all-7’s).

I’m using a semi-transparent gray graphic to “hide” the unselected operation on the card. I want students to know that they can do either operation, so I don’t want it to disappear, but I want to show that it’s not selected.

Currently, the user presses the “Do It” button to process the individual transformations, or to do the whole program of transformations. You can have an arbitrary number of operations, and you can do some interesting transformations, like image negation:

Or making a scene look like it’s nearing sunset:

Press the Change Picture button to get a different input picture. So, the list of operations specifies a process which can be applied to an arbitrary input. It’s a program (for a specific task), but it’s not coding (an important distinction that Wil Doane pointed out to me at ICER this year).

I built these in Livecode. The source file is available here. I’m also making binaries available for Macintosh and for Windows. (Yes, I can provide Linux, if someone’s interested.) These are provided with no warranty or guarantee — these barely do what I describe, and probably have lots of other mistakes, holes, weaknesses, and vulnerabilities. They’re slow as molasses. I strongly discourage you from using my prototypes with students. As I describe in Part 4 of this series, this is nowhere near where it needs to be in order to be useful.

Iterative and interdisciplinary participatory design sessions: Seeking advice on a research method

Here’s an unusual post for this blog: I’m looking for a research methodology, and I don’t know where to look for it. I’m hoping somebody here will have a suggestion — please do forward this blog post to others you think might have suggestions for me.

We’re running participatory design sessions with teachers — asking teachers to try out programming languages with scaffolded activities, and then tell us about what they’d like for their classroom. I’m collaborating with Tammy Shreiner and Bradford Dykes at Grand Valley State University around having social studies teachers build data visualizations. We’re scouring the book Participatory Design for Learning, and in particular, we’re using Michelle Wilkerson’s chapter (which I’ve read twice now) because it matches the kind of work we’re doing.

Michelle uses a technique called Conjecture Mapping to describe how her teams thinks about the components of a participatory design session. A session has a specific embodiment (things you put into the classroom or session), which you hope will lead to mediating processes (e.g., participants exploring data, people talking about their code, etc.). These are processes which should lead to desired outcomes based on theory. A conjecture map is like a logic model in that it connects your design to what you want to have happen, but a conjecture map is less about measuring outcomes. Rather, it’s more about describing mediating processes, which you theorize will lead to desired outcomes. The mediating process column is really key — it tells you what to look for when you run the design session. If you don’t hear the kind of talk you want and see participant success in the activity, something has gone wrong. Fix it before the next iteration. The paper on this technique is Conjecture Mapping: An Approach to Systematic Educational Design Research by William Sandoval.

So here’s the first problem: We have different set of outcomes for our sessions. They’re interdisciplinary. I want to see the teachers being successful with their programs, my collaborators Tammy Shreiner and Bradford Dyke wants to see them talking about their data, and we all want to see participants relating their data visualizations to their history class (e.g., they shouldn’t be just making pretty pictures, and they should be connecting the visual elements to the historical meaning). Should we put all of these mediating processes and outcomes into one big conjecture map?

We are not satisfied with this combined approach, because we’re going to be iterating on our designs over time. Most participatory design approaches are iterative, but I haven’t seen a way of tracking changes (in embodiment or mediating practices) over time. Right now, we’re working with Vega-Lite and JavaScript. In our next iterations, we’ll likely do different examples with Vega-Lite. Over time, we want to be building prototypes of data visualization languages designed explicitly for social studies educators (task-specific programming languages).

We are concerned about two big problems as we iterate:

• Missing Out. I don’t want to lose any of our mediating processes. I want to make sure that we continue to see success with programming, engagement with data, and meaningful visualizations.
• Changing the balance. The easiest trap for our work will be to over-emphasize the programming, and swamp out the data literacy and data visualization processes and outcomes. If our sessions become perceived as primarily a programming activity, we’re moving in the wrong direction. We want the data literacy and data visualization to be primary, with programming as a supporting activity.

The diagram at the bottom may help describe the problem — it’s the sketch that my PhD student Bahare Naimipour and I made while talking through this problem. We need to track multiple disciplinary processes and outcomes over time as we iterate across different embodiments. This isn’t about assessing an intervention or design. This is about gathering input as we design and implement technology prototypes. We want to be moving in the right direction, for the processes and outcomes that we want.

Here’s where I’m asking for help: Where should we be looking for exemplars? Who else is doing iterative, multidisciplinary participatory design sessions? What are good methods for use to use?

Thanks!

Why I say task-specific programming languages instead of domain-specific programming languages

I’ve written several posts about task-specific programming languages over the last few weeks (here’s the first one), culminating in my new understanding of computational thinking (see that blog post).

The programming languages community talks about “domain-specific programming languages.”  That makes a lot of sense, as a contrast with “general purpose programming languages.” Why am I using a different term?

It’s inspired from my interaction with social studies teachers. They talk about “the language used in math class” and about “what language should we use in history?” History and mathematics are domains. If we talk about a programming language for all of history, that’s too big. It will be difficult to design languages to be easily learned and used.  There are lots of tasks in history that are amenable to using computing to improve learning, including data visualization and testing the rigor of arguments.

“Task-specific programming language” makes clear that we’re talking about a task, not a whole domain. I don’t want teachers rejecting a language because “I can’t use it for everything.”  I want teachers to accept a language because it helps their students learn something. I want it to be so easy to learn and use, that (a) it’s not adding much additional load and (b) it’s obvious that it would help.

I like “task-specific programming language,” too, because the name suggests how we might design them. Human-computer interface researchers and designers have been developing methods to analyze tasks and design interfaces for those tasks for decades. The purpose of that analysis is to create interfaces for users to achieve those tasks easily and with minimal up-front learning.  For 25 years (Soloway, Guzdial, and Hay, 1994) , we have been trying to extend those techniques to design for learners, so that users achieve the tasks and learn in the process.

Task-specific programming languages are domain-specific programming languages (from the PL community) that are designed using learner-centered design methods (HCI).  It’s about integrating between two communities to create something that enables integration of computing across the curriculum.

 

A new definition of Computational Thinking: It’s the Friction that we want to Minimize unless it’s Generative,

David Benedetto wrote a blog post about computational thinking for CSTA that gave me new insight into Computational Thinking (thanks to Shuchi Grover whose tweets drew me to it):

http://advocate.csteachers.org/2019/02/27/situated-computational-thinking/

David says:

I think this definition of CT is as good a starting point as any:

Computational Thinking is the thought processes involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information-processing agent (Cuny, Snyder, Wing, 2010).

He evolves this position until, like Shuchi, he comes up with two definitions of CT:

What are the implications of this? I think there are two clear options for how we define CT:

(A) Restrict what we mean by CT. This is perfectly reasonable and probably necessary for most practical purposes. However, this has the inevitable consequence of fragmenting our understanding of CT. There will be different CTs in different disciplines / fields. We will do this, but we should try to understand the restrictions that we are imposing, and the consequences of imposing them.

(B) Break our concept of CT wide open. I think the scientific community (at least, those who are studying the construct of CT and how it plays out in real cultural contexts) should do this, so that we can explore how CT is understood and practiced in a variety of contexts and for a wide range of purposes.

As a researcher, I’m more in favor of the former — let’s define Computational Thinking precisely.  David’s concern is really about the social context around CT. People want to call lots of things Computational Thinking. Can we come up with a definition for CT that bridges these? That represents the discipline-specific uses of CT, and is well enough defined that we can actually measure something about it?

There are many other “thinkings” that lay claim to providing students with critical skills. Admiral Grace Hopper would likely support “mathematical thinking” more than “computational thinking,” as this interesting essay from Yale points out. Skills like “decomposition” or “abstraction” are included in many definitions of computational thinking (eg this blog post), and it’s true that you need those in computing.  But those skills first belonged to mathematics, engineering, and science, and I’d argue that the teachers in those subjects might be in a better position to teach them and to measure them. Computation can play an important role in learning decomposition and abstraction, but those skills don’t belong uniquely to computation, or to a class on computational thinking. So, what is unique about computation?

The tension between HCI and Computational Thinking

On the computer science side of my life, my research community is human-computer interaction.  I’ve published in CHI, DIS, CSCW, VL/HCC, and UIST. The Cuny, Snyder, and Wing definition is hard for me to reconcile with being an HCI researcher.  The point of HCI research is to minimize the amount that a user has to learn in order to “formulate problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information-processing agent.”  HCI is trying to make it easier for the user to think with a computer whatever they want to think about. Computational Thinking is about what you need to think with a computer.

Over the last few weeks in this blog, I’ve been exploring the notion of task-specific programming languages. I was amazed at how much social studies teachers could do with Vega-Lite in a participatory design session we ran in early March. Sarah Chasin’s work with Helena and Rousillon is absolutely stunning for how much people could achieve with no training. Hariharan Subramonyam sent me this fascinating essay on end-user programming and about how to minimize the effort it takes end users to start programming: https://www.inkandswitch.com/end-user-programming.html. As I talked about in my SIGCSE 2019 keynote, Bootstrap:Algebra and most uses of Scratch actually rely on a small number of computational ideas. There is expressive and learning power in even a small amount of computation.

Michael Mateas wrote an essay back in 2009 that has been influential in my thinking. I blogged about it here: “There will always be friction.” Michael looked at the Alan Perlis talk of 1961 (that I talk and write about often), and particularly, at the exchange with Peter Elias. Elias argued that students shouldn’t have to learn to program — the computer should learn to understand us. Both Perlis and Mateas disagree. The computer can never understand us completely. We have to smooth the communication because the computer cannot. There will always be a challenge to human-computer interaction. There will always be friction, and it’s the human’s job to manage that friction..

A New Definition for Computational Thinking

So, here’s my new take on Computational Thinking: It’s the friction. Let’s take the original Cuny, Snyder, and Wing definition — computational thinking is about framing problems so that computers can solve them. The work around task-specific programming languages is showing us that we can make that amount that the user has to learn in order to use programming for their problem very small.

To meet Alan Kay’s point about generativity, there are some things in computing that we want to teach because they give us new leverage on thinking. We want to teach things that are useful, but not those that are necessary just because we have bad user interfaces.

A minimal definition of Computational Thinking: The stuff that we have to learn in order to communicate our tasks with a computer. It should be small, and the part that we learn should be generative, useful for new problems and new thinking. Everything else should be eliminated by good user interfaces.

You don’t have to master abstraction and decomposition just to use a programming language to help you learn. Our social studies teachers modified Vega-Lite programs, made mistakes (every single one of them) and recovered from them, and tried new things that they figured out on their own — all in 10-20 minutes. They already have problem solving skills. They didn’t need any “computational problem solving skills.” They certainly didn’t learn any special computational abilities to abstract and decompose in 10 minutes. They already know enough to use programming to learn. If we can eliminate the need for a particular skill in order to use computing to learn something else, we should.

This meshes with David Weintrop and Uri Wilesnky’s definition — it’s the computational practices of actual scientists and engineers who use computing. Their definition is particularly strong because it’s empirically grounded. They asked computational scientists and engineers what they actually do. Weintrop and Wilesnky’s participants want to do their work, not programming for its own sake. So they use a minimal subset of computing that buys them something for their thinking and in their tasks.

I like this definition because it’s aspirational.  Today, there’s a lot of stuff that you have to learn to use a computer to solve problems. Philip Guo gave a talk here at Michigan recently (similar to one he gave at U-W) and described how data scientists have to become system administrators to manage all the various packages and databases to do their job.  That’s a problem. That’s not computational thinking. That’s stuff to get rid of. How small can we make computational thinking?