Learning to build machine learning applications without code as an example of computing education research

July 8, 2019 at 2:00 am 11 comments


Ben Shapiro shared a nice video that his student Abbie Zimmermann-Niefield made about their new paper at IDC. They built a system that allows kids to build applications with machine learning to detect (in this example) good from bad soccer passes.

The video (and paper) are perfect for IDC. It’s a wonderful example of giving students a new computational medium to build new kinds of applications that they couldn’t previously.  But the video also raised a bunch of questions for me.  Abbie at one point talks about users of her system as “learners.”  What are they learning?

David Moon put his finger on some of the issues for me with his tweet:

You don’t have to code to build ML applications.  But then, is it programming?  In the About page for this blog, I have defined computing education research as studying how people come to understand computing, and how to improve that process.  What are the students coming to understand in Abbie and Ben’s application?  Is studying how students come to build, understand, and debug their ML applications an example of computing education research?

I exchanged some messages with Ben, and came to an understanding of what he’s doing — which in turn gave me a new understanding of what I do.

In a blog post inspired by Juha Sorva, I suggested a refinement of my original definition.  “Coming to understand computing” means to develop a workable mental model or to learn a notional machine of a computing system. Programming is about intentionally defining a process for a computational agent to execute at another time. A notional machine is an explanation for the behavior of a system — it’s a teacher’s attempt to influence the mental model that the student is forming about the system.  I learned more about notional machines at a later Dagstuhl, and I’m excited to be attending a Dagstuhl Seminar this week where I’ll learn a lot more about notional machines.

Abbie’s participants are developing a mental model of how the system works — it’s not very elaborate, and it’s mostly wrong.  One student tells Abbie that she needs to have more both good and bad examples to make the system more accurate.  Okay, but why?

Ben says that they want to reach the point where students develop a working mental model of the system: About why they need to oversample some kinds of events, to be able to choose between different kinds of machine learning models, to be able able to judge what makes for a good data set, and to decide how to test the system to determine if it’s classifying all the desired inputs correctly.  Really, these are all the kinds of things we want students building any kind of program to think about — did I build this correctly (what I wanted it to be), how do I know, and how do I test to make sure?  Whether it’s by constructing data or by writing code, it’s still about intentionally defining a process for a computational agent, and then testing that process to determine if it matches the desired function.

It’s a fascinating question (which I expect we’ll be discussing this week) about what notional machines one uses to explain machine learning models.  It’s an important computing education research question: what mental models do students form about machine learning systems?  A different one is: what notional machines do we teach in order to improve the mental models that students develop about machine learning models?

Now, does it matter if students can’t see the code?  I don’t think so.  It probably matters for CS major undergraduates (which Ben, Peter, and Rebecca have argued elsewhere), but for the general population?  What does it mean to “see the code” anyway?

  • At a high level, I’m a big fan of block-based languages (as mentioned in a recent blog post about one of David Weintrop’s results). Block-based languages are also a higher-level representation of code.  That doesn’t matter. It’s still programming. It’s still computing.
  • At a low level, who really understand what their code does anymore?  With code optimization, processor prefetching, cache memory, and branch prediction, it’s really hard to know what’s really going on anyways.  Some people do. Most people don’t. And it really doesn’t matter.

The lack of code might make the notional machine harder to teach.  There is no code to point at when explaining an algorithm (see Amber Solomon’s work on the role of gestures in teaching CS).  Maybe you wouldn’t explain an algorithm.  Maybe instead you’d point at examples and point at key features of those examples.  Maybe. It’s an open and interesting research question.

So. Computing education is about helping students to develop mental models of computing systems. These models must be workable to the point of being usable for intentional construction and debugging. Studying how students build machine learning applications without code is also computing education research.

Entry filed under: Uncategorized. Tags: , , .

Iterative and interdisciplinary participatory design sessions: Seeking advice on a research method How to reduce the defensive climate, and what students really need to understand code: ITICSE 2019 Preview

11 Comments Add your own

  • 1. alanone1  |  July 8, 2019 at 3:37 am

    This seems very close to the 1980s “Learning to build communications applications without code” (i.e. multi-media WYSIWYG documents) as a meager notion of “computer literacy”.

    As Euclid said to Ptolemy: “Sire, there is no royal road to Geometry”.

    There’s no question that “code” is a slithery idea — just as is “language” in general. We often use single words — just like monkeys — to convey important ideas, but most notions of language involve combinations of words, where the organization of them means something important.

    So, there’s a threshold idea here (as there is for most subjects). Just being in the spectrum doesn’t count, the learner needs to be above a decent threshold. People who are only looking for general success tend to lower thresholds in a variety of ways, sometimes inadvertently — but always with rationalizations (because are the animal that feels it must generate reasons for its actions).

    For most modern knowledge (17th century onwards), the thresholds have been high enough to require a fair amount of learning effort. The effort is repaid by delivering “better brains” via the internal changes wrought by the difficult learning. This is a crowning glory of real education.

    Poor education — in this larger sense — is fitting tools to humans which don’t produce great upward changes (or, worse, wind up taking away something good from what was there).

    The pop culture is quite intertwined with seemingly desirable empty gesturing — and education has unfortunately more and more given in to the “Guitar Hero” approach to “guitar playing”.

  • 2. Ben Shapiro  |  July 8, 2019 at 2:40 pm

    Dear Alan,

    Thanks for your attention to our work.

    Obviously I disagree with your characterization of the work as meager and analogous to Guitar Hero.

    This is but a pilot study in a line of work that we very much intend to a) involve youth learning some nuanced ideas about applied ML, b) yield knowledge about how youth develop expertise, over time, for applying ML to personally meaningful problems.

    Like most design work, this work begins with the seed of an idea, and like much design-based educational research, this work also begins with an exploratory study to see whether what we envision might potentially have legs. We think it does.


    • 3. alanone1  |  July 8, 2019 at 2:49 pm

      HI Ben

      I was making a general statement about the difficulty the current culture has with valuing “success” over “content” rather than a specific criticism of your work (which I don’t know enough about to criticize at all).

      You are in a better position to assess the educational level of what you are doing than I am (hence my retreat to the general in my comment). So: how do you think this compares with e.g. learning real science and real mathematics?

      Best wishes


      • 4. Ben Shapiro  |  July 8, 2019 at 3:45 pm


        Got it. Thanks.

        I think it is real science and real mathematics (if you count applied statistics as mathematics). Or at least, that’s where we are heading: toward kids learning to develop useful measurements of physical phenomena that vary over time, to use those data to create inferential models around those data, to empirically evaluate the performance of those models, and then, based on that evaluation, to make choices about how to improve the model by modifying their data (e.g. by collecting additional data, removing data from their data set, or changing how/what they measure). I’m doubtful that Abbie (who studied math and is the daughter of two mathematicians) would be happy with anything less.

        Another question to ask is: is this real sports? That’s one of the harder problems for us right now. How do we integrate this into the work of athletes/dancers/cheerleaders in a way that actually enhances how they train, rather than distracts. The first study about that will begin next month.



        • 5. alanone1  |  July 8, 2019 at 3:49 pm

          Hi Ben

          What is the age range of the children you are trying to help?



          • 6. Ben Shapiro  |  July 8, 2019 at 3:56 pm

            Middle and high school; for this project, mostly HS.

            • 7. alanone1  |  July 8, 2019 at 4:06 pm

              OK, I’ll ponder this. I’m in London right now and will see if I can come up with something tomorrow …



              • 8. Ben Shapiro  |  July 8, 2019 at 4:06 pm

                Cool. And hello from Germany 🙂 — Ben

                • 9. alanone1  |  July 14, 2019 at 5:16 am

                  Well, I’ve been meaning to try to do something for middle schools and earlier in the areas of measuring, aggregates, probability, causality, reasoning, etc., and this got me started working on this in earnest … (so there will be a bit of a delay before a real reply …)

                  • 10. Ben Shapiro  |  July 14, 2019 at 8:52 am

                    Awesome! I look forward to seeing what you come up with. If you’d ever like to visit to discuss, present, etc. let me know.

  • […] a Dagstuhl Seminar about the concept of notional machines, as I mentioned in an earlier blog post about the work of Ben Shapiro and his student Abbie Zimmermann-Niefield. There is an amazing amount being written about the seminar already (see the Twitter stream here), […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,185 other subscribers


Recent Posts

Blog Stats

  • 2,060,406 hits
July 2019

CS Teaching Tips

%d bloggers like this: