Paradigm shifts in education and educational technology: Influencing the students here and now

Back on my last blog post referencing Morgan Ames’ book The Charisma Machine, Alan Kay said in a comment, “What we have here is a whole world view and a whole different world.” I’ve been thinking about that sentence a lot because it captures what I think is going on here. A Kuhnian paradigm shift is happening (and maybe has already happened) in research around education and educational technology from the world of Papert and Bruner to the world of learning sciences. I am going to take a pass at describing the change that I see happening in the field, but I encourage you also to read the International Society of the Learning Science (ISLS) presidential address from Victor Lee here, which describes the field more authority and with more authenticity than me.

I remember asking Janet Kolodner (first editor-in-chief of the Journal of the Learning Sciences), “Why? Why learning sciences? We have educational psychology and cognitive science and so many other education disciplines.” She said that learning scientists were tired of just knowing what should happen. They wanted to get out to influence education practice and understand why learning doesn’t happen. Cognitive scientists mostly (at the time) ignored affect and motivation. Educational psychologists most often worked in controlled laboratories or experimental classrooms. Learning scientists wanted to understand and influence what was really happening in educational contexts, both formal and informal. More, they were devoted to expanding access to high-quality education. Yes, learning scientists explored cutting edge technologies to see what was possible, but even more, we try to figure out contexts that make or inhibit learning for real kids. Look at the titles of the Invited Speakers at ICLS 2020: Lost and found in dialogue: Embracing the promises of interdiscursivity and diminishing its risks, The Ed-Tech Imaginary, and Learning as an Act of Fugitivity. Words like “promises” and “imaginary” and “fugitivity” reflect a desire to change and to respond to what we thought might be, but discovered that reality is different. (Audrey Watters’ keynote is available as an essay here.)

David Feldon told me once that the field is misnamed. It’s much more “Learning humanities” than “Learning sciences.” Once you decide to study what’s going on in actual practice with actual students, you find that you’re mostly in studies with really small n. Contexts, teachers, and students vary wildly. Nobody that I know in learning sciences is trying to invent a general dynamic medium for thought, because it’s so hard to get anything actually adopted and used in an impactful manner. I see Jim Spohrer’s work in Service Science as being part of the same paradigm — how do you actually get services designed and implemented that work in practice?

This shift from the general to the specific, and from what could work to what does work is true in my research too. One of my recent NSF proposals is about working closely with a particular school district to figure out what is going to work there. What we know about Brookline or Brasil is almost irrelevant for this district. Another proposal is about inventing a dynamic medium for thought — but in a particular set of classes, in a task-specific form. I still would love to have a general dynamic medium for thought (as Alan suggests), but I believe we have to figure it out from the ground up. Over time, we will find specific notations that can work for specific tasks, and generalize as possible from there.

The majority of the literature that I draw on these days is about teachers: how they learn, why pre-service education has so little influence on actual teacher practice, and how to influence adoption. Teachers are a gateway for technology in the classroom. There are lots of technologies that could work with kids, but don’t work with teachers. In my work today, I draw on Bruner and Papert for their theoretical framings.. I draw on Bruner’s laboratory-based work (e.g., his definition of scaffolding). I draw on Papert’s descriptions of what the computer offers learners, e.g., its protean nature. But I draw less on their implementation work. Bruner’s MACOS was a brilliant project that had a catastrophic result because they didn’t consider enough what would actually work in US schools. Papert created interesting interventions that didn’t become systemic or sustained. Ames is telling me what’s going wrong in actual implementations of OLPC and may some of why it went wrong. If I want things to be actually adopted, I need to avoid the mistakes that The Charisma Machine is describing.

David’s description of what happened in Brasil in a comment to that earlier blog post is fascinating and super-useful, but doesn’t decrease the value of Ames’ description in Paraguay. I don’t agree with all of her rationalizations of why things turned out as they did (e.g., I don’t find the “technically precocious boys” perspective compelling or having explanatory power), and there are very likely things she missed. But what she describes obviously did happen. Learning from the experiences she describes informs our design processes and iterative feedback loops as a way of improving outcomes.

Like any paradigm shift, it doesn’t mean that all the work that went before is wrong. The questions being asked in each paradigm are different. They start from different world views. Papert and Bruner both offer a vision of what we want, Logo and MACOS. Both ran up against the reality of school in the US, where Thorndike won and Dewey lost. Now, how do we help every student, in real school contexts?

Nathan Holbert and David Weintrop recently told me a great phrase that’s common in the constructionism community (variously attributed to Seymour Papert or Uri Wilensky): “Are you designing for Someday or are you designing for Monday?” Are you designing for a world that might be, or are you designing for things that can go in the classroom soon? Neither are wrong. I don’t think that they even need to be a dichotomy. In my task-specific programming work, I’m making things that can’t go in the classroom Monday, but could go in the classroom next year, which is still a lot closer than Someday. Even to be in the classroom next year, I have to start from where schools are now. There won’t be a Dewey-an revolution in schools over the next year. But maybe Someday there will.

How do teachers teach recursion with embodiment, and why won’t students trace their programs: ICLS 2020 Preview

This coming week was supposed to be the International Conference of the Learning Sciences (ICLS) 2020 in Nashville (see conference website here). But like most conferences during the pandemic, the face-to-face meeting was cancelled (see announcement here). The on-line sessions are being announced on the ICLS2020 Twitter feed here.

I’m excited that two of my students had papers accepted at ICLS 2020. I haven’t published at ICLS since 2010. It’s nice to get back involved in the learning sciences community. Here’s a preview of their papers.

How do teachers teach recursion with embodiment

I’ve written here about Amber Solomon’s work on studying the role of space and embodiment in CS learning. This is an interesting question. We live in a physical world and think in terms of physical things, and we have to use that to understand the virtual, mostly invisible, mostly non-embodied world of computing. At ICER 2018, she used a taxonomy of gestures used in science learning to analyze the gestures she saw in a high school computer science classroom (see link here). Last summer at ITiCSE, she published a paper on how making CS visible in the classroom (through gesture and augmented reality) may reduce defensive climate (see link here). In her dissertation, she’s studying how teachers teach recursion and how learners learn recursion, with a focus on spatial symbol systems.

Her paper at ICLS 2020 is the first of these studies: Embodied Representations in Computing Education: How Gesture,Embodied Language, and Tool Use Support Teaching Recursion. She watched hours of video of teachers teaching recursion, and did a deep dive on two of them.

I’m fascinated by Amber’s findings. Looking at what teachers say and gesture about recursion from the perspective of physical embodiment, I’m amazed that students ever learn computer science. There are so many metaphors and assumptions that we make. One of the teachers says, when explaining a recursive function:

“Then it says “… “now I have to call.”

Let’s think about this from the perspective of the physical world (which is where we all start when trying to understand computing):

• What does it mean for a function to “say” something?
• The function “says” things, but I “call”? Who is the agent in this explanation, the function or me? It’s really the computer with the agency, but that doesn’t get referenced at all.
• Recursion is typically explained as a function calling itself. We typically “call” something that is physically distant from us. If a function is re-invoking itself, why does it have to “call” as if at a distance?

For most computer scientists, this may seem like explaining that the sky is blue or that gravel exists. It’s obvious what all of this means, isn’t it? It is to us, but we had to learn it. Maybe not everyone does. Remember how very few students take or succeed at computer science (for example, see this blog post), and what enormously high failure and drop-out rates we have in CS. Maybe only the students who pick up on these metaphors are the ones succeeding?

Why won’t students trace their programs?

Katie Cunningham’s first publication as a PhD student was her replication and extension of the Leeds Working group study, showing that students who trace program code successfully line-by-line are able to answer more accurately questions about the code (see blog post here). But one of her surprising results was that students who start tracing and give up do worse on prediction questions than those students who never traced at all. In her ITICSE 2019 paper (see post here), she got the chance to ask those students who stopped tracing why they did. She was extending that with a think-aloud protocol, when something unusual happened. Two data science students, who were successful at programming, frankly refused to trace code.

Her paper “I’m not a computer”: How identity informs value and expectancy during a programming activity is an exploration of why students would flat out refuse to trace code — and yet successfully program. She uses Eccle’s Expectancy Value Theory (which comes up pretty often in our thinking, see this blog post) to describe why the cost of tracing outweighs the utility for these students, which is defined in terms of their sense of identity — what they see themselves doing in the future. Sure, there will be some programs that they won’t be able to debug or understand because they won’t trace line-by-line. But maybe they’ll never actually have to deal with code that complex. Is this so bad?

Katie’s live session is 2:00-2:40pm Eastern time on June 23. The video link will be available on the conference website to registered attendees. A pre-print version of her paper is available here.

Both of these papers give us new insight into the unexpected consequences of how we teach computing. We currently expect students to figure out how their teachers are relating physical space and computation, through metaphors that we don’t typically explain. We currently teach computing expecting students to be able to trace code line-by-line, though some students will not do it (and maybe don’t really need to). If we want to grow who can succeed at computing education, we need to think through who might be struggling with how we’re teaching now, and how we might do better.

What do I mean by Computing Education Research? The Computer Science Perspective

 

Last week, I talked about how I explain what I do to social scientists. This time, let me explain what I do to computer scientists. I haven’t given this talk yet, and have only tried the ideas out on a few people. So consider this an experiment, and I’d appreciate your feedback.

Let’s simplify the problem of computing education research (maybe a case of a spherical cow). Let’s imagine that instead of classes of Real Humans, we are teaching programming to Human-like Turing Machines (HTMs). I’m not arguing that Turing machines are sufficient to represent human beings. I’m asking you to believe that (a) we might be able to create Turing Machines that could simulate humans, like those we have in our classes, (b) RH’s would only have additional capabilities beyond what HTM’s have, and (c) HTM’s and RH’s would similar mechanisms for cognition and learning. (Carl Hewitt has a great CACM blog post arguing that message passing is more powerful than TM’s or first order logic, so maybe these should be HMP, Human Message Passers. I don’t think I need more than TM’s for this post.)

This isn’t a radical simplification. Cognitive science started out using computation as a model for understanding cognition (see history here). Information processing theory in psychology starts from a belief that humans process information like a computer (see Wikipedia article and Ed Psychology reference). Newell and Simon won the ACM Turing award and in their Turing Award lecture introduced the physical symbol system hypothesis, “A physical symbol system has the necessary and sufficient means for general intelligent action.” If we have a program on a Turing machine that gives it the ability to process the world in symbols, our theory suggests that it would be capable of intelligence, even human-like intelligence. I’m applying this lens to how we think about humans learning to program.

This simplification buys me two claims:

• The Geek Gene is off the table. The Geek Gene is the belief that some people can’t learn to program (see blog post for more). Any Turing machine can simulate any other Turing machine. Our HTM’s are capable of tracing a program. If any HTM can also write code, then all HTM’s can write code. Everyone has the same computational capability. (If HTM’s can all code, then RH’s can all code, because HTM’s have a subset of RH cognitive capabilities.)
• Learning of our students can be analyzed and understood as information processing. The behavior of Turing machines is understandable with analysis. HTM’s are sophisticated Turing machines. The core mechanism of HTM’s can be analyzed and understood. If we think about our students as HTM’s, we might reason about their learning about computing.

Here are some of the research questions that I find interesting, within this framing.

How do HTM’s learn to program?

All HTM’s must learn, and learn at a level where their initial programming (the bootstrap code written on their tape when they come into our world) becomes indistinguishable from learned capabilities. HTM’s must have built-in programming to eat and to sleep. They learn to walk and run and decipher symbols like “A,” such that it’s hard to tell what was pre-programmed and what was learned. HTM’s can extend their programming.

There are lots of models that describe how HTM’s could learn, such as SOAR and ACT-R. But none so far has learned to program. The closest are the models used to build the cognitive tutors for programming, but those couldn’t debug and couldn’t design programs. They could work from a definition of a program to assemble a program, but that’s not what most of us would call coding. How would they do it?

How would HTM’s think about code? How would it be represented in memory (whether that memory is a tape, RAM, or human brains)? There is growing research interest in how people construct mental models of notional machines. Even experts don’t really know the formal semantics of a language. So instead, they have a common, “notional” way of thinking about the language. How does that notional machine get represented, and how does it get developed?

How do we teach HTM’s to learn to program?

You shouldn’t be able to just reprogram HTM’s or extend their programs by some manipulation of the HTM’s. That would be dangerous. The HTM might be damaged, or learn something that led them into danger. Instead, extending HTM’s programming can only be done by conscious effort by the HTM. That’s a core principle of Piaget’s Theory of Cognitive Development — children (RH’s and HTM’s) learn by consciously constructing a model of the world.

So, we can’t just tell an HTM how to program. Instead, we have to give them experiences and situations where they learn to program when trying to make sense of their world. We could just make them program a lot, on increasingly harder programs. Not only is that de-motivating (maybe not an issue for HTM’s, but certainly is for RH’s), but it’s inefficient. Turns out that we can use worked examples with subgoal labeling and techniques like Parson’s problems and peer instruction to dramatically improve learning in less time.

What native capabilities of HTM’s are used when they learn to code?

We know that learning to read involves re-using more primitive mechanisms to see patterns (see article here). When HTM’s learn to program, what parts of the native programming are being re-used for programming?

Programming in RH’s may involve re-use of our built-in ability to reason about space and language. My colleague Wes Weimer (website) is doing FMRI studies showing that programmers tend to use the parts of their brain associated with language and spatial reasoning. In our work, we have been studying the role of spatial reasoning and gesture in learning to program (see summaries of our ICER 2018 papers). We don’t know why spatial reasoning might be playing a role in learning to program. Maybe it’s not spatial reasoning, but some aspect of spatial reasoning or maybe it’s even some other native ability that is related to spatial reasoning.

How does code work as an external representation of HTM’s, and where does it help?

We can safely assume that HTM’s, like RH’s, would enhance their cognition through the use of external representations. Cognition and memory are limited. Even an infinite tape has limitations in terms of time to access. Human cognitive systems are limited in terms of how much can be attended to at once. RH’s use external representations (writing notes, making diagrams, sketches) to enhance their cognition. We’re assuming that HTM’s have a subset of RH abilities, so external representations would help HTM’s, too.

My students and I talk about a wonderful paper by David Kirsh, Thinking with External Representations (see link here). It’s a compelling view of how external representations give us abilities to think that we don’t have with just our brain alone.

How can program code be a useful external representation for HTM’s? When does it help, e.g., with what cognitive tasks is code a useful external representation? For example, a natural one is modeling and simulation — we can model more complex situations with program code than we can keep in our head, and we can simulate that model for a much larger range of time and possible values. Are there cognitive tasks where code by itself, as a notation like written language or mathematics, can enhance cognition? Here I’m thinking about the ability of code to represent causal relationships (e.g., as in Bruce Sherin’s work) or algebraic forms (e.g., as in Bootstrap) — see here for discussion of both.  I’m intrigued by the idea of the affordances of reading code even before writing it.

What makes programming worth learning for HTM’s?

Why should an HTM learn programming? Let’s assume that an HTM’s basic programming is going to be about staying alive, e.g., Maslow’s hierarchy of needs. When would an HTM want to learn programming?

The most obvious reason to learn programming is because you can get paid to do it. It’s about meeting physiological needs and safety. But, if you can meet those needs doing something that’s easier or more pleasant or has fewer barriers, you’ll likely do that.

Sometimes, you’ll want to learn programming because it makes easier something you want to do anyway. Brian Dorn’s graphic designers wanted to learn programming (see here) because they used Photoshop or GIMP and wanted a way to do that easier and faster. Maybe that’s about safety and physiological needs, but maybe it was about esteem or even self-actualization (if HTM’s care about those things).

Where my simplification breaks down: Real humans learn in situated and social contexts

Our learning theory about RH’s say that they are unlikely to start a new subject unless there’s social pressure to do so (see Pat Alexander’s Model of Domain Learning). Would HTM’s feel social pressure? Maybe.

As I described in the previous blog post, much of my work is framed around sociocultural models of learning, like Lave and Wenger’s situated learning. I use Communities of Practice to understand a lot of the situations that I explore. We can only go so far in thinking about programming as just being inside of individual minds (HTM or RH). Much of the interesting stuff comes when we realize that (a) our cognition interacts with the environments and situations around us, and (b) our motivation, affect, and cognition are influenced by our social world.

Setting aside whether it’s social science or computer science, I am still driven by a paper I read in 1982, which was five years after it was written: “Personal Dynamic Media” by Alan Kay and Adele Goldberg (see copy here).  I want people to be to use coding like they use other literacies, to create a literature, and in a casual, informal and still insightful way.  Mitchel Resnick often talks about people using Scratch to write a card to their mother or grandmother — that’s the kind of thing I want to see.  I want people to be able to make small computational models that answer questions, in the same way that people do “back of the envelope” calculations today. I also want great literature — we need Shakespeares and daVinci’s who convey great thoughts with computing (an argument that Andrea diSessa made recently at the PPIG conference which Felienne Hermans blogged about here.) That’s the vision that drives me, whether I’m using cognitive science or situated learning.

 

Applying diSessa’s Knowledge in Pieces Framework to Understanding the Notional Machine

In Lauren Margulieux’s blog where she summarizes papers from learning sciences and educational psychology, she takes on Andy diSessa’s 1993 paper “Toward an epistemology of physics” where diSessa applies his “knowledge in pieces” framework to how students develop an understanding of physics.  (See blog post here.)

The idea is that humans assemble their understanding of complex phenomenon out of knowledge of physical experiences, p-prims. Quoting Lauren:

Elements: P-prims are knowledge structures that are minimal abstractions of common phenomena and typically involve only a few simple parts, e.g., an observed phenomenon, like a person hitting a pen and that pen rolling across the table, and an explanation, like when people hit things, they move. P-prims are both phenomenological, meaning that they are interpretations of reality, and primitive, meaning that are (1) based on often rudimentary self-explanations and (2) an atomic-level mental structure that is only separated into parts by excessive force.

Cognitive Mechanism: P-prims are only activated when the learner recognizes similarities between a p-prim and the current phenomena. Recognition is impacted by many different features, such as cuing, frequency of activation, suppression, salience, and reinforcement. Because activation of p-prims depends on contextual features of phenomena, novices often fail to recognize relevant p-prims unless the contextual features align.

I find diSessa’s framework fascinating, and I’ve always wondered how we could apply it to students learning the notional machine (see blog post here on notional machine). My guess is that students use p-prims to develop their mental model of how the computer works, because — what else could they use? In the end, isn’t all our understanding grounded in physical experiences?  But using p-prims will likely lead to misconceptions since the notional machine is not based in the physical world.

Maybe this is a source of common misconceptions in learning computing.  The list of misconceptions that students have about variables, loops, scope, conditionals, and data structures is long and surprisingly consistent — across languages, over time.  What could possibly be the common source of all those misconceptions?  Maybe it’s physical reality.  Maybe students generally apply the same p-prims when trying to understand computing, and that’s why the same misconceptions arise. It’s sort of like using a metaphor to understand something in computing, but then realizing that the metaphor itself is leading to misconceptions.  And the metaphor that’s getting in our way is the use of physical world primitives for understanding the computational world.

Colleen Lewis, as a student of diSessa’s, uses the Knowledge in Pieces framework in her work.  In her terrific ICER 2012 paper, she does a detailed analysis of students’ debugging to identify misconceptions that they have about state. State is an interesting concept to study from a KiP perspective. It’s a common issue in CS, but less common in Physics. It’s not clear to me how students connect computational state to state in the real world.  Is it state like water being frozen or liquid, or state like being painted blue?  Do they get that state is malleable?

This is a rich space to explore in computing education. What are the p-prims for understanding the notional machine? How do students use the physical world to understand the computational one?

Read more of Lauren’s post here: Article Summary: diSessa (1993) Knowledge in Pieces Framework

How computing education researchers and learning scientists might better collaborate

Lauren Margulieux has started a blog which is pretty terrific.  I wrote about Lauren’s doctoral studies here, and I last blogged about her work (a paper comparing learning in programming, statistics, and chemistry) here.

In her blog, Lauren is explaining in lay terms papers from learning sciences, educational psychology, and educational technology.  She’s an interdisciplinary researcher, and she’s blogging to help others connect across disciplines.

Her most recent blog post is about an issue I’ve been thinking about a lot lately. I wrote a blog post in the summer about the challenge of bridging the modes of science and truth-seeking in (computing) education vs. computer science. Lauren summarizes a paper by Peffer and Renken about concrete strategies to be used between discipline-based education researchers (like math education researchers, science education researchers, or computing education researchers) and learning scientists. Quoting part of it below:

Challenges in Interdisciplinary Research: Collaboration within a field can be difficult as people attempt to reconcile different ideas towards one goal. Collaboration between fields, each with its own traditions in theory and methodology, can seem like a minefield. Below are some common challenges that DBERers and learning scientists face.

1. Differences in hard and soft sciences – researchers in the hard sciences can often feel frustrated by the lack of predictability in human-subjects research, and researchers in social sciences can become frustrated when those in the hard sciences have unrealistic expectations or view research in the soft sciences as non-scientific.

2. Differences in theories and frameworks – What constitutes a theory or framework can be different in different domains, confusing what is often a fundamental building block of research.

3. Differences in research methodologies – those unfamiliar with human-subjects research can find its methodologies complex, varied, and full of uncertainty, and those who have endured countless hours of training in these methodologies can find it difficult to describe or justify methodological decisions in a concise way.

See more at https://laurenmarg.com/2018/07/29/peffer-renken-2016-dber-and-learning-sciences-collaboration-strategies/

Constructivism vs. Constructivism vs. Constructionism

I wrote the below in 1997. I’m surprised that I still find references to it from time-to-time. That website may be going away soon, so I thought I’d put it here (only very slightly edited) in case others may find it useful.

I’d like to offer my take on the meaning of these words. I hear them used in so many ways that I often get confused what others mean by them.

Constructivism, the cognitive theory, was invented by Jean Piaget. His idea was that knowledge is constructed by the learner. There was a prevalent idea at the time (and perhaps today as well) that knowledge is transmitted, that the learner was copying ideas read or heard in lecture directly into his or her mind. Piaget theorized that that’s not true. Instead, learning is the compilation of complex knowledge structures. The learner must consciously make an effort to derive meaning, and through that effort, meaning is constructed through the knowledge structures. Piaget liked to emphasize learning through play, but the basic cognitive theory of constructivism certainly supports learning through lecture — as long as that basic construction of meaning takes place.

I don’t know who invented the notion of Constructivism, the educational philosophy, but it says that each students constructs their own, unique meaning for everything that is learned. This isn’t the same as what Piaget said. Piaget’s theory does not rule out the possibility that you and I may construct exactly the same meaning (i.e., exactly the same knowledge constructions) for some concept or domain. The philosophy of constructivism say that learners will construct their own unique meanings for concepts, so it is not at all reasonable to evaluate students as to how well they have all met some normative goal. (Radical constructivists go so far as to say that the whole concept of a curriculum makes no sense since we cannot teach anyone anything — students will always simply create their own meaning, regardless of what teachers do.) Philosophical constructivists emphasize having students take control of their own learning, and they de-emphasize lecture and other transmissive forms of instruction. This philosophical approach gets complicated by varying concepts of reality: If we all interpret things differently, is there any correct reality?

From my perspective, the assumption of constructivists is currently an untestable hypothesis. We know of no way to peer into someone’s mental constructions. Until we can, we do not know if you and I think about the concept of velocity differently or the same.

Constructionism is more of an educational method which is based on the constructivist learning theory. Constructionism, invented by Seymour Papert who was a student of Piaget’s, says that learning occurs “most felicitously” when constructing a public artifact “whether a sand castle on the beach or a theory of the universe.” (Quotes from his chapter “Situating Constructionism” in the book “Constructionism” edited by Papert and Idit Harel.) Seymour does lean toward the constructivist learning philosophy in his writings, where he talks about the difficulty of conveying a complex concept when the reader is going to construct their own meaning. In general, though, his claim is more about method. He believes that students will be more deeply involved in their learning if they are constructing something that others will see, critique, and perhaps use. Through that construction, students will face complex issues, and they will make the effort to problem-solve and learn because they are motivated by the construction.

The confusion that I and others have about these terms stems from (a) similar looking words and (b) meaning at different levels of the word construct. Piaget was talking about how mental constructions get formed, philosophical constructivists talk about how these constructions are unique (noun construction), and Papert is simply saying that constructing is a good way to get mental constructions built. Levels here are shifting from the physical (constructionism) to the mental (constructivism), from theory to philosophy to method, from science to approach to practice.

How do we create cyberattack defenders?

 

Roger Schank (famous AI and cognitive science researcher, the guy who coined the term “learning sciences”) is putting his expertise to the task of creating cyberattack defenders.  The description of his process (linked below) is interesting.  It has all the hallmarks of his work — innovative, informed by research, driven by concrete tasks.  Notice the strong claim that I quoted below.  We shouldn’t be aiming for general cyber attack defense skills.  These skills are going to be industry-by-industry specific.  He’s directly informed by the research that suggests that these skills are unlikely to generalize.

One of the big questions is: where are we going to get the students?  How do we recruit students into this kind of program?

How can we help? The cyber attack course Socratic Arts is building for the DOD will be modified to make the projects specific to particular industries. The banks’ problems are obvious: hackers might want to steal money. Pharma’s problems are obvious: hackers might want to steal secrets. We intend to put out versions of our cyber attack course for each industry. These courses will take 6 months for a student to complete. We are not interested in giving an overview in the typical one week course that is no more than an intro. We want to train real cyber attackers who can help. The only way to learn is by practice (with advice). That’s how you learn to ride a bike and that’s how you learn to do anything.

Source: Cyber Attack Academy

Learning Myths And Realities From Brain Science

Interesting results, but also, concerning.  People really believe that intelligence is “fixed at birth” and that teachers don’t need to know content?  The article has more of these:

On the topic of “growth mindset,” more than one-quarter of respondents believed intelligence is “fixed at birth”. Neuroscience says otherwise.

Nearly 60 percent argued that quizzes are not an effective way to gain new skills and knowledge. In fact, quizzing yourself on something you’ve just read is a great example of active learning, the best way to learn.

More than 40 percent of respondents believed that teachers don’t need to know a subject area such as math or science, as long as they have good instructional skills. In fact, research shows that deep subject matter expertise is a key element in helping teachers excel.

Source: Learning Myths And Realities From Brain Science : NPR Ed : NPR

Passing of William G. Bowen: Walk Deliberately, Don’t Run, Toward Online Education

William G. Bowen of Princeton and of the Mellon Foundation recently died at the age of 83. His article about MOOCs in 2013 is still relevant today.

In particular is his note about “few of those studies are relevant to the teaching of undergraduates.”  As I look at the OMS CS results and the empirical evidence about MOOC completers (which matches results of other MOOC experiments of which I’m aware at Georgia Tech), I see that MOOCs are leading to learning and serving a population, but that tends to be the most privileged population.  Higher education is critiqued for furthering inequity and not doing enough to serve underprivileged students.  MOOCs don’t help with that.  It reminds me of Annie Murphy Paul’s article on lecture — they best serve the privileged students that campuses already serve well.  That’s a subtle distinction: MOOCs help, but not the students who most need help.

What needs to be done in order to translate could into will? The principal barriers are the lack of hard evidence about both learning outcomes and potential cost savings; the lack of shared but customizable teaching and learning platforms (or tool kits); and the need for both new mind-sets and fresh thinking about models of decision making.

How effective has online learning been in improving (or at least maintaining) learning outcomes achieved by various populations of students in various settings? Unfortunately, no one really knows the answer to either that question or the important follow-up query about cost savings. Thousands of studies of online learning have been conducted, and my colleague Kelly Lack has continued to catalog them and summarize their findings.

It has proved to be a daunting task—and a discouraging one. Few of those studies are relevant to the teaching of undergraduates, and the few that are relevant almost always suffer from serious methodological deficiencies. The most common problems are small sample size; inability to control for ubiquitous selection effects; and, on the cost side, the lack of good estimates of likely cost savings.

Source: Walk Deliberately, Don’t Run, Toward Online Education – The Chronicle of Higher Education

What engagement looks like in a MOOC-based CS class

My colleague, Ashok Goel, is getting a lot of (deserved) attention for exploring the role of a cognitive assistant as a teaching assistant, known as Jill Watson. The question he’s exploring is: How do we measure the effect of this assistant?

One exploration involves engagement. I thought that these numbers were interesting, because they’re comparable to the ones I explored in my information ecology paper in CSCL many years ago. 38 or 32 notes student in a 15 week class is a couple per week. That’s not a dialogue, but it might be more engagement. What should we expect? Could those couple notes per week be suggesting greater learning elsewhere? Is it an indicator?

“We’re seeing more engagement in the course. For instance, in fall of 2015 before Jill Watson, each student averaged 32 comments during the semester. This fall it was close to 38 comments per student, on average,” Goel said. “I attribute this increased involvement partly to our AI TAs. They’re able to respond to inquiries more quickly than us.”

Source: Jill Watson, Round Three

How the tech sector could move in One Direction to get more women in computing

Thanks to Greg Wilson for sending this to me.  It takes a while to get to the point about computing education, but it’s worthwhile. The notion is related to my post earlier in the month about engagement and motivation.

I’d been socialised out of using computers at high school, because there weren’t any girls in the computer classes, and it wasn’t cool, and I just wanted to fit in.  I wound up becoming a lawyer, and spending the better part of twenty years masquerading as someone who wasn’t part of the “tech” industry, even though basically all of my time was spent online.

And I can’t begin to tell you how common it is. So what if your first experience of “code” is cutting and pasting something to bring back replies because Tumblr took them away and broke your experience of the site.

Is that any more or less valid than any dev cutting and pasting from Stack Exchange all day long?What if your first online experiences were places like Myspace and Geocities. Or if you started working with WordPress and then eventually moved into more complex themes and then eventually into plugin development? Is that more or less valid than the standard “hacker archetype”? Aurynn gave a great talk recently about the language we use to describe roles in tech. How “wizards” became “rockstars” and “ninjas”.  But also, and crucially, how we make people who haven’t followed a traditional path feel excluded.  Because they haven’t learnt the “right” programming language, or they haven’t been programming since they were four, or because, god forbid, they use the wrong text editor.

Source: How the tech sector could move in One Direction — Sacha Judd

A review of one of my favorite papers: Cognitive Apprenticeship (Collins, Brown, Newman)

I drew on Cognitive Apprenticeship a lot in my dissertation — so much so that Carl Berger asked me at my proposal, “Are you testing Cognitive Apprenticeship as a model?”  I had no idea how to respond, and 25 years later, I still don’t.  How do you test a conceptual framework?

Cognitive apprenticeship, like situated learning, starts from the assumption that apprenticeship is a particularly effective form of education. Then it asks, “How do you offer an apprenticeship around invisible tasks?”

What I like about the essay linked below is that it places cognitive apprenticeship in a broader context.  Apprenticeship isn’t always the best option (as discussed in the post about the Herb Simon paper).

Active listeners or readers, who test their understanding and pursue the issues that are raised in their minds, learn things that apprenticeship can never teach. To the degree that readers or listeners are passive, however, they will not learn as much as they would by apprenticeship, because apprenticeship forces them to use their knowledge. Moreover, few people learn to be active readers and listeners on their own, and that is where cognitive apprenticeship is critical–observing the processes by which an expert listener or reader thinks and practicing these skills under the guidance of the expert can teach students to learn on their own more skillfully.

Source: Cognitive Apprenticeship (Collins, Brown, Newman) | Reading for Pleasure

Power law of practice in software implementation: Does this explain the “W” going away?

I wonder if this result explains why the second semester students in Briana’s studies (see previous blog post) didn’t have the “W” effect.  If you do enough code, you move down the power law of practice, and now you can attend to things like context and generating subgoal labels.

Different subjects start the experiment with different amounts of ability and past experience. Before starting, subjects took a multiple choice test of their knowledge. If we take the results of this test as a proxy for the ability/knowledge at the start of the experiment, then the power law equation becomes (a similar modification can be made to the exponential equation):

That is, the test score is treated as equivalent to performing some number of rounds of implementation). A power law is a better fit than exponential to this data (code+data); the fit captures the general shape, but misses lots of what look like important details.

Source: The Shape of Code » Power law of practice in software implementation

Balancing cognition and motivation in computing education: Herbert Simon and evidence-based education

Education is a balancing act between optimally efficient instruction and motivating students. It’s not the same thing to meet the needs of the head and of the heart.

Shuchi Grover tweeted this interesting piece (quoted below) that reviews an article by Herb Simon (and John Anderson and Lynne Reder) which I hadn’t previously heard of.  The reviewer sees Herb Simon as taking a stand against discovery-based, situated, and constructivist learning, and in favor of direct instruction. When I read the article, I saw a more subtle message.  I do recommend reading the review piece linked below.

He [Herbert Simon] rejects discovery learning, and praises teacher instruction

When, for whatever reason, students cannot construct the knowledge for themselves, they need some instruction. The argument that knowledge must be constructed is very similar to the earlier arguments that discovery learning is superior to direct instruction. In point of fact, there is very little positive evidence for discovery learning and it is often inferior (e.g., Charney, Reder & Kusbit, 1990). Discovery learning, even when successful in acquiring the desired construct, may take a great deal of valuable time that could have been spent practicing this construct if it had been instructed. Because most of the learning in discovery learning only takes place after the construct has been found, when the search is lengthy or unsuccessful, motivation commonly flags.

Source: Herbert Simon and evidence-based education | The Wing to Heaven

Some cognitive scientists have been railing against the constructivist and situated approaches to learning for years. Probably the most important paper representing the cognitivist perspective is the Kirschner, Sweller, and Clark paper, “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.”  I talked about the Kirschner, Sweller, and Clark paper in this blog post with its implication for how we teach computer science.

The conclusion is pretty straightforward: Direct instruction is far more efficient than making the students work it out for themselves. Students struggling to figure something out for themselves does not lead to deeper learning or more transfer than simply telling students what they ought to do. Drill and practice is important. Learning in authentic, complex situations is unnecessary and often undesirable because failure increases with complexity.

The Anderson, Reder, and Simon article does something important that the famous Kirschner, Sweller, and Clark paper doesn’t — it talks about motivation. The words “motivation” and “interests” don’t appear anywhere in the Kirschner, Sweller, and Clark paper. Important attitudes about learning (like Carol Dweck’s fixed and growth mindsets, or Angela Duckworth’s grit) are not even considered.

In contrast, Anderson, Reder, and Simon understand that motivation is a critical part of learning.

Motivational questions lie outside our present discussion, but are at least as complex as the cognitive issues. In particular, there is no simple relation between level of motivation, on the one hand, and the complexity or realism of the context in which the learning takes place, on the other. To cite a simple example, learning by doing in the real-life domain of application is sometimes claimed to be the optimum procedure. Certainly, this is not true, when the tasks are life-threatening for novices (e.g., firefighting), when relevant learning opportunities are infrequent and unpredictable (e.g., learning to fly a plane in bad weather), or when the novice suffers social embarrassment from using inadequate skills in a real-life context (e.g., using a foreign language at a low level of skill). The interaction of motivation with cognition has been described in information-processing terms by Simon (1967, 1994). But an adequate discussion of these issues would call for a separate paper as long as this one.

…

There are, of course, reasons sometimes to practice skills in their complex setting. Some of the reasons are motivational and some reflect the special skills that are unique to the complex situation. The student who wishes to play violin in an orchestra would have a hard time making progress if all practice were attempted in the orchestra context. On the other hand, if the student never practiced as a member of an orchestra, critical skills unique to the orchestra would not be acquired. The same arguments can be made in the sports context, and motivational arguments can also be made for complex practice in both contexts. A child may not see the point of isolated exercises, but will when they are embedded in the real-world task. Children are motivated to practice sports skills because of the prospect of playing in full-scale games. However, they often spend much more time practicing component skills than full-scale games. It seems important both to motivation and to learning to practice one’s skills from time to time in full context, but this is not a reason to make this the principal mechanism of learning.

As a constructionist-oriented learning scientist, I’d go further with the benefits of a motivating context (which is a subset of what they’re calling a “complex setting”). When you “figure it out for yourself,” you have a different relationship to the domain. You learn about process, as well as content, as in learning what it means to be a scientist or how a programmer thinks. When you are engaged in the context, practice is no longer onerous but an important part of developing expertise — still arduous, but with meaning. Yasmin Kafai and Quinn Burke talk about changing students’ relationship with technology. Computer science shouldn’t just be about learning knowledge, but developing a new sense of empowerment with technology.

I’ve been wondering about what (I think) is an open research question about cognitivist vs. situationist approaches on lifelong learning. I bet you’re more likely to continue learning in a domain when you are a motivated and engaged learner. An efficiently taught but unmotivated learner is less likely to continue learning in the discipline, I conjecture.

While they underestimate the motivational aspect of learning, Anderson, Reder, and Simon are right about the weaknesses of an authentic context. We can’t just throw students into complex situations. Many students will fail, and those that succeed won’t be learning any better. They will learn slower.

Anderson, Reder, and Simon spend much of their paper critiquing Lave & Wenger’s Situated Learning. I draw on situated learning in my work (e.g., see post here) and reference it frequently in my book on Learner-Centered Computing Education, but I agree with their critique. Lave & Wenger are insightful about the motivation part, but miss on the cognitive part. Situated learning, in particular, provides insight into how learning is a process of developing identity. Lave & Wenger value apprenticeship as an educational method too highly. Apprenticeship has lots of weaknesses: inefficient, inequitable, and difficulty to scale.

The motivational component of learning is particularly critical in computing education. Most of our hot issues are issues of motivation:

• Female and under-represented minority students don’t learn differently than the white and Asian males who dominate computing. But they’re much less interested in learning computer science. “Computer science isn’t that difficult but wanting to learn it is.”
• Many students (especially female students) enter computer science class with low self-efficacy (see ICER 2016 award-winning paper). If they don’t believe they can do it, it’s not surprising that they don’t.
• We have too few computer science teachers because it is hard to motivate teachers to teach computer science.
• Students want authentic, complex learning situations, which is why we tend to value CS1 programming languages that are popular in industry. They don’t want practice and abstraction. Anderson, Reder, and Simon point out that more complex learning situations lead to more failure and lagging motivation.

The challenge to being an effective computing educator is to be authentic and complex enough to maintain motivation, and to use scaffolding to support student success and make learning more efficient. That’s the point of Phyllis Blumenfeld et al.’s “Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning.” (I’m in the “et al,” and it’s the most cited paper I’ve ever been part of.) Project-based learning is complex and authentic, but has the weaknesses that the cognitivists describe. Blumenfeld et al. suggest using technology to help students sustain their motivation and support their learning.

Good teaching is not just a matter of choosing the most efficient forms of learning. It’s also about motivating students to persevere, to tell them the benefits that make the efforts worthwhile. It’s about feeding the heart in order to feed the head.

Graduating Dr. Briana Morrison: Posing New Puzzles for Computing Education Research

I am posting this on the day that I am honored to “hood” Dr. Briana Morrison. “Hooding” is where doctoral candidates are given their academic regalia indicating their doctorate degree. It’s one of those ancient parts of academia that I find really cool. I like the way that the Wikiversity describes it: “The Hooding Ceremony is symbolic of passing the guard from one generation of doctors to the next generation of doctors.”

I’ve written about Briana’s work a lot over the years here:

• Her proposal is described here, “Cognitive Load as a significant problem in Learning Programming.”
• Her first major dissertation accomplishment was developing (with Dr. Brian Dorn) a measurement instrument for cognitive load.
• One of her bigger wins for her dissertation was showing that subgoal labels work for text languages too (ICER 2015).
• Another really significant result was showing that Parson’s Problems were a more sensitive measure of learning than asking students to write code in an assessment, and that subgoal labels make Parson’s Problems better, too.
• She worked a lot with Lauren Margulieux, so many of the links I listed when Dr. Margulieux defended are also relevant for Dr. Morrison.
• At ICER 2016, she presented a replication study of her first given vs. generated subgoals study.

But what I find most interesting about Briana’s dissertation work were the things that didn’t work:

• She tried to show a difference in getting program instruction via audio or text. She didn’t find one. The research on modality effects suggested that she would.
• She tried to show a difference between loop-and-a-half and exit-in-the-middle WHILE loops. Previous studies had found one. She did not.

These kinds of results are so cool to me, because they point out what we don’t know about computing education yet. The prior results and theory were really clear. The study was well-designed and vetted by her committee. The results were contrary to what we expected. WHAT HAPPENED?!? It’s for the next group of researchers to try to figure out.

The most interesting result of that kind in Briana’s dissertation is one that I’ve written about before, but I’d like to pull it all together here because I think that there are some interesting implications of it. To me, this is a Rainfall Problem kind of question.

Here’s the experimental set-up. We’ve got six groups.

1. All groups are learning with pairs of a worked example (a completely worked out piece of code) and then a practice problem (maybe a Parson’s Problem, maybe writing some code). We’ll call these WE-P pairs (Worked Example-Practice). Now, some WE-P pairs have the same context (think of it as the story of a story problem), and some have different contexts. Maybe in the same context, you’re asked to compute the average tips for several days of tips as a barista. Maybe in a different context, you compute tips in the worked example, but you compute the average test score in the practice. In general, we predict that different contexts will be harder for the student than having everything the same.
2. So we’ve got same context vs different context as one variable we’re manipulating. The other variable is whether the participants get the worked example with NO subgoal labels, or GENERATED subgoal labels, or the participant has to GENERATE subgoal labels. Think of a subgoal label as a comment that explains some code, but it’s the same comment that will appear in several different programs. It’s meant to encourage the student to abstract the meaning of the code.

In the GENERATE condition, the participants get blanks, to encourage them to abstract for themselves. Typically, we’d expect (for research in other parts of STEM with subgoal labels) that GENERATE would lead to more learning than GIVEN labels, but it’s harder. We might get cognitive overload.

In general, GIVEN labels beats out no labels. No problem — that’s what we expect given all the past work on subgoal labels. But when we consider all six groups, we get this picture.

Why would having the same context do worse with GIVEN labels than no labels? Why would the same context do much better with GENERATE labels, but worse when it’s different contexts?

So, Briana, Lauren, and Adrienne Decker replicated the experiment with Adrienne’s students at RIT (ICER 2016). And they found:

The same strange “W” pattern, where we have this odd interaction between context and GIVEN vs. GENERATE that we just don’t have an explanation for.

But here’s the really intriguing part: they also did the experiment with second semester students at RIT. All the weird interactions disappeared! Same context beat different context. GIVEN labels beat GENERATE labels. No labels do the worst. When students get enough experience, they figure things out and behave like students in other parts of STEM.

The puzzle for the community is WHY. Briana has a hypothesis. Novice students don’t attend to the details that they need, unless you change the contexts. Without changing contexts, students even GIVEN labels don’t learn because they’re not paying enough attention. Changing contexts gets them to think, “What’s going on here?” GENERATE is just too hard for novices — the cognitive load of figuring out the code and generating labels is just overwhelming for students, so they do badly when we’d expect them to do better.

Here we have a theory-conflicting result, that has been replicated in two different populations. It’s like the Rainfall Problem. Nobody expected the Rainfall Problem to be hard, but it was. More and more people tried it with their students, and still, it was hard. It took Kathi Fisler to figure out how to teach CS so that most students could succeed at the Rainfall Problem. What could we teach novice CS students so that they avoid the “W” pattern? Is it just time? Will all second semester students avoid the “W”?

Dr. Morrison gave us a really interesting dissertation — some big wins, and some intriguing puzzles for the next researchers to wrestle with. Briana has now joined the computing education research group at U. Nebraska – Omaha, where I expect to see more great results.