Posts tagged ‘learning sciences’
William G. Bowen of Princeton and of the Mellon Foundation recently died at the age of 83. His article about MOOCs in 2013 is still relevant today.
In particular is his note about “few of those studies are relevant to the teaching of undergraduates.” As I look at the OMS CS results and the empirical evidence about MOOC completers (which matches results of other MOOC experiments of which I’m aware at Georgia Tech), I see that MOOCs are leading to learning and serving a population, but that tends to be the most privileged population. Higher education is critiqued for furthering inequity and not doing enough to serve underprivileged students. MOOCs don’t help with that. It reminds me of Annie Murphy Paul’s article on lecture — they best serve the privileged students that campuses already serve well. That’s a subtle distinction: MOOCs help, but not the students who most need help.
What needs to be done in order to translate could into will? The principal barriers are the lack of hard evidence about both learning outcomes and potential cost savings; the lack of shared but customizable teaching and learning platforms (or tool kits); and the need for both new mind-sets and fresh thinking about models of decision making.
How effective has online learning been in improving (or at least maintaining) learning outcomes achieved by various populations of students in various settings? Unfortunately, no one really knows the answer to either that question or the important follow-up query about cost savings. Thousands of studies of online learning have been conducted, and my colleague Kelly Lack has continued to catalog them and summarize their findings.
It has proved to be a daunting task—and a discouraging one. Few of those studies are relevant to the teaching of undergraduates, and the few that are relevant almost always suffer from serious methodological deficiencies. The most common problems are small sample size; inability to control for ubiquitous selection effects; and, on the cost side, the lack of good estimates of likely cost savings.
My colleague, Ashok Goel, is getting a lot of (deserved) attention for exploring the role of a cognitive assistant as a teaching assistant, known as Jill Watson. The question he’s exploring is: How do we measure the effect of this assistant?
One exploration involves engagement. I thought that these numbers were interesting, because they’re comparable to the ones I explored in my information ecology paper in CSCL many years ago. 38 or 32 notes student in a 15 week class is a couple per week. That’s not a dialogue, but it might be more engagement. What should we expect? Could those couple notes per week be suggesting greater learning elsewhere? Is it an indicator?
“We’re seeing more engagement in the course. For instance, in fall of 2015 before Jill Watson, each student averaged 32 comments during the semester. This fall it was close to 38 comments per student, on average,” Goel said. “I attribute this increased involvement partly to our AI TAs. They’re able to respond to inquiries more quickly than us.”
Source: Jill Watson, Round Three
Thanks to Greg Wilson for sending this to me. It takes a while to get to the point about computing education, but it’s worthwhile. The notion is related to my post earlier in the month about engagement and motivation.
I’d been socialised out of using computers at high school, because there weren’t any girls in the computer classes, and it wasn’t cool, and I just wanted to fit in. I wound up becoming a lawyer, and spending the better part of twenty years masquerading as someone who wasn’t part of the “tech” industry, even though basically all of my time was spent online.
And I can’t begin to tell you how common it is. So what if your first experience of “code” is cutting and pasting something to bring back replies because Tumblr took them away and broke your experience of the site.
Is that any more or less valid than any dev cutting and pasting from Stack Exchange all day long?What if your first online experiences were places like Myspace and Geocities. Or if you started working with WordPress and then eventually moved into more complex themes and then eventually into plugin development? Is that more or less valid than the standard “hacker archetype”? Aurynn gave a great talk recently about the language we use to describe roles in tech. How “wizards” became “rockstars” and “ninjas”. But also, and crucially, how we make people who haven’t followed a traditional path feel excluded. Because they haven’t learnt the “right” programming language, or they haven’t been programming since they were four, or because, god forbid, they use the wrong text editor.
I drew on Cognitive Apprenticeship a lot in my dissertation — so much so that Carl Berger asked me at my proposal, “Are you testing Cognitive Apprenticeship as a model?” I had no idea how to respond, and 25 years later, I still don’t. How do you test a conceptual framework?
Cognitive apprenticeship, like situated learning, starts from the assumption that apprenticeship is a particularly effective form of education. Then it asks, “How do you offer an apprenticeship around invisible tasks?”
What I like about the essay linked below is that it places cognitive apprenticeship in a broader context. Apprenticeship isn’t always the best option (as discussed in the post about the Herb Simon paper).
Active listeners or readers, who test their understanding and pursue the issues that are raised in their minds, learn things that apprenticeship can never teach. To the degree that readers or listeners are passive, however, they will not learn as much as they would by apprenticeship, because apprenticeship forces them to use their knowledge. Moreover, few people learn to be active readers and listeners on their own, and that is where cognitive apprenticeship is critical–observing the processes by which an expert listener or reader thinks and practicing these skills under the guidance of the expert can teach students to learn on their own more skillfully.
I wonder if this result explains why the second semester students in Briana’s studies (see previous blog post) didn’t have the “W” effect. If you do enough code, you move down the power law of practice, and now you can attend to things like context and generating subgoal labels.
Different subjects start the experiment with different amounts of ability and past experience. Before starting, subjects took a multiple choice test of their knowledge. If we take the results of this test as a proxy for the ability/knowledge at the start of the experiment, then the power law equation becomes (a similar modification can be made to the exponential equation):
That is, the test score is treated as equivalent to performing some number of rounds of implementation). A power law is a better fit than exponential to this data (code+data); the fit captures the general shape, but misses lots of what look like important details.
Balancing cognition and motivation in computing education: Herbert Simon and evidence-based education
Education is a balancing act between optimally efficient instruction and motivating students. It’s not the same thing to meet the needs of the head and of the heart.
Shuchi Grover tweeted this interesting piece (quoted below) that reviews an article by Herb Simon (and John Anderson and Lynne Reder) which I hadn’t previously heard of. The reviewer sees Herb Simon as taking a stand against discovery-based, situated, and constructivist learning, and in favor of direct instruction. When I read the article, I saw a more subtle message. I do recommend reading the review piece linked below.
He [Herbert Simon] rejects discovery learning, and praises teacher instruction
When, for whatever reason, students cannot construct the knowledge for themselves, they need some instruction. The argument that knowledge must be constructed is very similar to the earlier arguments that discovery learning is superior to direct instruction. In point of fact, there is very little positive evidence for discovery learning and it is often inferior (e.g., Charney, Reder & Kusbit, 1990). Discovery learning, even when successful in acquiring the desired construct, may take a great deal of valuable time that could have been spent practicing this construct if it had been instructed. Because most of the learning in discovery learning only takes place after the construct has been found, when the search is lengthy or unsuccessful, motivation commonly flags.
Some cognitive scientists have been railing against the constructivist and situated approaches to learning for years. Probably the most important paper representing the cognitivist perspective is the Kirschner, Sweller, and Clark paper, “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.” I talked about the Kirschner, Sweller, and Clark paper in this blog post with its implication for how we teach computer science.
The conclusion is pretty straightforward: Direct instruction is far more efficient than making the students work it out for themselves. Students struggling to figure something out for themselves does not lead to deeper learning or more transfer than simply telling students what they ought to do. Drill and practice is important. Learning in authentic, complex situations is unnecessary and often undesirable because failure increases with complexity.
The Anderson, Reder, and Simon article does something important that the famous Kirschner, Sweller, and Clark paper doesn’t — it talks about motivation. The words “motivation” and “interests” don’t appear anywhere in the Kirschner, Sweller, and Clark paper. Important attitudes about learning (like Carol Dweck’s fixed and growth mindsets, or Angela Duckworth’s grit) are not even considered.
In contrast, Anderson, Reder, and Simon understand that motivation is a critical part of learning.
Motivational questions lie outside our present discussion, but are at least as complex as the cognitive issues. In particular, there is no simple relation between level of motivation, on the one hand, and the complexity or realism of the context in which the learning takes place, on the other. To cite a simple example, learning by doing in the real-life domain of application is sometimes claimed to be the optimum procedure. Certainly, this is not true, when the tasks are life-threatening for novices (e.g., firefighting), when relevant learning opportunities are infrequent and unpredictable (e.g., learning to fly a plane in bad weather), or when the novice suffers social embarrassment from using inadequate skills in a real-life context (e.g., using a foreign language at a low level of skill). The interaction of motivation with cognition has been described in information-processing terms by Simon (1967, 1994). But an adequate discussion of these issues would call for a separate paper as long as this one.
There are, of course, reasons sometimes to practice skills in their complex setting. Some of the reasons are motivational and some reflect the special skills that are unique to the complex situation. The student who wishes to play violin in an orchestra would have a hard time making progress if all practice were attempted in the orchestra context. On the other hand, if the student never practiced as a member of an orchestra, critical skills unique to the orchestra would not be acquired. The same arguments can be made in the sports context, and motivational arguments can also be made for complex practice in both contexts. A child may not see the point of isolated exercises, but will when they are embedded in the real-world task. Children are motivated to practice sports skills because of the prospect of playing in full-scale games. However, they often spend much more time practicing component skills than full-scale games. It seems important both to motivation and to learning to practice one’s skills from time to time in full context, but this is not a reason to make this the principal mechanism of learning.
As a constructionist-oriented learning scientist, I’d go further with the benefits of a motivating context (which is a subset of what they’re calling a “complex setting”). When you “figure it out for yourself,” you have a different relationship to the domain. You learn about process, as well as content, as in learning what it means to be a scientist or how a programmer thinks. When you are engaged in the context, practice is no longer onerous but an important part of developing expertise — still arduous, but with meaning. Yasmin Kafai and Quinn Burke talk about changing students’ relationship with technology. Computer science shouldn’t just be about learning knowledge, but developing a new sense of empowerment with technology.
I’ve been wondering about what (I think) is an open research question about cognitivist vs. situationist approaches on lifelong learning. I bet you’re more likely to continue learning in a domain when you are a motivated and engaged learner. An efficiently taught but unmotivated learner is less likely to continue learning in the discipline, I conjecture.
While they underestimate the motivational aspect of learning, Anderson, Reder, and Simon are right about the weaknesses of an authentic context. We can’t just throw students into complex situations. Many students will fail, and those that succeed won’t be learning any better. They will learn slower.
Anderson, Reder, and Simon spend much of their paper critiquing Lave & Wenger’s Situated Learning. I draw on situated learning in my work (e.g., see post here) and reference it frequently in my book on Learner-Centered Computing Education, but I agree with their critique. Lave & Wenger are insightful about the motivation part, but miss on the cognitive part. Situated learning, in particular, provides insight into how learning is a process of developing identity. Lave & Wenger value apprenticeship as an educational method too highly. Apprenticeship has lots of weaknesses: inefficient, inequitable, and difficulty to scale.
The motivational component of learning is particularly critical in computing education. Most of our hot issues are issues of motivation:
- Female and under-represented minority students don’t learn differently than the white and Asian males who dominate computing. But they’re much less interested in learning computer science. “Computer science isn’t that difficult but wanting to learn it is.”
- Many students (especially female students) enter computer science class with low self-efficacy (see ICER 2016 award-winning paper). If they don’t believe they can do it, it’s not surprising that they don’t.
- We have too few computer science teachers because it is hard to motivate teachers to teach computer science.
- Students want authentic, complex learning situations, which is why we tend to value CS1 programming languages that are popular in industry. They don’t want practice and abstraction. Anderson, Reder, and Simon point out that more complex learning situations lead to more failure and lagging motivation.
The challenge to being an effective computing educator is to be authentic and complex enough to maintain motivation, and to use scaffolding to support student success and make learning more efficient. That’s the point of Phyllis Blumenfeld et al.’s “Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning.” (I’m in the “et al,” and it’s the most cited paper I’ve ever been part of.) Project-based learning is complex and authentic, but has the weaknesses that the cognitivists describe. Blumenfeld et al. suggest using technology to help students sustain their motivation and support their learning.
Good teaching is not just a matter of choosing the most efficient forms of learning. It’s also about motivating students to persevere, to tell them the benefits that make the efforts worthwhile. It’s about feeding the heart in order to feed the head.
I am posting this on the day that I am honored to “hood” Dr. Briana Morrison. “Hooding” is where doctoral candidates are given their academic regalia indicating their doctorate degree. It’s one of those ancient parts of academia that I find really cool. I like the way that the Wikiversity describes it: “The Hooding Ceremony is symbolic of passing the guard from one generation of doctors to the next generation of doctors.”
I’ve written about Briana’s work a lot over the years here:
- Her proposal is described here, “Cognitive Load as a significant problem in Learning Programming.”
- Her first major dissertation accomplishment was developing (with Dr. Brian Dorn) a measurement instrument for cognitive load.
- One of her bigger wins for her dissertation was showing that subgoal labels work for text languages too (ICER 2015).
- Another really significant result was showing that Parson’s Problems were a more sensitive measure of learning than asking students to write code in an assessment, and that subgoal labels make Parson’s Problems better, too.
- She worked a lot with Lauren Margulieux, so many of the links I listed when Dr. Margulieux defended are also relevant for Dr. Morrison.
- At ICER 2016, she presented a replication study of her first given vs. generated subgoals study.
But what I find most interesting about Briana’s dissertation work were the things that didn’t work:
- She tried to show a difference in getting program instruction via audio or text. She didn’t find one. The research on modality effects suggested that she would.
- She tried to show a difference between loop-and-a-half and exit-in-the-middle WHILE loops. Previous studies had found one. She did not.
These kinds of results are so cool to me, because they point out what we don’t know about computing education yet. The prior results and theory were really clear. The study was well-designed and vetted by her committee. The results were contrary to what we expected. WHAT HAPPENED?!? It’s for the next group of researchers to try to figure out.
The most interesting result of that kind in Briana’s dissertation is one that I’ve written about before, but I’d like to pull it all together here because I think that there are some interesting implications of it. To me, this is a Rainfall Problem kind of question.
Here’s the experimental set-up. We’ve got six groups.
- All groups are learning with pairs of a worked example (a completely worked out piece of code) and then a practice problem (maybe a Parson’s Problem, maybe writing some code). We’ll call these WE-P pairs (Worked Example-Practice). Now, some WE-P pairs have the same context (think of it as the story of a story problem), and some have different contexts. Maybe in the same context, you’re asked to compute the average tips for several days of tips as a barista. Maybe in a different context, you compute tips in the worked example, but you compute the average test score in the practice. In general, we predict that different contexts will be harder for the student than having everything the same.
- So we’ve got same context vs different context as one variable we’re manipulating. The other variable is whether the participants get the worked example with NO subgoal labels, or GENERATED subgoal labels, or the participant has to GENERATE subgoal labels. Think of a subgoal label as a comment that explains some code, but it’s the same comment that will appear in several different programs. It’s meant to encourage the student to abstract the meaning of the code.
In the GENERATE condition, the participants get blanks, to encourage them to abstract for themselves. Typically, we’d expect (for research in other parts of STEM with subgoal labels) that GENERATE would lead to more learning than GIVEN labels, but it’s harder. We might get cognitive overload.
In general, GIVEN labels beats out no labels. No problem — that’s what we expect given all the past work on subgoal labels. But when we consider all six groups, we get this picture.
Why would having the same context do worse with GIVEN labels than no labels? Why would the same context do much better with GENERATE labels, but worse when it’s different contexts?
So, Briana, Lauren, and Adrienne Decker replicated the experiment with Adrienne’s students at RIT (ICER 2016). And they found:
The same strange “W” pattern, where we have this odd interaction between context and GIVEN vs. GENERATE that we just don’t have an explanation for.
But here’s the really intriguing part: they also did the experiment with second semester students at RIT. All the weird interactions disappeared! Same context beat different context. GIVEN labels beat GENERATE labels. No labels do the worst. When students get enough experience, they figure things out and behave like students in other parts of STEM.
The puzzle for the community is WHY. Briana has a hypothesis. Novice students don’t attend to the details that they need, unless you change the contexts. Without changing contexts, students even GIVEN labels don’t learn because they’re not paying enough attention. Changing contexts gets them to think, “What’s going on here?” GENERATE is just too hard for novices — the cognitive load of figuring out the code and generating labels is just overwhelming for students, so they do badly when we’d expect them to do better.
Here we have a theory-conflicting result, that has been replicated in two different populations. It’s like the Rainfall Problem. Nobody expected the Rainfall Problem to be hard, but it was. More and more people tried it with their students, and still, it was hard. It took Kathi Fisler to figure out how to teach CS so that most students could succeed at the Rainfall Problem. What could we teach novice CS students so that they avoid the “W” pattern? Is it just time? Will all second semester students avoid the “W”?
Dr. Morrison gave us a really interesting dissertation — some big wins, and some intriguing puzzles for the next researchers to wrestle with. Briana has now joined the computing education research group at U. Nebraska – Omaha, where I expect to see more great results.