Posts tagged ‘learning sciences’

Balancing cognition and motivation in computing education: Herbert Simon and evidence-based education

Education is a balancing act between optimally efficient instruction and motivating students. It’s not the same thing to meet the needs of the head and of the heart.

Shuchi Grover tweeted this interesting piece (quoted below) that reviews an article by Herb Simon (and John Anderson and Lynne Reder) which I hadn’t previously heard of.  The reviewer sees Herb Simon as taking a stand against discovery-based, situated, and constructivist learning, and in favor of direct instruction. When I read the article, I saw a more subtle message.  I do recommend reading the review piece linked below.

He [Herbert Simon] rejects discovery learning, and praises teacher instruction

When, for whatever reason, students cannot construct the knowledge for themselves, they need some instruction. The argument that knowledge must be constructed is very similar to the earlier arguments that discovery learning is superior to direct instruction. In point of fact, there is very little positive evidence for discovery learning and it is often inferior (e.g., Charney, Reder & Kusbit, 1990). Discovery learning, even when successful in acquiring the desired construct, may take a great deal of valuable time that could have been spent practicing this construct if it had been instructed. Because most of the learning in discovery learning only takes place after the construct has been found, when the search is lengthy or unsuccessful, motivation commonly flags.

Source: Herbert Simon and evidence-based education | The Wing to Heaven

Some cognitive scientists have been railing against the constructivist and situated approaches to learning for years. Probably the most important paper representing the cognitivist perspective is the Kirschner, Sweller, and Clark paper, “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.”  I talked about the Kirschner, Sweller, and Clark paper in this blog post with its implication for how we teach computer science.

The conclusion is pretty straightforward: Direct instruction is far more efficient than making the students work it out for themselves. Students struggling to figure something out for themselves does not lead to deeper learning or more transfer than simply telling students what they ought to do. Drill and practice is important. Learning in authentic, complex situations is unnecessary and often undesirable because failure increases with complexity.

The Anderson, Reder, and Simon article does something important that the famous Kirschner, Sweller, and Clark paper doesn’t — it talks about motivation. The words “motivation” and “interests” don’t appear anywhere in the Kirschner, Sweller, and Clark paper. Important attitudes about learning (like Carol Dweck’s fixed and growth mindsets, or Angela Duckworth’s grit) are not even considered.

In contrast, Anderson, Reder, and Simon understand that motivation is a critical part of learning.

Motivational questions lie outside our present discussion, but are at least as complex as the cognitive issues. In particular, there is no simple relation between level of motivation, on the one hand, and the complexity or realism of the context in which the learning takes place, on the other. To cite a simple example, learning by doing in the real-life domain of application is sometimes claimed to be the optimum procedure. Certainly, this is not true, when the tasks are life-threatening for novices (e.g., firefighting), when relevant learning opportunities are infrequent and unpredictable (e.g., learning to fly a plane in bad weather), or when the novice suffers social embarrassment from using inadequate skills in a real-life context (e.g., using a foreign language at a low level of skill). The interaction of motivation with cognition has been described in information-processing terms by Simon (1967, 1994). But an adequate discussion of these issues would call for a separate paper as long as this one.

There are, of course, reasons sometimes to practice skills in their complex setting. Some of the reasons are motivational and some reflect the special skills that are unique to the complex situation. The student who wishes to play violin in an orchestra would have a hard time making progress if all practice were attempted in the orchestra context. On the other hand, if the student never practiced as a member of an orchestra, critical skills unique to the orchestra would not be acquired. The same arguments can be made in the sports context, and motivational arguments can also be made for complex practice in both contexts. A child may not see the point of isolated exercises, but will when they are embedded in the real-world task. Children are motivated to practice sports skills because of the prospect of playing in full-scale games. However, they often spend much more time practicing component skills than full-scale games. It seems important both to motivation and to learning to practice one’s skills from time to time in full context, but this is not a reason to make this the principal mechanism of learning.

As a constructionist-oriented learning scientist, I’d go further with the benefits of a motivating context (which is a subset of what they’re calling a “complex setting”). When you “figure it out for yourself,” you have a different relationship to the domain. You learn about process, as well as content, as in learning what it means to be a scientist or how a programmer thinks. When you are engaged in the context, practice is no longer onerous but an important part of developing expertise — still arduous, but with meaning. Yasmin Kafai and Quinn Burke talk about changing students’ relationship with technology. Computer science shouldn’t just be about learning knowledge, but developing a new sense of empowerment with technology.

I’ve been wondering about what (I think) is an open research question about cognitivist vs. situationist approaches on lifelong learning. I bet you’re more likely to continue learning in a domain when you are a motivated and engaged learner. An efficiently taught but unmotivated learner is less likely to continue learning in the discipline, I conjecture.

While they underestimate the motivational aspect of learning, Anderson, Reder, and Simon are right about the weaknesses of an authentic context. We can’t just throw students into complex situations. Many students will fail, and those that succeed won’t be learning any better. They will learn slower.

Anderson, Reder, and Simon spend much of their paper critiquing Lave & Wenger’s Situated Learning. I draw on situated learning in my work (e.g., see post here) and reference it frequently in my book on Learner-Centered Computing Education, but I agree with their critique. Lave & Wenger are insightful about the motivation part, but miss on the cognitive part. Situated learning, in particular, provides insight into how learning is a process of developing identity. Lave & Wenger value apprenticeship as an educational method too highly. Apprenticeship has lots of weaknesses: inefficient, inequitable, and difficulty to scale.

The motivational component of learning is particularly critical in computing education. Most of our hot issues are issues of motivation:

The challenge to being an effective computing educator is to be authentic and complex enough to maintain motivation, and to use scaffolding to support student success and make learning more efficient. That’s the point of Phyllis Blumenfeld et al.’s “Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning.” (I’m in the “et al,” and it’s the most cited paper I’ve ever been part of.) Project-based learning is complex and authentic, but has the weaknesses that the cognitivists describe. Blumenfeld et al. suggest using technology to help students sustain their motivation and support their learning.

Good teaching is not just a matter of choosing the most efficient forms of learning. It’s also about motivating students to persevere, to tell them the benefits that make the efforts worthwhile. It’s about feeding the heart in order to feed the head.

January 6, 2017 at 7:00 am 5 comments

Graduating Dr. Briana Morrison: Posing New Puzzles for Computing Education Research

I am posting this on the day that I am honored to “hood” Dr. Briana Morrison. “Hooding” is where doctoral candidates are given their academic regalia indicating their doctorate degree. It’s one of those ancient parts of academia that I find really cool. I like the way that the Wikiversity describes it: “The Hooding Ceremony is symbolic of passing the guard from one generation of doctors to the next generation of doctors.”

I’ve written about Briana’s work a lot over the years here:

But what I find most interesting about Briana’s dissertation work were the things that didn’t work:

  • She tried to show a difference in getting program instruction via audio or text. She didn’t find one. The research on modality effects suggested that she would.
  • She tried to show a difference between loop-and-a-half and exit-in-the-middle WHILE loops. Previous studies had found one. She did not.

These kinds of results are so cool to me, because they point out what we don’t know about computing education yet. The prior results and theory were really clear. The study was well-designed and vetted by her committee. The results were contrary to what we expected. WHAT HAPPENED?!? It’s for the next group of researchers to try to figure out.

The most interesting result of that kind in Briana’s dissertation is one that I’ve written about before, but I’d like to pull it all together here because I think that there are some interesting implications of it. To me, this is a Rainfall Problem kind of question.

Here’s the experimental set-up. We’ve got six groups.

  1. All groups are learning with pairs of a worked example (a completely worked out piece of code) and then a practice problem (maybe a Parson’s Problem, maybe writing some code). We’ll call these WE-P pairs (Worked Example-Practice). Now, some WE-P pairs have the same context (think of it as the story of a story problem), and some have different contexts. Maybe in the same context, you’re asked to compute the average tips for several days of tips as a barista. Maybe in a different context, you compute tips in the worked example, but you compute the average test score in the practice. In general, we predict that different contexts will be harder for the student than having everything the same.
  2. So we’ve got same context vs different context as one variable we’re manipulating. The other variable is whether the participants get the worked example with NO subgoal labels, or GENERATED subgoal labels, or the participant has to GENERATE subgoal labels. Think of a subgoal label as a comment that explains some code, but it’s the same comment that will appear in several different programs. It’s meant to encourage the student to abstract the meaning of the code.

In the GENERATE condition, the participants get blanks, to encourage them to abstract for themselves. Typically, we’d expect (for research in other parts of STEM with subgoal labels) that GENERATE would lead to more learning than GIVEN labels, but it’s harder. We might get cognitive overload.

In general, GIVEN labels beats out no labels. No problem — that’s what we expect given all the past work on subgoal labels. But when we consider all six groups, we get this picture.

Why would having the same context do worse with GIVEN labels than no labels? Why would the same context do much better with GENERATE labels, but worse when it’s different contexts?

So, Briana, Lauren, and Adrienne Decker replicated the experiment with Adrienne’s students at RIT (ICER 2016). And they found:

The same strange “W” pattern, where we have this odd interaction between context and GIVEN vs. GENERATE that we just don’t have an explanation for.

But here’s the really intriguing part: they also did the experiment with second semester students at RIT. All the weird interactions disappeared! Same context beat different context. GIVEN labels beat GENERATE labels. No labels do the worst. When students get enough experience, they figure things out and behave like students in other parts of STEM.

The puzzle for the community is WHY. Briana has a hypothesis. Novice students don’t attend to the details that they need, unless you change the contexts. Without changing contexts, students even GIVEN labels don’t learn because they’re not paying enough attention. Changing contexts gets them to think, “What’s going on here?” GENERATE is just too hard for novices — the cognitive load of figuring out the code and generating labels is just overwhelming for students, so they do badly when we’d expect them to do better.

Here we have a theory-conflicting result, that has been replicated in two different populations. It’s like the Rainfall Problem. Nobody expected the Rainfall Problem to be hard, but it was. More and more people tried it with their students, and still, it was hard. It took Kathi Fisler to figure out how to teach CS so that most students could succeed at the Rainfall Problem. What could we teach novice CS students so that they avoid the “W” pattern? Is it just time? Will all second semester students avoid the “W”?

Dr. Morrison gave us a really interesting dissertation — some big wins, and some intriguing puzzles for the next researchers to wrestle with. Briana has now joined the computing education research group at U. Nebraska – Omaha, where I expect to see more great results.

December 16, 2016 at 7:00 am Leave a comment

Learning Curves, Given vs Generated Subgoal Labels, Replicating a US study in India, and Frames vs Text: More ICER 2016 Trip Reports

My Blog@CACM post for this month is a trip report on ICER 2016. I recommend Andy Ko’s excellent ICER 2016 trip report for another take on the conference. You can also see the Twitter live feed with hashtag #ICER2016.

I write in the Blog@CACM post about three papers (and reference two others), but I could easily write reports on a dozen more. The findings were that interesting and that well done. I’m going to give four more mini-summaries here, where the results are more confusing or surprising than those I included in the CACM Blog post.

This year was the first time we had a neck-and-neck race for the attendee-selected award, the “John Henry” award. The runner-up was Learning Curve Analysis for Programming: Which Concepts do Students Struggle With? by Kelly Rivers, Erik Harpstead, and Ken Koedinger. Tutoring systems can be used to track errors on knowledge concepts over multiple practice problems. Tutoring systems developers can show these lovely decreasing error curves as students get more practice, which clearly demonstrate learning. Kelly wanted to see if she could do that with open editing of code, not in a tutoring system. She tried to use AST graphs as a sense of programming “concepts,” and measure errors in use of the various constructs. It didn’t work, as Kelly explains in her paper. It was a nice example of an interesting and promising idea that didn’t pan out, but with careful explanation for the next try.

I mentioned in this blog previously that Briana Morrison and Lauren Margulieux had a replication study (see paper here), written with Adrienne Decker using participants from Adrienne’s institution. I hadn’t read the paper when I wrote that first blog post, and I was amazed by their results. Recall that they had this unexpected result where changing contexts for subgoal labeling worked better (i.e., led to better performance) for students than keeping students in the same context. The weird contextual-transfer problems that they’d seen previously went away in the second (follow-on) CS class — see below snap from their slides. The weird result was replicated in the first class at this new institution, so we know it’s not just one strange student population, and now we know that it’s a novice problem. That’s fascinating, but still doesn’t really explain why. Even more interesting was that when the context transfer issues go away, students did better when they were given subgoal labels than when they generated them. That’s not what happens in other fields. Why is CS different? It’s such an interesting trail that they’re exploring!

img_3874

Mike Hewner and Shitanshu Mishra replicated Mike’s dissertation study about how students choose CS as a major, but in Indian institutions rather than in US institutions: When Everyone Knows CS is the Best Major: Decisions about CS in an Indian context. The results that came out of the Grounded Theory analysis were quite different! Mike had found that US students use enjoyment as a proxy for ability — “If I like CS, I must be good at it, so I’ll major in that.” But Indian students already thought CS was the best major. The social pressures were completely different. So, Indian students chose CS — if they had no other plans. CS was the default behavior.

One of the more surprising results was from Thomas W. Price, Neil C.C. Brown, Dragan Lipovac, Tiffany Barnes, and Michael Kölling, Evaluation of a Frame-based Programming Editor. They asked a group of middle school students in a short laboratory study (not the most optimal choice, but an acceptable starting place) to program in Java or in Stride, the new frame-based language and editing environment from the BlueJ/Greenfoot team.  They found no statistically significant differences between the two different languages, in terms of number of objectives completed, student frustration/satisfaction, or amount of time spent on the tasks. Yes, Java students got more syntax errors, but it didn’t seem to have a significant impact on performance or satisfaction. I found that totally unexpected. This is a result that cries out for more exploration and explanation.

There’s a lot more I could say, from Colleen Lewis’s terrific ideas to reduce the impact of CS stereotypes to a promising new method of expert heuristic evaluation of cognitive load.  I recommend reviewing the papers while they’re still free to download.

September 16, 2016 at 7:07 am 4 comments

Preview ICER 2016: Ebooks Design-Based Research and Replications in Assessment and Cognitive Load Studies

The International Computing Education Research (ICER) Conference 2016 is September 8-12 in Melbourne, Australia (see website here). There were 102 papers submitted, and 26 papers accepted for a 25% acceptance rate. Georgia Tech computing education researchers are justifiably proud — we submitted three papers to ICER 2016, and we had three acceptances. We’re over 10% of all papers at ICER 2016.

One of the papers extends the ebook work that I’ve reported on here (see here where we made them available and our paper on usability and usage from WiPSCE 2015). Identifying Design Principles for CS Teacher Ebooks through Design-Based Research (click on the title to get to the ACM DL page) by Barbara Ericson, Kantwon Rogers, Miranda Parker, Briana Morrison, and I use a Design-Based Research perspective on our ebooks work. We describe our theory for the ebooks, then describe the iterations of what we designed, what happened when we deployed (data-driven), and how we then re-designed.

Two of our papers are replication studies — so grateful to the ICER reviewers and communities for seeing the value of replication studies. The first is Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment by Miranda Parker, me, and Shelly Engleman. This is Miranda’s paper expanding on her SIGCSE 2016 poster introducing the SCS1 validated and language-independent measure of CS1 knowledge. The paper does a great survey of validated measures of learning, explains her process, and then presents what one can and can’t claim with a validated instrument.

The second is Learning Loops: A Replication Study Illuminates Impact of HS Courses by Briana Morrison, Adrienne Decker, and Lauren Margulieux. Briana and Lauren have both now left Georgia Tech, but they were still here when they did this paper, so we’re claiming them. Readers of this blog may recall Briana and Lauren’s confusing results from SIGCSE 2016 result that suggest that cognitive load in CS textual programming is so high that it blows away our experimental instructional treatments. Was that an aberration? With Adrienne Decker’s help (and student participants), they replicated the study. I’ll give away the bottom line: It wasn’t an aberration. One new finding is that students who did not have high school CS classes caught up with those who did in the experiment, with respect to understanding loops

We’re sending three of our Human-Centered Computing PhD students to the ICER 2016 Doctoral Consortium. These folks will be in the DC on Sept 8, and will present posters to the conference on Sept 9 afternoon.

September 2, 2016 at 7:53 am 14 comments

Seeking Collaborators for a Study of Achievement Goal Theory in CS1: Guest blog post by Daniel Zingaro

I have talked about Dan’s work here before, such as his 2014 award-winning ICER paper and his Peer Instruction in CS website. I met with Dan at the last SIGCSE where he told me about the study that he and Leo Porter were planning. Their results are fascinating since they are counter to what Achievement Goal Theory predicts. I invited him to write a guest blog post to seek collaborators for his study, and am grateful that he sent me this.

Why might we apply educational theory to our study of novice programmers? One core reason lies in theory-building: if someone has developed a general learning theory, then we might do well to co-opt and extend it for the computing context. What we get for free is clear: a theoretical basis, perhaps with associated experimental procedures, scales, hypotheses, and predictions. Unfortunately, however, there is often a cost in appropriating this theory: it may not replicate for us in the expected ways.

Briana Morrison’s recent work nicely highlights this point. In two studies, Briana reports her efforts to replicate what is known about subgoals and worked examples. Briefly, a worked example is a sample problem whose step-by-step solution is given to students. And subgoals are used to break that solution into logical chunks to hopefully help students map out the ways that the steps fit together to solve the problem.

Do subgoals help? Well, it’s supposed to go like this, from the educational psychology literature: having students generate their own labeled goals is best, giving students the subgoal labels is worse, and not using subgoals at all is worse still. But that isn’t what Briana found. For example, Briana reports [1] that, on Parsons puzzles, students who are given subgoal labels do better than both those who generate their own subgoal labels and those not given subgoals at all. Why the differences? One possibility is that programming exerts considerable cognitive load on the learner, and that the additional load incurred by generating subgoal labels overloads the student and harms learning.

The point here is that taking seriously the idea of leveraging existing theory requires concomitant attention to how and why the theory may operate differently in computing.

My particular interest here is in another theory from educational psychology: achievement goal theory (AGT). AGT studies the goals that students adopt in achievement situations, and the positive and negative consequences of those goals in terms of educationally-relevant outcomes. AGT zones in on two main goal types: mastery goals (where performance is defined intrapersonally) and performance goals (where performance is defined normatively in comparison to others).

Do these goals matter? Well, it’s supposed to go roughly like this: mastery goals are positively associated with many outcomes of value, such as interest, enjoyment, self-efficacy, and deep study strategies (but not academic performance); performance goals, surprisingly and confusingly, are positively associated with academic performance. But, paralleling the Briana studies from above, this isn’t what we’ve found in CS. With Leo Porter and my students, we’ve been studying goal-outcome links in novice CS students. We’ve found, contrary to theoretical expectations, that performance goals appear to be null or negative predictors of performance, and that mastery goals appear to be positive predictors of performance [2,3].

We are now conducting a larger study of achievement goals and outcomes of CS1 students — larger than that achievable with the couple of institutions to which we have access on our own. We are asking for your help.

The study involves administering two surveys to students in a CS1 course. The first survey, at the beginning of the semester, measures student achievement goals. The second survey, close to the end of the semester, measures potential mediating variables. We plan to collect exam grade, interest in CS, and other outcome variables.

The hope is that we can conduct a multi-institutional study of a variety of CS1 courses to strengthen what we know about achievement goals in CS.

Please contact me at daniel dot zingaro at utoronto dot ca if you are interested in participating in this work. Thanks!

[1] Briana Morrison. Subgoals Help Students Solve Parsons Problems. SIGCSE, 2016. ACM DL link.

[2] Daniel Zingaro. Examining Interest and Performance in Computer Science 1: A Study of Pedagogy and Achievement Goals. TOCE, 2015. ACM DL link.

[3] Daniel Zingaro and Leo Porter. Impact of Student Achievement Goals on CS1 Outcomes. SIGCSE, 2016. ACM DL link.

July 15, 2016 at 7:30 am Leave a comment

Are there elements of human nature that could be better harnessed for better educational outcomes?

I don’t often link to Quora, but when it’s Steven Pinker pointing out the relationship between our human nature to educational goals, it’s worth it.

One potential insight is that educators begin not with blank slates but with minds that are adapted to think and reason in ways that may be at cross-purposes with the goals of education in a modern society. The conscious portion of language consists of words and meanings, but the portion that connects most directly to print consists of phonemes, which ordinarily are below the level of consciousness. We intuitively understand living species as having essences, but the theory of evolution requires us to rethink them as populations of variable individuals. We naturally assess probability by dredging up examples from memory, whereas real probability takes into account the number of occurrences and the number of opportunities. We are apt to think that people who disagree with us are stupid and stubborn, while we are overconfident and self-deluded about our own competence and honesty.

Source: (3) Are there elements of human nature that could be better harnessed for better educational outcomes? – Quora

July 13, 2016 at 7:57 am 2 comments

Motivating STEM Engagement in Children, Families, and Communities

I’ve known Dan Hickey for many years, and got to spend some time with him at Indiana when I visited there a couple years ago.  He’s dealing with an issue in this blog post that is critical to CS Education.  If we want students to value computing, it has to be valued and promoted in their families and communities.  How do we get engagement at a beyond-school level in computing education?

These issues of trajectories and non-participation in STEM learning have personal relevance for me and my own family. I was quite pleased a few years ago when my son Lucas enrolled in a computer programming class in high school. I never learned to program myself and these days it I find it quite a handicap. While I bought an Apple II+ computer in 1982 (!) and taught myself BASIC, an instructional technology professor discouraged me from delving too deeply into technology or programming (because “it changes too often”). While I still want to learn how to code, my non-participation in programming clearly helped define my trajectory towards a Ph.D in Psychology and satisfying career as a Learning Scientist.Unfortunately, the curriculum in my son’s programming class was like the typical secondary computer science instruction that Mark Guzdial chronicles in his Computing Education blog. The coding worksheets seemed to have been haphazardly created to match various videos located on the web. My son wanted to use the much more professional videos and exercises that we were able to access via my university’s account at Lynda.com, but his teacher insisted that my son complete the worksheets as well (so teacher could grade them).

Source: re-mediating assessment: Motivating STEM Engagement in Children, Families, and Communities

May 27, 2016 at 8:04 am Leave a comment

Older Posts


Recent Posts

January 2017
M T W T F S S
« Dec    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Feeds

Blog Stats

  • 1,302,392 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 4,608 other followers

CS Teaching Tips