Posts tagged ‘cognitive science’

Elementary School Computer Science – Misconceptions and Developmental Progressions: Papers from SIGCSE 2017

March 8-11, Seattle hosted the ACM SIGCSE Technical Symposium for 2017. This was the largest SIGCSE ever, with over 1500 attendees. I was there and stayed busy (as I described here). This post isn’t a trip report. I want to talk about two of my favorite papers (and one disappointing one) that I’ve read so far.

We are starting to gather evidence on what makes elementary school computer science different than undergraduate computer science. Most of our research on learning programming and computer science is from undergraduates, published in SIGCSE venues. We know relatively little about elementary school students, and it’s obvious that it’s going to be different. But how?

Shuchi Grover and Satabdi Basu of SRI are starting to answer that question in their paper “Measuring Student Learning in Introductory Block-Based Programming: Examining Misconceptions of Loops, Variables, and Boolean Logic.” They looked at the problems that 6th, 7th, and 8th graders had when programming in Scratch. They’re reporting on things that I’ve never heard of before as misconceptions at the undergraduate level. Like this quote:

Students harbored the misconception that a variable is a letter that is used as a short form for an unknown number – an idea that comes from middle school mathematics classes. Together, this led students to believe that repeat(NumberOfTimes) was a new command. One student conjectured it was a command for multiplication by 5 (the value of NumberOfTimes), while another thought it would print each number five times… After being told that NumberOfTimes was indeed a variable, the students could correctly predict the program output, though they continued to take issue with the length of the variable name.

I find their description believable and fascinating. Their paper made me realize that middle school students are expending cognitive load on issues like multi-character variable names that probably no computer scientist even considered. That’s a real problem, but probably fixable — though the fix might be in the mathematics classes, as well as in the CS classes.

The paper that most impressed me was from Diana Franklin’s group, “Using Upper-Elementary Student Performance to Understand Conceptual Sequencing in a Blocks-based Curriculum.” They’re studying over 100 students, and starting to develop general findings about what works at each of these grade levels. Three of their findings are quoted here:

Finding 1: Placing simple instructions in sequence and using simple events in a block-based language is accessible to 4th-6th grade students.

Finding 2: Initialization is challenging for 4th and 5th grade students.

Finding 3: 6th grade students are more precise at 2-dimension navigation than 4th and 5th grade students.

I’ve always suspected that there was likely to be an interaction between a student’s level of cognitive development and what they would likely be able to do in programming, given how much students are learning about abstraction and representation at these ages. Certainly, programming might influence cognitive development. It’s important to figure out what we might expect.

That’s what Diana’s group is doing. She isn’t saying that fourth grader’s can’t initialize variables and properties. She’s saying it’s challenging for them. Her results are likely influenced by Scratch and by how the students were taught — it’s still an important result. Diana’s group is offering a starting point for exploring these interactions and understanding what we can expect to be easy and what might be hard for the average elementary school student at different ages.  There may be studies that also tell us about developmental progressions in countries that are ahead of the US in elementary school CS (e.g., maybe Israel or Germany). This is the first study of its kind that I’ve read.

SIGCSE 2017 introduced having Best Paper awards in multiple categories and Exemplary Paper awards. I applaud these initiatives. Other conferences have these kinds of awards. The awards helps our authors stand out in job searches and promotion time.

To be really meaningful awards, though, SIGCSE has to fix the reviewing processes. There were hiccups in this year’s reviewing where there wasn’t much of a match between reviewer expertise and the paper’s topic. The hiccups led to papers with significant flaws getting high rankings.

The Best Paper award in the Experience Report category was “Making Noise: Using Sound-Art to Explore Technological Fluency.” The authors describe a really nifty idea. They implement a “maker” kind of curriculum. One of the options is that students get toys that make noise then modify and reprogram them. The toys already work, so it’s about understanding a system, then modifying and augmenting it. The class sounds great, but as Leah Buchele has pointed out, “maker” curricula can be overwhelmingly male. I was surprised that this award-winning paper doesn’t mention females or gender — at all. (There is one picture of a female student in the paper.) I understand that it’s an Experience Report, but gender diversity is a critical issue in CS education, particularly with maker curricula. I consider the omission of even a mention of gender to be a significant flaw in the paper.

April 3, 2017 at 7:00 am 9 comments

How the Pioneers of the MOOC Got It Wrong (from IEEE), As Predicted

There is a sense of vindication that the predictions that many of us made about MOOCs have been proven right, e.g., see this blog post where I explicitly argue (as the article below states) that MOOCs misunderstand the importance of active learning. It’s disappointing that so much effort went wasted.  MOOCs do have value, but it’s much more modest than the sales pitch.

What accounts for MOOCs’ modest performance? While the technological solution they devised was novel, most MOOC innovators were unfamiliar with key trends in education. That is, they knew a lot about computers and networks, but they hadn’t really thought through how people learn.

It’s unsurprising then that the first MOOCs merely replicated the standard lecture, an uninspiring teaching style but one with which the computer scientists were most familiar. As the education technology consultant Phil Hill recently observed in the Chronicle of Higher Education, “The big MOOCs mostly employed smooth-functioning but basic video recording of lectures, multiple-choice quizzes, and unruly discussion forums. They were big, but they did not break new ground in pedagogy.”

Indeed, most MOOC founders were unaware that a pedagogical revolution was already under way at the nation’s universities: The traditional lecture was being rejected by many scholars, practitioners, and, most tellingly, tech-savvy students. MOOC advocates also failed to appreciate the existing body of knowledge about learning online, built over the last couple of decades by adventurous faculty who were attracted to online teaching for its innovative potential, such as peer-to-peer learning, virtual teamwork, and interactive exercises. These modes of instruction, known collectively as “active” learning, encourage student engagement, in stark contrast to passive listening in lectures. Indeed, even as the first MOOCs were being unveiled, traditional lectures were on their way out.

Source: How the Pioneers of the MOOC Got It Wrong – IEEE Spectrum

February 17, 2017 at 7:17 am 2 comments

A review of one of my favorite papers: Cognitive Apprenticeship (Collins, Brown, Newman)

I drew on Cognitive Apprenticeship a lot in my dissertation — so much so that Carl Berger asked me at my proposal, “Are you testing Cognitive Apprenticeship as a model?”  I had no idea how to respond, and 25 years later, I still don’t.  How do you test a conceptual framework?

Cognitive apprenticeship, like situated learning, starts from the assumption that apprenticeship is a particularly effective form of education. Then it asks, “How do you offer an apprenticeship around invisible tasks?”

What I like about the essay linked below is that it places cognitive apprenticeship in a broader context.  Apprenticeship isn’t always the best option (as discussed in the post about the Herb Simon paper).

Active listeners or readers, who test their understanding and pursue the issues that are raised in their minds, learn things that apprenticeship can never teach. To the degree that readers or listeners are passive, however, they will not learn as much as they would by apprenticeship, because apprenticeship forces them to use their knowledge. Moreover, few people learn to be active readers and listeners on their own, and that is where cognitive apprenticeship is critical–observing the processes by which an expert listener or reader thinks and practicing these skills under the guidance of the expert can teach students to learn on their own more skillfully.

Source: Cognitive Apprenticeship (Collins, Brown, Newman) | Reading for Pleasure

January 20, 2017 at 7:03 am Leave a comment

Balancing cognition and motivation in computing education: Herbert Simon and evidence-based education

Education is a balancing act between optimally efficient instruction and motivating students. It’s not the same thing to meet the needs of the head and of the heart.

Shuchi Grover tweeted this interesting piece (quoted below) that reviews an article by Herb Simon (and John Anderson and Lynne Reder) which I hadn’t previously heard of.  The reviewer sees Herb Simon as taking a stand against discovery-based, situated, and constructivist learning, and in favor of direct instruction. When I read the article, I saw a more subtle message.  I do recommend reading the review piece linked below.

He [Herbert Simon] rejects discovery learning, and praises teacher instruction

When, for whatever reason, students cannot construct the knowledge for themselves, they need some instruction. The argument that knowledge must be constructed is very similar to the earlier arguments that discovery learning is superior to direct instruction. In point of fact, there is very little positive evidence for discovery learning and it is often inferior (e.g., Charney, Reder & Kusbit, 1990). Discovery learning, even when successful in acquiring the desired construct, may take a great deal of valuable time that could have been spent practicing this construct if it had been instructed. Because most of the learning in discovery learning only takes place after the construct has been found, when the search is lengthy or unsuccessful, motivation commonly flags.

Source: Herbert Simon and evidence-based education | The Wing to Heaven

Some cognitive scientists have been railing against the constructivist and situated approaches to learning for years. Probably the most important paper representing the cognitivist perspective is the Kirschner, Sweller, and Clark paper, “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.”  I talked about the Kirschner, Sweller, and Clark paper in this blog post with its implication for how we teach computer science.

The conclusion is pretty straightforward: Direct instruction is far more efficient than making the students work it out for themselves. Students struggling to figure something out for themselves does not lead to deeper learning or more transfer than simply telling students what they ought to do. Drill and practice is important. Learning in authentic, complex situations is unnecessary and often undesirable because failure increases with complexity.

The Anderson, Reder, and Simon article does something important that the famous Kirschner, Sweller, and Clark paper doesn’t — it talks about motivation. The words “motivation” and “interests” don’t appear anywhere in the Kirschner, Sweller, and Clark paper. Important attitudes about learning (like Carol Dweck’s fixed and growth mindsets, or Angela Duckworth’s grit) are not even considered.

In contrast, Anderson, Reder, and Simon understand that motivation is a critical part of learning.

Motivational questions lie outside our present discussion, but are at least as complex as the cognitive issues. In particular, there is no simple relation between level of motivation, on the one hand, and the complexity or realism of the context in which the learning takes place, on the other. To cite a simple example, learning by doing in the real-life domain of application is sometimes claimed to be the optimum procedure. Certainly, this is not true, when the tasks are life-threatening for novices (e.g., firefighting), when relevant learning opportunities are infrequent and unpredictable (e.g., learning to fly a plane in bad weather), or when the novice suffers social embarrassment from using inadequate skills in a real-life context (e.g., using a foreign language at a low level of skill). The interaction of motivation with cognition has been described in information-processing terms by Simon (1967, 1994). But an adequate discussion of these issues would call for a separate paper as long as this one.

There are, of course, reasons sometimes to practice skills in their complex setting. Some of the reasons are motivational and some reflect the special skills that are unique to the complex situation. The student who wishes to play violin in an orchestra would have a hard time making progress if all practice were attempted in the orchestra context. On the other hand, if the student never practiced as a member of an orchestra, critical skills unique to the orchestra would not be acquired. The same arguments can be made in the sports context, and motivational arguments can also be made for complex practice in both contexts. A child may not see the point of isolated exercises, but will when they are embedded in the real-world task. Children are motivated to practice sports skills because of the prospect of playing in full-scale games. However, they often spend much more time practicing component skills than full-scale games. It seems important both to motivation and to learning to practice one’s skills from time to time in full context, but this is not a reason to make this the principal mechanism of learning.

As a constructionist-oriented learning scientist, I’d go further with the benefits of a motivating context (which is a subset of what they’re calling a “complex setting”). When you “figure it out for yourself,” you have a different relationship to the domain. You learn about process, as well as content, as in learning what it means to be a scientist or how a programmer thinks. When you are engaged in the context, practice is no longer onerous but an important part of developing expertise — still arduous, but with meaning. Yasmin Kafai and Quinn Burke talk about changing students’ relationship with technology. Computer science shouldn’t just be about learning knowledge, but developing a new sense of empowerment with technology.

I’ve been wondering about what (I think) is an open research question about cognitivist vs. situationist approaches on lifelong learning. I bet you’re more likely to continue learning in a domain when you are a motivated and engaged learner. An efficiently taught but unmotivated learner is less likely to continue learning in the discipline, I conjecture.

While they underestimate the motivational aspect of learning, Anderson, Reder, and Simon are right about the weaknesses of an authentic context. We can’t just throw students into complex situations. Many students will fail, and those that succeed won’t be learning any better. They will learn slower.

Anderson, Reder, and Simon spend much of their paper critiquing Lave & Wenger’s Situated Learning. I draw on situated learning in my work (e.g., see post here) and reference it frequently in my book on Learner-Centered Computing Education, but I agree with their critique. Lave & Wenger are insightful about the motivation part, but miss on the cognitive part. Situated learning, in particular, provides insight into how learning is a process of developing identity. Lave & Wenger value apprenticeship as an educational method too highly. Apprenticeship has lots of weaknesses: inefficient, inequitable, and difficulty to scale.

The motivational component of learning is particularly critical in computing education. Most of our hot issues are issues of motivation:

The challenge to being an effective computing educator is to be authentic and complex enough to maintain motivation, and to use scaffolding to support student success and make learning more efficient. That’s the point of Phyllis Blumenfeld et al.’s “Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning.” (I’m in the “et al,” and it’s the most cited paper I’ve ever been part of.) Project-based learning is complex and authentic, but has the weaknesses that the cognitivists describe. Blumenfeld et al. suggest using technology to help students sustain their motivation and support their learning.

Good teaching is not just a matter of choosing the most efficient forms of learning. It’s also about motivating students to persevere, to tell them the benefits that make the efforts worthwhile. It’s about feeding the heart in order to feed the head.

January 6, 2017 at 7:00 am 8 comments

Graduating Dr. Briana Morrison: Posing New Puzzles for Computing Education Research

I am posting this on the day that I am honored to “hood” Dr. Briana Morrison. “Hooding” is where doctoral candidates are given their academic regalia indicating their doctorate degree. It’s one of those ancient parts of academia that I find really cool. I like the way that the Wikiversity describes it: “The Hooding Ceremony is symbolic of passing the guard from one generation of doctors to the next generation of doctors.”

I’ve written about Briana’s work a lot over the years here:

But what I find most interesting about Briana’s dissertation work were the things that didn’t work:

  • She tried to show a difference in getting program instruction via audio or text. She didn’t find one. The research on modality effects suggested that she would.
  • She tried to show a difference between loop-and-a-half and exit-in-the-middle WHILE loops. Previous studies had found one. She did not.

These kinds of results are so cool to me, because they point out what we don’t know about computing education yet. The prior results and theory were really clear. The study was well-designed and vetted by her committee. The results were contrary to what we expected. WHAT HAPPENED?!? It’s for the next group of researchers to try to figure out.

The most interesting result of that kind in Briana’s dissertation is one that I’ve written about before, but I’d like to pull it all together here because I think that there are some interesting implications of it. To me, this is a Rainfall Problem kind of question.

Here’s the experimental set-up. We’ve got six groups.

  1. All groups are learning with pairs of a worked example (a completely worked out piece of code) and then a practice problem (maybe a Parson’s Problem, maybe writing some code). We’ll call these WE-P pairs (Worked Example-Practice). Now, some WE-P pairs have the same context (think of it as the story of a story problem), and some have different contexts. Maybe in the same context, you’re asked to compute the average tips for several days of tips as a barista. Maybe in a different context, you compute tips in the worked example, but you compute the average test score in the practice. In general, we predict that different contexts will be harder for the student than having everything the same.
  2. So we’ve got same context vs different context as one variable we’re manipulating. The other variable is whether the participants get the worked example with NO subgoal labels, or GENERATED subgoal labels, or the participant has to GENERATE subgoal labels. Think of a subgoal label as a comment that explains some code, but it’s the same comment that will appear in several different programs. It’s meant to encourage the student to abstract the meaning of the code.

In the GENERATE condition, the participants get blanks, to encourage them to abstract for themselves. Typically, we’d expect (for research in other parts of STEM with subgoal labels) that GENERATE would lead to more learning than GIVEN labels, but it’s harder. We might get cognitive overload.

In general, GIVEN labels beats out no labels. No problem — that’s what we expect given all the past work on subgoal labels. But when we consider all six groups, we get this picture.

Why would having the same context do worse with GIVEN labels than no labels? Why would the same context do much better with GENERATE labels, but worse when it’s different contexts?

So, Briana, Lauren, and Adrienne Decker replicated the experiment with Adrienne’s students at RIT (ICER 2016). And they found:

The same strange “W” pattern, where we have this odd interaction between context and GIVEN vs. GENERATE that we just don’t have an explanation for.

But here’s the really intriguing part: they also did the experiment with second semester students at RIT. All the weird interactions disappeared! Same context beat different context. GIVEN labels beat GENERATE labels. No labels do the worst. When students get enough experience, they figure things out and behave like students in other parts of STEM.

The puzzle for the community is WHY. Briana has a hypothesis. Novice students don’t attend to the details that they need, unless you change the contexts. Without changing contexts, students even GIVEN labels don’t learn because they’re not paying enough attention. Changing contexts gets them to think, “What’s going on here?” GENERATE is just too hard for novices — the cognitive load of figuring out the code and generating labels is just overwhelming for students, so they do badly when we’d expect them to do better.

Here we have a theory-conflicting result, that has been replicated in two different populations. It’s like the Rainfall Problem. Nobody expected the Rainfall Problem to be hard, but it was. More and more people tried it with their students, and still, it was hard. It took Kathi Fisler to figure out how to teach CS so that most students could succeed at the Rainfall Problem. What could we teach novice CS students so that they avoid the “W” pattern? Is it just time? Will all second semester students avoid the “W”?

Dr. Morrison gave us a really interesting dissertation — some big wins, and some intriguing puzzles for the next researchers to wrestle with. Briana has now joined the computing education research group at U. Nebraska – Omaha, where I expect to see more great results.

December 16, 2016 at 7:00 am 2 comments

Transfer of learning: Making sense of what education research is telling us

I enjoy reading “Gas station without pumps,” and the below-quoted post was one I wanted to respond to.

Two of the popular memes of education researchers, “transferability is an illusion” and “the growth mindset”, are almost in direct opposition, and I don’t know how to reconcile them.

One possibility is that few students actually attempt to learn the general problem-solving skills that math, CS, and engineering design are rich domains for.  Most are content to learn one tiny skill at a time, in complete isolation from other skills and ideas. Students who are particularly good at memory work often choose this route, memorizing pages of trigonometric identities, for example, rather than learning how to derive them at need from a few basics. If students don’t make an attempt to learn transferable skills, then they probably won’t.  This is roughly equivalent to claiming that most students have a fixed mindset with respect to transferable skills, and suggests that transferability is possible, even if it is not currently being learned.

Teaching and testing techniques are often designed to foster an isolation of ideas, focusing on one idea at a time to reduce student confusion. Unfortunately, transferable learning comes not from practice of ideas in isolation, but from learning to retrieve and combine ideas—from doing multi-step problems that are not scaffolded by the teacher.

Source: Transfer of learning | Gas station without pumps

The problem with “transferability” is that it’s an ill-defined term.  Certainly, there is transfer of skill between domains.  Sharon Carver showed a long time ago that she could teach debugging Logo programs, and students would transfer that debugging process to instructions on a map (mentioned in post here).  That’s transferring a skill or a procedure.  We probably do transfer big, high-level heuristics like “divide-and-conquer” or “isolate the problem.”  One issue is whether we can teach them.  John Sweller says that we can’t — we must learn them (it’s a necessary survival skill), but they’re learned from abstracting experience (see Neil Brown’s nice summary of Sweller’s SIGCSE keynote).

Whether we can teach them or not, what we do know is that higher-order thinking is built on lots of content knowledge. Novices are unlikely to transfer until they know a lot of stuff, a lot of examples, a lot of situations. For example, novice designers often have “design fixation.”  They decide that the first thing they think of must be the right answer.  We can insist that novice designers generate more designs, but they’re not going to generate more good designs until they know more designs.  Transfer happens pretty easily when you know a lot of content and have seen a lot of situations, and you recognize that one situation is actually like another.

Everybody starts out learning one tiny skill at a time.  If you know a lot of skills (maybe because you have lots of prior experience, maybe because you have thought about these skills a lot and have recognized the general principles), you can start chunking these skills and learning whole schema and higher-level skills.  But you can’t do that until you know lots of skills.  Students who want to learn one tiny skill at a time may actually need to still learn one tiny skill at a time. People abstract (e.g., able to derive a solution rather than memorize it) when they know enough content that it’s useful and possible for them to abstract over it.  I completely agree that students have to try to abstract.  They have to learn a lot of stuff, and then they have to be in a situation where it’s useful for them to abstract.

“Growth mindset” is a necessity for any of this to work.  Students have to believe that content is worth knowing and that they can learn it.  If students believe that content is useless, or that they just “don’t do math” or “am not a computer person” (both of which I’ve heard in just the last week), they are unlikely to learn content, they are unlikely to see patterns in it, and they are unlikely to abstract over it.

Kevin is probably right that we don’t teach problem solving in engineering or computing well.  I blogged on this theme for CACM last month — laboratory experiments work better for a wider range students than classroom studies.  Maybe we teach better in labs than in classrooms?  The worked examples effect suggests that we may be asking students to problem solve too much.  We should show students more completely worked out problems.  As Sweller said at SIGCSE, we can’t expect students to solve novel problems.  We have to expect students to match new problems to solutions that they have already seen.  We do want students to solve problems, too, and not just review example solutions. Trafton and Reiser showed that these should be interleaved: Example, Problem, Example, Problem… (see this page for a summary of some of the worked examples research, including Trafton & Reiser).

When I used to do Engineering Education research, one of my largest projects was a complete flop.  We had all this prior work showing the benefits of a particular collaborative learning technology and technique, then we took it into the engineering classroom and…poof! Nothing happened.  In response, we started a project to figure out why it failed so badly.  One of our findings was that “learned helplessness” was rampant in our classes, which is a symptom of a fixed mindset.  “I know that I’m wrong, and there’s nothing that I can do about it.  Collaboration just puts my errors on display for everyone,” was the kind of response we’ve got. (See here for one of our papers on this work.)

I believe that all the things Kevin sees going wrong in his classes really are happening.  I believe he’s not seeing transfer that he might reasonably expect to see.  I believe that he doesn’t see students trying to abstract across lower-level skills.  But I suspect that the problem is the lack of a growth mindset.  In our work, we saw Engineering students simply give up.  They felt like they couldn’t learn, they couldn’t keep up, so they just memorized.  I don’t know that that’s the cause of the problems that Kevin is seeing.  In my work, I’ve often found that motivation and incentive are key to engagement and learning.

April 25, 2016 at 7:33 am 1 comment

Moving Beyond MOOCS: Could we move to understanding learning and teaching?

We’re years into the MOOC phenomenon, and I’d hoped that we’d get past MOOC hype. But we’re not.  The article below shows the same misunderstandings of learning and teaching that we heard at the start — misunderstandings that even MOOC supporters (like here and here) have stopped espousing.

The value of being in the front row of a class is that you talk with the teacher.  Getting physically closer to the lecturer doesn’t improve learning.  Engagement improves learning.  A MOOC puts everyone at the back of the class, listening only and doing the homework.

In many ways, we have a romanticized view of college. Popular portrayals of a typical classroom show a handful of engaged students sitting attentively around a small seminar table while their Harrison Ford-like professor shares their wisdom about the world. We all know the real classroom is very different. Especially in big introductory classes — American history, U.S. government, human psychology, etc. — hundreds of disinterested, and often distracted, students cram into large impersonal lecture halls, passively taking notes, occasionally glancing up at the clock waiting for the class to end. And it’s no more engaging for the professor. Usually we can’t tell whether students are taking notes or updating their Facebook page. For me, everything past the ninth row was distance learning. A good online platform puts every student in the front row.

via Moving Beyond MOOCS | Steven M. Gillon.

June 5, 2015 at 7:14 am 9 comments

Older Posts


Recent Posts

October 2017
M T W T F S S
« Sep    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Feeds

Blog Stats

  • 1,440,481 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 5,159 other followers

CS Teaching Tips