Posts tagged ‘cognitive load’

Announcing Barbara Ericson’s Defense on Effectiveness and Efficiency of Parsons Problems and Dynamically Adaptive Parsons Problems: Next stop, University of Michigan

Today, Barbara Ericson defends her dissertation. I usually do a blog post talking about the defending student’s work as I’ve blogged about it in the past, but that’s really hard with Barb.  I’ve written over 90 blog posts referencing Barb in the last 9 years.  That happens when we have been married for 32 years and collaborators on CS education work for some 15 years.

Barb did her dissertation on adaptive Parsons problems, but she could have done it on Project Rise Up or some deeper analysis of her years of AP CS analyses. She chose well. Her results are fantastic, and summarized below. (Yes, she does have six committee members, including two external members.)

Starting September 1, Barbara and I will be faculty at the University of Michigan. Barb will be an assistant professor in the University of Michigan School of Information (UMSI). I will be a professor in the Computer Science and Engineering (CSE) Division of the Electrical Engineering and Computer Science Department, jointly with their new Engineering Education Research program. Moving from Georgia Tech and Atlanta will be hard — all three of our children will still be here as we leave. We are excited about the opportunities and new colleagues that we will have in Ann Arbor.

Title: Evaluating the Effectiveness and Efficiency of Parsons Problems and Dynamically Adaptive Parsons Problems as a Type of Low Cognitive Load Practice Problem

Barbara J. Ericson

Human-Centered Computing

School of Interactive Computing

College of Computing

Georgia Institute of Technology

Date: Monday, March 12, 2018

Time: 12pm – 3pm

Location: TSRB 222

Committee:

Dr. Jim Foley (Advisor, School of Interactive Computing, Georgia Institute of Technology)

Dr. Amy Bruckman (School of Interactive Computing, Georgia Institute of Technology)

Dr. Ashok K. Goel (School of Interactive Computing, Georgia Institute of Technology)

Dr. Richard Catrambone (School of Psychology, Georgia Institute of Technology)

Dr. Alan Kay (Computer Science Department, University of California, Los Angeles)

Dr. Mitchel Resnick (Media Laboratory, Massachusetts Institute of Technology)

Abstract:

Learning to program can be difficult and time consuming.  Learners can spend hours trying to figure out why their program doesn’t compile or run correctly. Many countries, including the United States, want to train thousands of secondary teachers to teach programming.  However, busy in-service teachers do not have hours to waste on compiler errors or debugging.  They need a more efficient way to learn.

One way to reduce learning time is to use a completion task.  Parsons problems are a type of code completion problem in which the learner must place blocks of correct, but mixed up, code in the correct order. Parsons problems can also have distractor blocks, which are not needed in a correct solution.  Distractor blocks include common syntax errors like a missing colon on a for loop or semantic errors like the wrong condition on a loop.

In this dissertation, I conducted three studies to compare the efficiency and effectiveness of solving Parsons problems, fixing code, and writing code. (Editor’s note: I blogged on her first study here.) I also tested two forms of adaptation. For the second study, I added intra-problem adaptation, which dynamically makes the current problem easier.  For the last study, I added inter-problem adaptation which makes the next problem easier or harder depending on the learner’s performance.  The studies provided evidence that students can complete Parsons problems significantly faster than fixing or writing code while achieving the same learning gains from pretest to posttest.  The studies also provided evidence that adaptation helped more learners successfully solve Parsons problems.

These studies were the first to empirically test the efficiency and effectiveness of solving Parsons problems versus fixing and writing code.  They were also the first to explore the impact of both intra-problem and inter-problem adaptive Parsons problems.  Finding a more efficient and just as effective form of practice could reduce the frustration that many novices feel when learning programming and help prepare thousands of secondary teachers to teach introductory computing courses.

March 12, 2018 at 7:00 am 15 comments

What does it mean for Computer Science to be harder to learn than other STEM subjects?

I made an argument in my Blog@CACM Post for this month that “Learning Computer Science is Different than Learning Other STEM Disciplines,” and on Twitter, I explicitly added “It’s harder.”

In my Blog@CACM post, I thought it was a no-brainer that CS is harder:

  1. Our infrastructure for teaching CS is younger, smaller, and weaker  (CS is so new, and we don’t have the decades of experience to figure out how to do it well yet.)

  2. We don’t realize how hard learning to program is (The fact that the Rainfall problem seems easy, but it’s clearly not easy, means that CS teachers don’t know how to estimate yet what’s hard for students, so our classes are probably harder than we mean them to be.)

  3. CS is so valuable that it changes the affective components of learning (Classes that are stuffed full of both CS majors and non-majors means that issues of self-efficacy, motivation, and belonging are much bigger in CS than in other STEM disciplines.)

The push back was really interesting.  People pointed out that they took CS classes and math classes, or CS and physics, and CS seemed easy in comparison.  They may be right, but that’s self-report on introspection by people who succeeded at both classes.  My point is that we are probably flunking out (or students are giving up, or opting out) of CS at much higher rates than any other STEM subject, because of the reasons I give.  We’re really using two different measures of “harder” — harder to succeed, or harder in retrospect once succeeded.

I only have a qualitative argument for “It’s harder.” I’m not sure how one would even evaluate the point empirically.  Any suggestions?  How could we measure when one subject is harder than another?

It’s not an important question to answer which is harder, CS vs math, or CS vs physics. A much more important and supportable claim is that CS “is harder” than it needs to be.  We have a lot of extraneous complexity and cognitive load in learning CS.

January 19, 2018 at 7:00 am 18 comments

Elementary School Computer Science – Misconceptions and Developmental Progressions: Papers from SIGCSE 2017

March 8-11, Seattle hosted the ACM SIGCSE Technical Symposium for 2017. This was the largest SIGCSE ever, with over 1500 attendees. I was there and stayed busy (as I described here). This post isn’t a trip report. I want to talk about two of my favorite papers (and one disappointing one) that I’ve read so far.

We are starting to gather evidence on what makes elementary school computer science different than undergraduate computer science. Most of our research on learning programming and computer science is from undergraduates, published in SIGCSE venues. We know relatively little about elementary school students, and it’s obvious that it’s going to be different. But how?

Shuchi Grover and Satabdi Basu of SRI are starting to answer that question in their paper “Measuring Student Learning in Introductory Block-Based Programming: Examining Misconceptions of Loops, Variables, and Boolean Logic.” They looked at the problems that 6th, 7th, and 8th graders had when programming in Scratch. They’re reporting on things that I’ve never heard of before as misconceptions at the undergraduate level. Like this quote:

Students harbored the misconception that a variable is a letter that is used as a short form for an unknown number – an idea that comes from middle school mathematics classes. Together, this led students to believe that repeat(NumberOfTimes) was a new command. One student conjectured it was a command for multiplication by 5 (the value of NumberOfTimes), while another thought it would print each number five times… After being told that NumberOfTimes was indeed a variable, the students could correctly predict the program output, though they continued to take issue with the length of the variable name.

I find their description believable and fascinating. Their paper made me realize that middle school students are expending cognitive load on issues like multi-character variable names that probably no computer scientist even considered. That’s a real problem, but probably fixable — though the fix might be in the mathematics classes, as well as in the CS classes.

The paper that most impressed me was from Diana Franklin’s group, “Using Upper-Elementary Student Performance to Understand Conceptual Sequencing in a Blocks-based Curriculum.” They’re studying over 100 students, and starting to develop general findings about what works at each of these grade levels. Three of their findings are quoted here:

Finding 1: Placing simple instructions in sequence and using simple events in a block-based language is accessible to 4th-6th grade students.

Finding 2: Initialization is challenging for 4th and 5th grade students.

Finding 3: 6th grade students are more precise at 2-dimension navigation than 4th and 5th grade students.

I’ve always suspected that there was likely to be an interaction between a student’s level of cognitive development and what they would likely be able to do in programming, given how much students are learning about abstraction and representation at these ages. Certainly, programming might influence cognitive development. It’s important to figure out what we might expect.

That’s what Diana’s group is doing. She isn’t saying that fourth grader’s can’t initialize variables and properties. She’s saying it’s challenging for them. Her results are likely influenced by Scratch and by how the students were taught — it’s still an important result. Diana’s group is offering a starting point for exploring these interactions and understanding what we can expect to be easy and what might be hard for the average elementary school student at different ages.  There may be studies that also tell us about developmental progressions in countries that are ahead of the US in elementary school CS (e.g., maybe Israel or Germany). This is the first study of its kind that I’ve read.

SIGCSE 2017 introduced having Best Paper awards in multiple categories and Exemplary Paper awards. I applaud these initiatives. Other conferences have these kinds of awards. The awards helps our authors stand out in job searches and promotion time.

To be really meaningful awards, though, SIGCSE has to fix the reviewing processes. There were hiccups in this year’s reviewing where there wasn’t much of a match between reviewer expertise and the paper’s topic. The hiccups led to papers with significant flaws getting high rankings.

The Best Paper award in the Experience Report category was “Making Noise: Using Sound-Art to Explore Technological Fluency.” The authors describe a really nifty idea. They implement a “maker” kind of curriculum. One of the options is that students get toys that make noise then modify and reprogram them. The toys already work, so it’s about understanding a system, then modifying and augmenting it. The class sounds great, but as Leah Buchele has pointed out, “maker” curricula can be overwhelmingly male. I was surprised that this award-winning paper doesn’t mention females or gender — at all. (There is one picture of a female student in the paper.) I understand that it’s an Experience Report, but gender diversity is a critical issue in CS education, particularly with maker curricula. I consider the omission of even a mention of gender to be a significant flaw in the paper.

April 3, 2017 at 7:00 am 12 comments

Preview ICER 2016: Ebooks Design-Based Research and Replications in Assessment and Cognitive Load Studies

The International Computing Education Research (ICER) Conference 2016 is September 8-12 in Melbourne, Australia (see website here). There were 102 papers submitted, and 26 papers accepted for a 25% acceptance rate. Georgia Tech computing education researchers are justifiably proud — we submitted three papers to ICER 2016, and we had three acceptances. We’re over 10% of all papers at ICER 2016.

One of the papers extends the ebook work that I’ve reported on here (see here where we made them available and our paper on usability and usage from WiPSCE 2015). Identifying Design Principles for CS Teacher Ebooks through Design-Based Research (click on the title to get to the ACM DL page) by Barbara Ericson, Kantwon Rogers, Miranda Parker, Briana Morrison, and I use a Design-Based Research perspective on our ebooks work. We describe our theory for the ebooks, then describe the iterations of what we designed, what happened when we deployed (data-driven), and how we then re-designed.

Two of our papers are replication studies — so grateful to the ICER reviewers and communities for seeing the value of replication studies. The first is Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment by Miranda Parker, me, and Shelly Engleman. This is Miranda’s paper expanding on her SIGCSE 2016 poster introducing the SCS1 validated and language-independent measure of CS1 knowledge. The paper does a great survey of validated measures of learning, explains her process, and then presents what one can and can’t claim with a validated instrument.

The second is Learning Loops: A Replication Study Illuminates Impact of HS Courses by Briana Morrison, Adrienne Decker, and Lauren Margulieux. Briana and Lauren have both now left Georgia Tech, but they were still here when they did this paper, so we’re claiming them. Readers of this blog may recall Briana and Lauren’s confusing results from SIGCSE 2016 result that suggest that cognitive load in CS textual programming is so high that it blows away our experimental instructional treatments. Was that an aberration? With Adrienne Decker’s help (and student participants), they replicated the study. I’ll give away the bottom line: It wasn’t an aberration. One new finding is that students who did not have high school CS classes caught up with those who did in the experiment, with respect to understanding loops

We’re sending three of our Human-Centered Computing PhD students to the ICER 2016 Doctoral Consortium. These folks will be in the DC on Sept 8, and will present posters to the conference on Sept 9 afternoon.

September 2, 2016 at 7:53 am 17 comments

Transfer of learning: Making sense of what education research is telling us

I enjoy reading “Gas station without pumps,” and the below-quoted post was one I wanted to respond to.

Two of the popular memes of education researchers, “transferability is an illusion” and “the growth mindset”, are almost in direct opposition, and I don’t know how to reconcile them.

One possibility is that few students actually attempt to learn the general problem-solving skills that math, CS, and engineering design are rich domains for.  Most are content to learn one tiny skill at a time, in complete isolation from other skills and ideas. Students who are particularly good at memory work often choose this route, memorizing pages of trigonometric identities, for example, rather than learning how to derive them at need from a few basics. If students don’t make an attempt to learn transferable skills, then they probably won’t.  This is roughly equivalent to claiming that most students have a fixed mindset with respect to transferable skills, and suggests that transferability is possible, even if it is not currently being learned.

Teaching and testing techniques are often designed to foster an isolation of ideas, focusing on one idea at a time to reduce student confusion. Unfortunately, transferable learning comes not from practice of ideas in isolation, but from learning to retrieve and combine ideas—from doing multi-step problems that are not scaffolded by the teacher.

Source: Transfer of learning | Gas station without pumps

The problem with “transferability” is that it’s an ill-defined term.  Certainly, there is transfer of skill between domains.  Sharon Carver showed a long time ago that she could teach debugging Logo programs, and students would transfer that debugging process to instructions on a map (mentioned in post here).  That’s transferring a skill or a procedure.  We probably do transfer big, high-level heuristics like “divide-and-conquer” or “isolate the problem.”  One issue is whether we can teach them.  John Sweller says that we can’t — we must learn them (it’s a necessary survival skill), but they’re learned from abstracting experience (see Neil Brown’s nice summary of Sweller’s SIGCSE keynote).

Whether we can teach them or not, what we do know is that higher-order thinking is built on lots of content knowledge. Novices are unlikely to transfer until they know a lot of stuff, a lot of examples, a lot of situations. For example, novice designers often have “design fixation.”  They decide that the first thing they think of must be the right answer.  We can insist that novice designers generate more designs, but they’re not going to generate more good designs until they know more designs.  Transfer happens pretty easily when you know a lot of content and have seen a lot of situations, and you recognize that one situation is actually like another.

Everybody starts out learning one tiny skill at a time.  If you know a lot of skills (maybe because you have lots of prior experience, maybe because you have thought about these skills a lot and have recognized the general principles), you can start chunking these skills and learning whole schema and higher-level skills.  But you can’t do that until you know lots of skills.  Students who want to learn one tiny skill at a time may actually need to still learn one tiny skill at a time. People abstract (e.g., able to derive a solution rather than memorize it) when they know enough content that it’s useful and possible for them to abstract over it.  I completely agree that students have to try to abstract.  They have to learn a lot of stuff, and then they have to be in a situation where it’s useful for them to abstract.

“Growth mindset” is a necessity for any of this to work.  Students have to believe that content is worth knowing and that they can learn it.  If students believe that content is useless, or that they just “don’t do math” or “am not a computer person” (both of which I’ve heard in just the last week), they are unlikely to learn content, they are unlikely to see patterns in it, and they are unlikely to abstract over it.

Kevin is probably right that we don’t teach problem solving in engineering or computing well.  I blogged on this theme for CACM last month — laboratory experiments work better for a wider range students than classroom studies.  Maybe we teach better in labs than in classrooms?  The worked examples effect suggests that we may be asking students to problem solve too much.  We should show students more completely worked out problems.  As Sweller said at SIGCSE, we can’t expect students to solve novel problems.  We have to expect students to match new problems to solutions that they have already seen.  We do want students to solve problems, too, and not just review example solutions. Trafton and Reiser showed that these should be interleaved: Example, Problem, Example, Problem… (see this page for a summary of some of the worked examples research, including Trafton & Reiser).

When I used to do Engineering Education research, one of my largest projects was a complete flop.  We had all this prior work showing the benefits of a particular collaborative learning technology and technique, then we took it into the engineering classroom and…poof! Nothing happened.  In response, we started a project to figure out why it failed so badly.  One of our findings was that “learned helplessness” was rampant in our classes, which is a symptom of a fixed mindset.  “I know that I’m wrong, and there’s nothing that I can do about it.  Collaboration just puts my errors on display for everyone,” was the kind of response we’ve got. (See here for one of our papers on this work.)

I believe that all the things Kevin sees going wrong in his classes really are happening.  I believe he’s not seeing transfer that he might reasonably expect to see.  I believe that he doesn’t see students trying to abstract across lower-level skills.  But I suspect that the problem is the lack of a growth mindset.  In our work, we saw Engineering students simply give up.  They felt like they couldn’t learn, they couldn’t keep up, so they just memorized.  I don’t know that that’s the cause of the problems that Kevin is seeing.  In my work, I’ve often found that motivation and incentive are key to engagement and learning.

April 25, 2016 at 7:33 am 2 comments

Optimizing Learning with Subgoal Labeling: Lauren Margulieux Defends her Dissertation

Lauren Margulieux successfully defended her dissertation Using Subgoal Learning and Self-Explanation to Improve Programming Education in March. Lauren’s been exploring subgoal labeling for improving programming education in a series of fascinating and influential papers. Subgoal labels are inserted into the steps of a worked example to explain the purpose for a set of steps.

  • At ICER 2012 (see post here), her paper showed that subgoal labels inserted into App Inventor videos led to improved learning, retention (a week later), and even transfer to new App building problems, all compared to the exact same videos without the subgoal labels. This paper was cited by Rob Moore and his students at MIT in their work developing crowdsourced subgoal labels for videos (see post here).
  • At ICER 2015 (see post here), Lauren and Briana Morrison showed that subgoal labels also improved learning for textual programming languages, but the high cognitive load of textual programming language made some forms of subgoal labeling less successful than studies in other disciplines would predict. That paper won the Chairs Award at ICER.
  • At SIGCSE 2016 (see post here), Briana presented a paper with Lauren where they showed that subgoal labeling also improved performance on Parson’s Problems.

In her dissertation work, Lauren returned to the challenges of the ICER 2015 paper: Can we make subgoal labeling even more successful? She went back to using App Inventor, to reduce the cognitive load from teaching a textual language.

She compared three different ways of using subgoal labeling.

  • In the passive condition, students were just given subgoal labels like in her first experiments.
  • In the active condition, students were given a list of subgoal labels. The worked example was segmented into sets of steps that achieved a subgoal, but the label was left blank. Students had to pick the right subgoal label each blank.
  • In the constructive condition, students were just given a blank and asked to generate a subgoal label. She had two kinds of constructive conditions. One was “guided” in that there were blanks above sets of steps. The other was “unguided” — just a long list of steps, and she asked students to write labels into margins

Lauren was building on a theory that predicted that the constructive condition would have the best learning, but would also be the hardest. She provided two scaffolds.

  • For the conditions where it made sense (i.e., not the passive condition), she provided feedback. She showed half the participants the same worked examples with experimenter labels.
  • For half the constructive participants, the label wasn’t blank. Instead there was a hint. All the steps that achieved the same subgoal were labeled “Label 1,” and all the steps that achieved a different subgoal were labelled “Label 2,” and so on.

Here’s the big “interesting/surprising” graph from her dissertation.

Lauren-interesting-graph

As predicted, constructive was better than active or passive. What’s interesting is that the very best performance was guided constructive without hints but with feedback AND with hints but without feedback. Now that’s weird. Why would having more support (both hints and feedback) lead to worse performance?

There are several possible hypotheses for these results, and Lauren pursued one of these one step further. Maybe students developed their own cognitive model when they constructed their own labels with hints, and seeing the feedback (experimenter’s labels) created some kind of dissonance or conflict. Without hints, maybe the feedback helped them make sense of the worked example.

Lauren ran one more experiment where she contrasted getting scaffolding with the experimenter’s labels versus getting scaffolding with the student’s labels (put in all the right places in the worked example). Students who were scaffolded with their own labels performed better on later problem solving than those who were scaffolded with experimenter labels. Students scaffolded with experimenter labels did not perform better than those who did not receive any scaffolding at all. Her results support this hypothesis — the experimenter’s labels can get in the way of the understanding that the students are building.

using-learner-labels

There are several implications from Lauren’s dissertation. One is that we can do even better than just giving students labels — getting them to write them themselves is even better for learning. Feedback isn’t the most critical part of the learning when subgoal labeling, which is surprising and fascinating. Constructive subgoal labeling lends itself to an online implementation, which is the direction Lauren that is explicitly exploring. How do we build effective programming education online?

Lauren has accepted an Assistant Professor position in the Learning Technologies Division at Georgia State University. I’m so glad for her, and even happier that she’s nearby so that we can continue collaborating!

March 29, 2016 at 9:41 pm 7 comments

SIGCSE 2016 Preview: Parsons Problems and Subgoal Labeling, and Improving Female Pass Rates on the AP CS exam

Our research group has two papers at this year’s SIGCSE Technical Symposium.

Subgoals help students solve Parsons Problems by Briana Morrison, Lauren Margulieux, Barbara Ericson, and Mark Guzdial. (Thursday 10:45-12, MCCC: L5-L6)

This is a continuation of our subgoal labeling work, which includes Lauren’s original work showing how subgoal labels improved learning, retention and transfer in learning App Inventor (see summary here), the 2015 ICER Chairs Paper Award-winning paper from Briana and Lauren showing that subgoals work for text languages (see this post for summary), and Briana’s recent dissertation proposal where she explores the cognitive load implications for learning programming (see this post for summary). This latest paper shows that subgoal labels improve success at Parson’s Problems, too. One of the fascinating results in this paper is that Parson’s Problems are more sensitive as a learning assessment than asking students to write programs.

Sisters Rise Up 4 CS: Helping Female Students Pass the Advanced Placement Computer Science A Exam by Barbara Ericson, Miranda Parker, and Shelly Engelman. (Friday 10:45-12, MCCC: L2-L3)

Barb has been developing Project Rise Up 4 CS to support African-American students in succeeding at the AP CS exam (see post here from RESPECT and this post here from last year’s SIGCSE). Sisters Rise Up 4 CS is a similar project targeting female students. These are populations that have lower pass rates than white or Asian males. These are examples of supporting equality and not equity. This paper introduces Sisters Rise Up 4 CS and contrasts it with Project Rise Up 4 CS. Barb has resources to support people who want to try these interventions, including a how-to ebook at http://ice-web.cc.gatech.edu/ce21/SRU4CS/index.html and an ebook for students to support preparation for the AP CS A.

February 29, 2016 at 7:56 am 9 comments

Blog Post #2000: Barbara Ericson Proposes: Effectiveness and Efficiency of Adaptive Parsons Problems #CSEdWeek

My 1000th blog post looked backward and forward.  This 2000th blog post is completely forward looking, from a personal perspective.  Today, my wife and research partner, Barbara Ericson, proposes her dissertation.

Interesting side note: One of our most famous theory professors just blogged on the theory implications of the Parsons Problems that Barb is studying. See post here.

Barb’s proposal is the beginning of the end of this stage in our lives.  Our youngest child is a senior in high school. When Barbara finishes her Human-Centered Computing PhD (expected mid-2017), we will be empty-nesters and ready to head out on a new adventure.

Title: EVALUATING THE EFFECTIVINESS AND EFFICIENCY OF PARSONS PROBLEMS AND DYNAMICALLY ADAPTIVE PARSONS PROBLEMS AS A TYPE OF LOW COGNITIVE LOAD PRACTICE PROBLEM

Barbara J. Ericson
Ph.D. student
Human Centered Computing
College of Computing
Georgia Institute of Technology

Date: Wednesday, December 9, 2015
Time: 12pm to 2pm EDT
Location: TSRB 223

Committee
————–
Dr. James Foley, School of Interactive Computing (advisor)
Dr. Amy Bruckman, School of Interactive Computing
Dr. Ashok Goel, School of Interactive Computing
Dr. Richard Catrambone, School of Psychology
Dr. Mitchel Resnick, Media Laboratory, Massachusetts Institute of Technology

Abstract
———–

Learning to program can be difficult and can result in hours of frustration looking
for syntactic or semantic errors. This can make it especially difficult to prepare inservice
(working) high school teachers who don’t have any prior programming
experience to teach programming, since it requires an unpredictable amount of time for
practice in order to learn programming. The United States is trying to prepare 10,000
high school teachers to teach introductory programming courses by fall 2016. Most
introductory programming courses and textbooks rely on having learners gain experience
by writing lots of programs. However, writing programs is a complex cognitive task,
which can easily overload working memory, which impedes learning.

One way to potentially decrease the cognitive load of learning to program is to
use Parsons problems to give teachers practice with syntactic and semantic errors as well
as exposure to common algorithms. Parsons problems are a type of low cognitive load
code completion problem in which the correct code is provided, but is mixed up and has
to be placed in the correct order. Some variants of Parsons problems also require the
code to be indented to show the block structure. Distractor code can also be provided
that contains syntactic and semantic errors.

In my research I will compare solving Parsons problems that contain syntactic and
semantic errors, to fixing code with the same syntactic and semantic errors, and to writing
the equivalent code. I will examine learning from pre- to post-test as well as student
reported cognitive load. In addition, I will create dynamically adaptive Parsons problems
where the difficulty level of the problem is based on the learners’ prior and current
progress. If the learner solves one Parsons problem in one attempt the next problem will
be made more difficult. If the learner is having trouble solving a Parsons problem the
current problem will be made easier. This should enhance learning by keeping the
problem in the learner’s zone of proximal development as described by Vygotsky. I will
compare non-adaptive Parsons problems to dynamically adaptive Parsons problems in
terms of enjoyment, completion, learning, and cognitive load.

The major contributions of this work are a better understanding of how variants of
Parsons problems can be used to improve the efficiency and effectiveness of learning to
program and how they relate to code fixing and code writing. Parsons problems can help
teachers practice programming in order to prepare them to teach introductory computer
science at the high school level and potentially help reduce the frustration and difficulty
all beginning programmers face in learning to program.

 

December 9, 2015 at 7:37 am 5 comments

ICER 2015 Preview: Subgoal Labeling Works for Text, Too

Briana Morrison is presenting the next stage of our work on subgoal labeled worked examples, with Lauren Margulieux. Their paper is “Subgoals, Context, and Worked Examples in Learning Computing Problem Solving.” As you may recall, Lauren did a terrific set of studies (presented at ICER 2012) showing how adding subgoal labels to videos of App Inventor worked examples had a huge effect on learning, retention, and transfer (see my blog post on this work here).

Briana and Lauren are now teaming up to explore new directions in educational psychology space and new directions in computing education research.

  • In the educational psychology space, they’re asking, “What if you make the students generate the subgoal labels?” Past research has found that generating the subgoal labels, rather than just having them given to the students, is harder on the students but leads to more learning.
  • They’re also exploring what if the example and the practice come from the same or different contexts (where the “context” here is the cover story or word problem story). For example, we might show people how to average test grades, but then ask them to average golf scores — that’s a shift in context.
  • In the computing education research space, Briana created subgoal labeled examples for a C-like pseudocode.

One of the important findings is that they replicated the earlier study, but now in a text-based language rather than a blocks-based language. On average, subgoal labels on worked examples improve performance over getting the same worked examples without subgoal labels. That’s the easy message.

The rest of the results are much more puzzling. Being in the same context (e.g., seeing averaging test scores in the worked examples, then being asked to average test scores in the practice) did statiscally worse than having to shift contexts (e.g., from test scores to golf scores). Why might that be?

Generating labels did seem to help performance. The Generate group had the highest attrition. That make sense, because increased complexity and cognitive load would predict that more participants would give up. But that drop-our rate makes it hard make strong claims. Now we’re comparing everyone in the other groups to only “those who gut it out” in the Generate group. The results are more suspect.

There is more nuance and deeper explanations in Briana’s paper than I’m providing here. I find this paper exciting. We have an example here of well-established educational psychology principles not quite working as you might expect in computer science. I don’t think it puts the principles in question. It suggests to me that there may be some unique learning challenges in computer science, e.g., if the complexity of computer science is greater than in other studies, then it’s easier for us to reach cognitive overload. Briana’s line of research may help us to understand how learning computing is different from learning statistics or physics.

August 7, 2015 at 7:40 am 8 comments

ICER 2014 Preview: Briana Morrison and an instrument for measuring cognitive load

The International Computing Education Research (ICER) conference 2014 is August 11-13 in Glasgow (see program here).  My involvement starts Saturday August 9 when we have the welcome dinner for the doctoral consortium, which will be run all day on Sunday August 10 (Sally Fincher and I are chairing).  The main conference presentations continue through noon on Wednesday August 13. The rest of August 13 and into Thursday August 14 will be a new kind  of ICER session: Critical Research Review for work-in-progress.  I’m presenting on some new work that I’m getting feedback on related to constructionism for adults. I’ll blog about that later.

Briana Morrison is presenting her paper on developing an instrument to measure cognitive load (early version of paper available here), with co-authors Brian Dorn (my former student, now a chaired assistant professor at U. Nebraska-Omaha) and me.  Briana’s research is looking at the impacts of modality on program understanding for students.  Does audio vs. video vs. both have an impact on student understanding?  She’s controlling for time in all her presentations, and plans to measure performance…and cognitive load.  Is it harder for students to understand audio descriptions of program code, or to try to read text descriptions while trying to read text programs?

There wasn’t a validated instrument for her to use to measure the components of cognitive load — so she created one.  She took an existing instrument, and adapted it to computer science.  She and Brian did the hard work of crunching all the correlations and load factors to make sure that the instrument is still valid after her adaptation.  It’s an important contribution in terms of giving computing education researchers another validated tool for measuring something important about learning.

August 8, 2014 at 7:50 am 3 comments

A Report on Worked Examples and Self-Explanations in Media Computation

I should give you a little report on how my worked examples/self-explanation intervention worked in my Media Computation class.  I have nothing close to real data, and you shouldn’t believe me if I offered any.  This is a rarified class: 22 students, meeting four days a week for 90 minutes, plus office hours for 90 minutes twice each week (that most of the students have come to), and the teacher (who is the author of the textbook) attends breakfast and dinner with the students.  I think it would be hard to get more student-teacher interaction than in this model.

That said, I would definitely do it again.  I was quite surprised at how seriously the students took the task of explaining these programs!  In retrospect, I shouldn’t have been surprised.  In most classes, aren’t students asked to analyze and explain situations, even asked to make sense of some text?  That’s exactly what I asked these students to do, and they really worked at it.  I had students coming to office hours to ask about their assigned programs, so that they could write up their one paragraph of explanation. There were things that I had to teach them about this process, e.g., teaching them to try a program with different data sets, to make sure that the odd result they got wasn’t an anomaly.  I gave them feedback (every single student, on every single program) about the quality of their explanations, and the explanations definitely got better over time.

The real benefit was that they were trying to understand some relatively complicated code before it was their own code that they were trying to understand (while also designing and debugging it, all before a deadline).   With the worked examples tasks, they were just trying to understand.  There clearly was a reduction in cognitive load.  Variations on the below program had lots of students coming to see me — combining sounds at different rates was a challenging idea, but students did a good job of getting a grasp on it:

def modifysound2(sound):
 retsound = makeEmptySound(2*getLength(sound))
 newsound = makeSound(getMediaPath("bassoon-c4.wav"))
 trgi = 0
 nsi = 0
 for i in range(getLength(sound)):
 value = getSampleValueAt(sound,i)
 if nsi < getLength(newsound):
 nsvalue = getSampleValueAt(newsound,int(nsi))
 else:
 nsvalue = 0
 setSampleValueAt(retsound,trgi,value+nsvalue)
 trgi = trgi + 1
 nsi = nsi + 0.5
 return resound

Because there four labs (that just involved explaining programs) and two homework’s (that involved typing in, executing, and explaining programs), the first real programming assignment was the collage assignment.  Everybody did it.  Everybody turned in a working program.  And some of these were huge.  This one (by Savannah Andersen) was over 100 lines of code:

This one, by Julianne Burch, is over 200 lines of code.  I’m posting shrunk versions here: Julianne’s is about 4000 pixels across, representing the travel portion of this study abroad program.

I suspect that the worked examples and self-explanations gave the students more confidence than they normally have when facing their first programs.  It’s unusual in my experience for students to be willing to write 50-200 lines of working code for their first programming assignment.

But some of these students were also getting it.  A few of my students realized that they could make their collages more easily by using a copy() method to reduce the complication of composing pictures.  I did prompt them to do that, and a few did — most just went with hard-coded FOR loops, because that was easier for them to understand.  When I described how to do that, one student asked, “Aren’t you just naming some of those lines of code?” Yes! Nice way to start thinking about functions and abstract: it’s about naming chunks of code.  One of my students, without prompting, also decided to create a copy() method for her sound collage.  They’re starting to grapple with abstraction.  Given that this is the third week of class, when none of them had any previous programming experience (all my students are liberal arts and management students), I think that they’re doing quite well at moving from notation into abstraction.

They’re working on their first midterm exam now, a take-home exam (to save classroom time.)  I think it’s significantly challenging for a first exam, but it doesn’t have much coding.  It has a lot of analysis of code, because that’s one of the key learning objectives.  I want them to be able to look at a piece of code and predict its behavior, to trace (if necessary) what’s going on.  For me, that’s a more important outcome from a first course than being able to write a lot of code.

July 16, 2012 at 4:53 am 10 comments

In Praise of Drill and Practice

Last night, Barb and I went out to dinner with our two teens.  (The house interior is getting painted, so it was way easier than trying to eat in our kitchen.)  We got to talking about the last academic year.  Our eldest graduated from high school last week, with only one B in four years, including 7 AP classes.  (While I take pride in our son, I do recognize that kids’ IQ is most highly correlated with mothers’ IQ. I married well.) Our middle child was moping a bit about how hard it was going to be to follow in his footsteps, though she’s doing very well at that so far.

Since our middle child had just finished her freshman year, we asked the two of them which teachers we should steer our youngest toward or away from.  As they compared notes on their experiences, I asked about their biology teacher, Mrs. A.  I couldn’t believe the homework load that Mrs. A. sent home with the kids each night — almost all worksheets, fill-in-the-blank, drill-and-practice.  Sometimes, our middle child would have 300 questions to complete in a night!

Both our kids loved Mrs. A!  No, they didn’t love the worksheets, but they said that they really liked how the worksheets “drilled the material into our heads.”  “She’s such a great teacher!” they both said.  They went on to talk about topics in biology, using terms that I didn’t know.  Our middle child said that she’s looking forward to taking anatomy with Mrs. A, and and our eldest said that many of his friends took anatomy just to have Mrs. A again.

I was surprised.  My kids are pretty high-ability, and this messes with my notions of Aptitude-Treatment Interactions.  High ability kids value worksheets, simple drill-and-practice — what I used to call “drill-and-kill”?

On the other hand, their experience meshes with the “brain as muscle” notions that Carl Wieman talked about at SIGCSE.  They felt that they really learned from all that practice in the fundamentals, in the language and terms of the field.  Cognitive load researchers would point out that worksheets have low cognitive load, and once that material is learned, students can build on it in more sophisticated and interesting ways.  That’s definitely what I heard my kids doing, in some really interesting discussions about the latest biology findings, using language that I didn’t know.

I realized again that we don’t have (or at least, use) the equivalent of worksheets in computer science.  Mathematics have them, but my sense is that mathematics educators are still figuring out how to make them work well, in that worksheets have low cognitive load but it’s still hard getting to what we want students to learn about mathematics.  I suspect that computational worksheets would serve mathematics and computer science better than paper-based ones.  A computational worksheet could allow for dynamics, the “playing-out” of the answer to a fill-in-the-blank question.  Much of what we teach in introductory computer science is about dynamics: about how that loop plays out, about how program state is influenced and manipulated by a given process, about how different objects interact.  That could be taught (partially, the foundational ideas) in a worksheet form, but probably best where the dynamics could be made explicit.

Overall, though, my conversation with my kids about Mrs. A and her worksheets reminded me that we really don’t have much for CS learners before throwing them in front of a speeding interpreter or compiler.  A blank editor window is a mighty big fill-in-the-blank question. We need some low cognitive load starting materials, even for the high ability learners.

May 26, 2010 at 10:15 am 15 comments


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,005 other followers

Feeds

Recent Posts

Blog Stats

  • 1,879,106 hits
October 2021
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

CS Teaching Tips