Posts tagged ‘behavioral economics’

Thought Experiments on Why Face-to-Face Teaching Beats On-Line Teaching: We are Humans, not Econs

With everything moving on-line, I’m seeing more discussion about whether this on-line life might just be better. Amy Ko recently blogged (see post here) about how virtual conferences are cheaper, more accessible, and lower carbon footprint than face-to-face conferences, ending with the conclusion for her “it is hard to make the case to continue meeting in person.” My colleague, Sarita Yardi, has been tweeting about her exploration of “medium-independent classes” where she considers (see tweet here), “Trying to use the block of class time just because that’s how we’ve always taught seems like something to revisit. Less synchronous time; support short, frequent individual/small group interaction, less class time.”

It’s hard to do on-line education well. I used to study this kind of learning a lot (see post on “What I have learned about on-line collaborative learning”). I recently wrote about how we’re mostly doing emergency remote teaching, not effective on-line learning (see post here). I am concerned that moving our classes on-line will hurt the most the students who most need our help (see post here).

It should come as no surprise then that I don’t think that we know how to do on-line teaching or on-line conferences in a way that is anywhere close to the effectiveness of face-to-face learning. I agree with both Amy and Sarita’s points. I’m only focusing on learning outcomes.

Let me offer a thought experiment on why face-to-face matters. How often do you…

  • Look at the movie trailer and not watch the movie.
  • Watch the first few minutes of a show on Netflix but never finish it.
  • Start a book and give up on it.
  • Start watching a YouTube video and immediately close it or click away.

Now contrast that with: How often do you…

  • Get up from a one-on-one meeting and walk out mid-discussion.
  • Get up in the middle of a small group discussion and leave.
  • Walk out of a class during a lecture.
  • Walk out of a conference session while the speaker is still presenting (not between talks or during Q&A).

For some people, the answers to the first set are like the answers for the second set. I tried this thought experiment on my family, and my wife pointed out that she finishes every book she starts. But for most people, the first set is much more likely to happen than the second set. This is particularly hard for professors and teachers to recognize. We are good at self-regulated learning. We liked school. We don’t understand as well the people who aren’t like us.

There are a lot of people who don’t really like school. There are good reasons for members of minority groups to distrust or dislike school. Most people engage in higher-education for the economic benefit. That means that they have a huge value for the reward at the end, but they don’t particularly want to go through the process. We have to incentivize them to be part of the process.

Yes, of course, many students skip classes. Some students skip many classes. But the odds are still in favor of the face-to-face classes. If you are signed up for a face-to-face class, you are much more likely to show up for that class compared to any totally free and absolutely relevant to your interests lecture, on-campus or on-line. Enrolling in a course is a nudge.

For most people, you are much more willing to walk away from an asynchronous, impersonal event than a face-to-face, personal event. The odds of you learning from face-to-face learning are much higher simply because you are more likely to show up and less likely to walk out. It’s a great design challenge to make on-line learning opportunities just as compelling and “sticky” as face-to-face learning. We’re not there yet.

I would be all in favor of efforts to teach people to be more self-regulated. It would be great if we all were better at learning from books, lectures, and on-line resources. But we’re not. The learners with the best preparation are likely the most privileged students. They were the ones who were taught how to learn well, how to learn from school, and how to enjoy school.

Here’s a second thought experiment, for people who work at Universities. At any University, there are many interesting talks happening every week. For me, at least a couple of those talks each week are faculty candidates, which I am highly encouraged to attend. Now, they’re all on-line. How many of those did you attend when they were face-to-face, and how many do you attend on-line? My guess is that both are small numbers, but I’ll bet that the face-to-face number is at least double the on-line number. Other people see that you’re there face-to-face. There are snacks and people to visit with face-to-face. The incentives are far fewer on-line.

On-line learning is unlikely to ever be as effective as face-to-face learning. Yes, we can design great on-line learning, but we do that fighting against how most humans learn most things. Studies that show on-line learning to be as effective (or even more effective) than face-to-face classes are holding all other variables equal. But holding all other variables equal takes real effort! To get people to show up just as much, to give people as much (or more) feedback, and to make sure that the demographics of the class stay the same on-line or face-to-face — that takes significant effort which is invisible in the studies that are trying to just ask face-to-face vs on-line. The reality is that education is an economic endeavor. Yes, you can get similar learning outcomes, at a pretty high cost. At exactly the same cost, you’re unlikely to get the same learning outcomes.

We are wired to show-up and learn from face-to-face events. I would love for all of us to be better self-regulated learners, to be better at learning from books and from lecture. But we’re not Econs, we’re Humans (to use the Richard Thaler distinction). We need incentives. We need prompts to reflect, like peer instruction. We need to see and be seen, and not just through a small box on a 2-D screen.

May 11, 2020 at 7:00 am 19 comments

Why are CS students so hard to nudge? A theory for why it’s so hard to promote a growth mindset in CS1

Pearson took a lot of heat recently for trying to improve students’ mindset in My Programming Lab.  I’m slightly worried about the ethics of their “embedded experiment.” I’m more worried that it didn’t work.

Titled “Embedding Research-Inspired Innovations in EdTech: An RCT of Social-Psychological Interventions, at Scale,” the study placed 9,000 students using MyLab Programming into three groups, each receiving different messages from the software as they attempted to solve questions. Some students received “growth-mindset messages,” while others received “anchoring of effect” messages. (A third control group received no messaging at all.) The intent was to see if such messages encouraged students to solve more problems. Neither the students nor the professors were ever informed of the experiment, raising concerns of consent.

The “growth mindset messages” emphasized that learning a skill is a lengthy process, cautioning students offering wrong answers not to expect immediate success. One example: “No one is born a great programmer. Success takes hours and hours of practice.” “Anchoring of effect” messages told students how much effort is required to solve problems, such as: “Some students tried this question 26 times! Don’t worry if it takes you a few tries to get it right.”

As Education Week reports, the interventions offered seemingly no benefit to the students. Students who received no special messages attempted to solve more problems (212) than students in either the growth-mindset (174) or anchoring groups (156). The researchers emphasized this could have been due any of a variety of factors, as the software is used differently in different schools.

Source: Pearson Embedded a ‘Social-Psychological’ Experiment in Students’ Educational Software [Updated]

Beth Simon and her colleagues tried a similar experiment, reported at ICER 2008.  They did get informed consent.  They tried a similar kind of “nudge” to get students to adopt a growth mindset.  It didn’t work for Beth et al., either.

I advised Kantwon Rogers’ MS in HCI project, where he tried to nudge CS1 students (both on-line and off-line) to have a greater sense of “belongingness” in CS.  Similar to these previous studies, he sent email prompts to students — some just encouraged study skills, and others promoted a sense that they belongs and could succeed in CS.  In almost all of his conditions, belongingness dropped.

What’s going on here?  Why are CS students so impervious to these prompts that have been successful in other settings?

I have a theory.  There’s a notion in the behavioral sciences literature that you get more success changing behavior or promoting attitudes by reducing barriers than by prompting for desired behavior or attitudes.  The analogy is to a large boulder that you want to move: You can push it and push it, or you can just dig away the dirt from the bottom.  The latter is likely to get the boulder rolling without as much effort.

Here’s my theory: Introductory CS classes have systemic issues that encourage a fixed mindset and discourage a sense of belongingThere are too many signals to students that they can’t succeed, that they can’t get better, and that they don’t belong — perhaps especially in times of rising enrollment. Mere nudges are not going to move the boulder.  We’re going to have to remove the barriers to belonging, self-efficacy, and the sense that students can succeed at CS.

 

May 7, 2018 at 7:00 am 19 comments

When more information leads to worse performance: Beware throwing in “something fun and totally optional”

Eliane Wiese gave a talk here this last week. She told a story that I found fascinating. It connects to a story I just read about from Kahneman and Tversky. The theme has important implications for the design of software for CS education.

Story One: In Eliane’s dissertation work she explored how to give grounded feedback that would lead students to learn from mistakes. Here (in summary form) is the result of one of her studies.

In some questions, students were shown graphical representations of fractions. In other questions, they were shown some combination of graphical representations and symbolic fractions. In a fourth kind of questions, they’re just shown symbolic fractions. The vertical axis is performance.

The part that I find amazing is the results for condition two and three for fraction addition. Getting more information led to worse performance. Symbolic fractions are so confusing that their appearance depresses performance, even when the graphical information is still there. The students don’t just ignore the fractions. The mere presence of the fractions makes the problem harder for students.

(Original paper available here. Her follow-up/replication study can be found here. Thanks to Eliane for reviewing this post and sending me these links!)

Story Two: I just finished reading The Undoing Project (Amazon link) by Michael Lewis, the story of Daniel Kahneman and Amos Tversky’s amazing collaboration and friendship. One of their experiments is particularly relevant to Eliane’s finding.

You tell people that they’re going to pick a person at random from a pool of 100 people, 70 of whom are engineers and 30 of whom are lawyers. What is the probability that you’re going to get an engineer? Participants in the studies correctly guess 70%. You can change it to lawyers, or change around the ratios, and people solve this problem correctly and easily.

Now you tell them that, from the same pool, they have selected “Dick.”

Dick is a 30 year old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.

Now, what is the probability that Dick is an engineer? Participants say that the probability is 50% — they can’t tell. Notice that the description of Dick offers no additional information to discern if he is an engineer or a lawyer. Yet, people can’t ignore the useless descriptive information. They can’t just rely on the numbers. Getting more information leads to worse performance. People seem to feel a need to use all available information, even if it’s not useful, even if leads to worst performance.

What’s the implication for CS Ed? Our programming languages and professional IDE’s are complex. How about public static void main(String[] args)? How about all the bells and whistles in Eclipse?

When I point these out to teachers, the most common response I get is, “It’s okay. Students just ignore that part.”

I’m not sure that they do, or that they even can. People try to make sense of the information in front of them. We are drawn to create narratives. It is difficult for us to ignore information and make decisions based on only the relevant information. This is particularly hard for novices who don’t understand the relevant information, let alone separate the relevant from the irrelevant.

Before we toss something into our classes, we should pause and consider these stories. Sure, your CS1 students could use a cool new library that lets them do something cool (whatever — robotics, data visualizations, social network analysis) but has a confusing API and almost no documentation. The new library will consume their time and effort to understand. Sure, you might decide to introduce something (maybe list comprehensions or lambda expressions) into your Python code, just as “something fun” and “totally optional.” But students will try to understand it, and might not learn the things you really want them to learn. Sure, you could throw in a quick algorithm animation or use some super cool new debugger, but if your students are already confused, you’ve now just given them yet another representation or interface to make sense of. Think about the fact that the additional/extra/irrelevant information may be distracting your students from what is important. And that might lead to worse performance.

March 23, 2018 at 7:00 am 16 comments

Human students need active learning and Econs learn from lecture: NYTimes Op-Ed in defense of lecture

I’m sympathetic to the author’s argument (linked below), that being able to understand an argument delivered as a lecture is difficult and worthwhile. Her characterization of active learning is wrong — it’s not “student-led discussion.”  Actually, what she describes as good lecture is close to good active learning.  Having students answering questions in discussion is good — but some students might disengage and not answer questions.  Small group activities, peer led team learning, or peer instruction would be better to make sure that all students engage. But that’s not the critical flaw in her argument.

Being able to listen to a complicated lecture is an important skill — but students (at least in STEM, at least in the US) don’t have that skill.  We can complain about that. We can reform primary and secondary schooling so that students develop that skill.  But if we want these students to learn, the ones who are in our classes today, we should use active learning strategies.

Richard Thaler introduced the term “Econs” to describe the rational beings that inhabit traditional economic theory. (See a review of his book Misbehaving for more discussion on Econs.)  Econs are completely rational.  They develop the skills to learn from lecture because it is the most efficient way to learn.  Unfortunately, we are not econs, and our classes are filled with humans. Humans are predictably irrational, as Daniel Ariely puts it. And there’s not much we can do about it. In his book Thinking, Fast and Slow, Daniel Kahneman complains that he knows how he is influenced by biases and too much System 1 thinking — and yet, he still makes the same mistakes.  The evidence is clear that the students in our undergraduate classes today need help to engage with and learn STEM skills and concepts.

The empirical evidence for the value of active learning over lecture is strong (see previous post).  It works for humans.  Lecture probably works for Econs.  If we could find enough of them, we could run an experiment.

In many quarters, the active learning craze is only the latest development in a long tradition of complaining about boring professors, flavored with a dash of that other great American pastime, populist resentment of experts. But there is an ominous note in the most recent chorus of calls to replace the “sage on the stage” with student-led discussion. These criticisms intersect with a broader crisis of confidence in the humanities. They are an attempt to further assimilate history, philosophy, literature and their sister disciplines to the goals and methods of the hard sciences — fields whose stars are rising in the eyes of administrators, politicians and higher-education entrepreneurs.

Source: Lecture Me. Really. – The New York Times

A similar argument to mine is below.  This author doesn’t use the Humans/Econs distinction that I’m using.  Instead, the author points out that lecturers too often teach only to younger versions of themselves.

I will grant that nothing about the lecture format as Worthen describes it is inherently bad. But Worthen’s elegy to a format that bores so many students reminds me of a bad habit that too many professors have: building their teaching philosophies around younger versions of themselves, who were often more conscientious, more interested in learning, and more patient than the student staring at his phone in the back of their classrooms.

Source: Professors shouldn’t only teach to younger versions of themselve

October 30, 2015 at 8:49 am 17 comments


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,030 other followers

Feeds

Recent Posts

Blog Stats

  • 1,994,757 hits
January 2022
M T W T F S S
 12
3456789
10111213141516
17181920212223
24252627282930
31  

CS Teaching Tips