When more information leads to worse performance: Beware throwing in “something fun and totally optional”

March 23, 2018 at 7:00 am 16 comments

Eliane Wiese gave a talk here this last week. She told a story that I found fascinating. It connects to a story I just read about from Kahneman and Tversky. The theme has important implications for the design of software for CS education.

Story One: In Eliane’s dissertation work she explored how to give grounded feedback that would lead students to learn from mistakes. Here (in summary form) is the result of one of her studies.

In some questions, students were shown graphical representations of fractions. In other questions, they were shown some combination of graphical representations and symbolic fractions. In a fourth kind of questions, they’re just shown symbolic fractions. The vertical axis is performance.

The part that I find amazing is the results for condition two and three for fraction addition. Getting more information led to worse performance. Symbolic fractions are so confusing that their appearance depresses performance, even when the graphical information is still there. The students don’t just ignore the fractions. The mere presence of the fractions makes the problem harder for students.

(Original paper available here. Her follow-up/replication study can be found here. Thanks to Eliane for reviewing this post and sending me these links!)

Story Two: I just finished reading The Undoing Project (Amazon link) by Michael Lewis, the story of Daniel Kahneman and Amos Tversky’s amazing collaboration and friendship. One of their experiments is particularly relevant to Eliane’s finding.

You tell people that they’re going to pick a person at random from a pool of 100 people, 70 of whom are engineers and 30 of whom are lawyers. What is the probability that you’re going to get an engineer? Participants in the studies correctly guess 70%. You can change it to lawyers, or change around the ratios, and people solve this problem correctly and easily.

Now you tell them that, from the same pool, they have selected “Dick.”

Dick is a 30 year old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.

Now, what is the probability that Dick is an engineer? Participants say that the probability is 50% — they can’t tell. Notice that the description of Dick offers no additional information to discern if he is an engineer or a lawyer. Yet, people can’t ignore the useless descriptive information. They can’t just rely on the numbers. Getting more information leads to worse performance. People seem to feel a need to use all available information, even if it’s not useful, even if leads to worst performance.

What’s the implication for CS Ed? Our programming languages and professional IDE’s are complex. How about public static void main(String[] args)? How about all the bells and whistles in Eclipse?

When I point these out to teachers, the most common response I get is, “It’s okay. Students just ignore that part.”

I’m not sure that they do, or that they even can. People try to make sense of the information in front of them. We are drawn to create narratives. It is difficult for us to ignore information and make decisions based on only the relevant information. This is particularly hard for novices who don’t understand the relevant information, let alone separate the relevant from the irrelevant.

Before we toss something into our classes, we should pause and consider these stories. Sure, your CS1 students could use a cool new library that lets them do something cool (whatever — robotics, data visualizations, social network analysis) but has a confusing API and almost no documentation. The new library will consume their time and effort to understand. Sure, you might decide to introduce something (maybe list comprehensions or lambda expressions) into your Python code, just as “something fun” and “totally optional.” But students will try to understand it, and might not learn the things you really want them to learn. Sure, you could throw in a quick algorithm animation or use some super cool new debugger, but if your students are already confused, you’ve now just given them yet another representation or interface to make sense of. Think about the fact that the additional/extra/irrelevant information may be distracting your students from what is important. And that might lead to worse performance.

Entry filed under: Uncategorized. Tags: , , , .

Constructivism vs. Constructivism vs. Constructionism New programming languages are important to develop as we improve our knowledge of how students learn computing

16 Comments Add your own

  • 1. Alfred Thompson  |  March 23, 2018 at 8:16 am

    The thought that came to me while reading this is, are large textbooks more harmful than helpful? Do they provide too much information and overwhelm students? I suspect that some students don’t even start reading because of too much information. Would it be better to have short books that provided minimal information and have students “discover” more on their own?

    Someone smarter than me will have to figure that out. I have mostly questions.

  • 2. John Zabroski  |  March 23, 2018 at 8:19 am

    Reminds me of the User Model Fallacy in usability research. UX designers ‘build a model’ of how they think the User thinks, then build a user interface that, in their judgment, facilitates how the user thinks. On the other side, the user is building their own model, and changing the UI merely forces the user to recalibrate.

  • 3. Raul Miller  |  March 23, 2018 at 8:29 am

    My one question here would be: are these reliably reproducible results, or are they artifacts of some other difference?

    Put different, I can easily imagine this kind of “student failure” if subject was not fun for the students: if their interests were elsewhere, if they were struggling with an earlier part of the class. So my suspicion is that what we’re looking at here has to do with the rapport between the students and the teacher, more than anything else.

    But rapport can be a tricky subject to talk about. Still – if this research result turns out to be tricky to reproduce – if, for example, it turns out that pacing the math class on fractions “too slowly” loses some students and pacing the class “too quickly” loses other students – it might be possible to characterize this in terms of student interests or something along that line?

    • 4. Mark Guzdial  |  March 23, 2018 at 8:35 am

      Do read the replication study that I linked to, and let me know if you think that’s a reasonable replication.

      • 5. Raul Miller  |  March 23, 2018 at 8:57 am

        I think a variety of details in that replication study make a lot of sense.

        That said, I also struggle with descriptions of student difficulties (from students and from educators) which may or may not be relevant here.

  • 6. Karyn Voldstad  |  March 23, 2018 at 9:01 am

    Is short term performance the same as long term understanding? The students performed best with the fraction pictures, but that is insufficient long term. Which model will gain the greatest understanding for the long haul? My high school math students certainly don’t know how to use fractions!

    • 7. Raul Miller  |  March 23, 2018 at 9:50 am

      Short term performance is different from long term understanding, of course. But abstractions will be useless or misleading for people who do not understand them.

      Which gets tricky, in the context of math, because we generally bring to bear a lot of different abstractions with the same symbols. For example, a fraction may be a pair of numbers separated by a slash, but it’s also an abstraction for a single number. You need to have some kind of understanding that “single number” abstraction to do addition.

      Meanwhile, though, having concrete experiences is essential to building up understandings of abstractions. And that ties in with your question about performance and understanding. So, for example, with fractions, having experiences where you deal with measurement and breaking things into pieces can turn into good motivation for understanding of the intended abstraction. (Food preparation, maybe – not all students will have that in their backgrounds, but an adult who does not know how to prepare their own food will likely be an impoverished adult. Still, … if most students share some other background which can motivate understanding of fractions, it would be a shame not to use that.)

      That said, big fractions and small fractions will seem to be qualitatively different. Similarly, there are different notations for fractions which each require some learning (this is similar to picking up vocabulary in a language class).

  • 8. Mike Panitz  |  March 23, 2018 at 1:06 pm

    In my face to face classes I can sometimes try to increase engagement amongst the more advanced students by improvising a more challenging task for them. It works because they’re looking for a challenge, and I’m talking to people individually/in pairs as I circulate around the classroom.

    I wonder about including advanced material like that in stuff that the whole class sees – either online, or making it part of the instructional ‘routine’ by including it in the handout that the entire (face to face) class works off of in class.

    What I understand from your summary is that adding more info to a particular problem makes the problem harder. I’d be curious to know if you can mark an entire section as ‘unless you’ve finished everything well and are bored, please ignore this section’ and have people actually ignore it, or if it would still trip them up.

  • 9. panoptical  |  March 24, 2018 at 11:43 am

    This seems like it may also be an example of the redundancy effect, identified by Cognitive Load theory.

    For example:

    “In situations where a source of textual instruction, or a source of graphical instruction alone provides full intelligibility then only one source of instruction should be used (either the textual or the graphical), and the other source, which is redundant, should be removed completely from the instructional materials. In these contexts a single source of instruction returns higher levels of learning than either an integrated format (text integrated into the graphic), or a dual format (both text and graphic presented in parallel).”

    (source: http://dwb4.unl.edu/Diss/Cooper/UNSW.htm)

    • 10. Mark Guzdial  |  March 24, 2018 at 2:10 pm

      That explains Elaine’s results but not Kahneman and Tversky.

      • 11. panoptical  |  March 25, 2018 at 3:58 am

        Indeed, but I think we should be open to the possibility that there are two different effects going on here. In Story One you’re talking about relevant but redundant information; in Story Two it’s irrelevant information.

        There’s evidence that we process socially salient information differently than abstract reasoning tasks (i.e. https://en.wikipedia.org/wiki/Wason_selection_task) which means that the interference in Story Two may come about in particular because the information appears to be socially salient (Dick’s relationships with his spouse and colleagues) rather than because it’s merely extraneous.

        • 12. Mark Guzdial  |  March 25, 2018 at 6:47 am

          It’s a good point. It’s hard to tell how students process this information. How do they decide what’s salient or not?

        • 13. rademi  |  March 25, 2018 at 10:29 am

          I am thinking that the “real issue” here – with “redundant material” might be pacing as much as anything else. Repetition can help. Moving on to other topics can help.

          There’s some qualitative judgements which are incredibly important here, and you really need a mindset dedicated to making the class work, and helping the student’s learn, as a basis for those judgements. But of course, this kind of rudimentary thinking can also become “redundant”…

  • 14. chaikens  |  March 24, 2018 at 11:05 pm

    This indicates better ways to teach particular items. But we must not forget that AFTER students learn those items, at some point in their advancement, they must be taught to connect what they’ve learned to new things and how to recognize and overcome one’s tendency to be mislead by irrelevant information. An example of new things is after learning the geometric significance of fraction addition, we must learn that (1) it applies to sums of fractions written numerically and (2) how to add numerical fractions. (It’s unclear and not important to me whether a geometric illustration or the numerical form of a fraction is “more abstract”; I’d guess the idea should be taught first geometrically.) The lawyer and engineer kind of questions and other intuitive paradoxes should confront every student after learning basic probability.

    That said, these results are valuable to help teachers be more effective by recognizing and avoiding temptations to confuse students by too soon adding important, relevant and exciting subsequent content! But don’t throw them out later.

  • 15. shriramkrishnamurthi  |  March 25, 2018 at 7:52 am

    Re. all the things in your last paragraph: DrRacket doesn’t only grow the language and notional machine in a progression for students. At each level, it also restricts the set of tools that are made available to students, eliminating several of the complex, “professional” tools entirely from the visible IDE (so even a student pulling down menus won’t see them — they really don’t “exist”). In other words, all the things you mention.

    We’ve done this since about 2000, when it became evident that this was the natural extension of the “language level” metaphor.

    But of course it’s all for some weird language, not for something robustly middle-class like Java or Python, so why would anyone care to explore such a thing or learn ideas from it…

  • […] Logo? Pascal?  Even if there’s a “Trough of Disillusionment” (e.g., when we realized just how hard C++ and Java are), we still see longterm use. Even if we later realize how good something was (e.g., Logo for […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,030 other followers


Recent Posts

Blog Stats

  • 1,994,883 hits
March 2018

CS Teaching Tips

%d bloggers like this: