When more information leads to worse performance: Beware throwing in “something fun and totally optional”

Eliane Wiese gave a talk here this last week. She told a story that I found fascinating. It connects to a story I just read about from Kahneman and Tversky. The theme has important implications for the design of software for CS education.

Story One: In Eliane’s dissertation work she explored how to give grounded feedback that would lead students to learn from mistakes. Here (in summary form) is the result of one of her studies.

In some questions, students were shown graphical representations of fractions. In other questions, they were shown some combination of graphical representations and symbolic fractions. In a fourth kind of questions, they’re just shown symbolic fractions. The vertical axis is performance.

The part that I find amazing is the results for condition two and three for fraction addition. Getting more information led to worse performance. Symbolic fractions are so confusing that their appearance depresses performance, even when the graphical information is still there. The students don’t just ignore the fractions. The mere presence of the fractions makes the problem harder for students.

(Original paper available here. Her follow-up/replication study can be found here. Thanks to Eliane for reviewing this post and sending me these links!)

Story Two: I just finished reading The Undoing Project (Amazon link) by Michael Lewis, the story of Daniel Kahneman and Amos Tversky’s amazing collaboration and friendship. One of their experiments is particularly relevant to Eliane’s finding.

You tell people that they’re going to pick a person at random from a pool of 100 people, 70 of whom are engineers and 30 of whom are lawyers. What is the probability that you’re going to get an engineer? Participants in the studies correctly guess 70%. You can change it to lawyers, or change around the ratios, and people solve this problem correctly and easily.

Now you tell them that, from the same pool, they have selected “Dick.”

Dick is a 30 year old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.

Now, what is the probability that Dick is an engineer? Participants say that the probability is 50% — they can’t tell. Notice that the description of Dick offers no additional information to discern if he is an engineer or a lawyer. Yet, people can’t ignore the useless descriptive information. They can’t just rely on the numbers. Getting more information leads to worse performance. People seem to feel a need to use all available information, even if it’s not useful, even if leads to worst performance.

What’s the implication for CS Ed? Our programming languages and professional IDE’s are complex. How about public static void main(String[] args)? How about all the bells and whistles in Eclipse?

When I point these out to teachers, the most common response I get is, “It’s okay. Students just ignore that part.”

I’m not sure that they do, or that they even can. People try to make sense of the information in front of them. We are drawn to create narratives. It is difficult for us to ignore information and make decisions based on only the relevant information. This is particularly hard for novices who don’t understand the relevant information, let alone separate the relevant from the irrelevant.

Before we toss something into our classes, we should pause and consider these stories. Sure, your CS1 students could use a cool new library that lets them do something cool (whatever — robotics, data visualizations, social network analysis) but has a confusing API and almost no documentation. The new library will consume their time and effort to understand. Sure, you might decide to introduce something (maybe list comprehensions or lambda expressions) into your Python code, just as “something fun” and “totally optional.” But students will try to understand it, and might not learn the things you really want them to learn. Sure, you could throw in a quick algorithm animation or use some super cool new debugger, but if your students are already confused, you’ve now just given them yet another representation or interface to make sense of. Think about the fact that the additional/extra/irrelevant information may be distracting your students from what is important. And that might lead to worse performance.