My Students Know Far Less Than I Ever Expected
January 19, 2011 at 1:33 pm 35 comments
Today was my very first lecture using peer instruction in a computer science class, and I’m already blown away. I had no idea that my students had these misconceptions.
I’m teaching a second semester course on data structures, using our Media Computation approach. All of my students have completed their first semester of computer science, either in Python (with robotics or with media) or in MATLAB. This is only the second lecture of the semester (due to the Snowcopalypse), so I’m just introducing the basics of Java.
I showed this slide.
I then posed this question:
Since Java uses zero-based indexing, and MATLAB uses one-based, I expected that the answers would be between A and B, and it would give me a chance to revisit (I’d already presented it) the zero vs. one-based indexing. I posed the question, let students answer via Ubiquitous Presenter, then asked them to argue with one another about the right answer, then vote again. Here was the distribution of votes after the second round of votes.
I don’t know about you, but there is NO WAY I would have predicted that the MAJORITY of the class would think that the answer was the sum. Even if you had never seen Java before, how could you think that getting the total changes the array?
Wow. I’m flabbergasted. I predict that I will cover FAR less material than I planned to this semester, since I now have a way of seeing what students are understanding on a day-by-day basis. I’m now a convert. Given what we know about how little students learn in the first years, one could argue that it’s unethical not to use peer instruction — how else will you know if your students are learning?
Entry filed under: Uncategorized. Tags: computing education, peer instruction, teachers.
35 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
Computing Ed Research - Guzdial's Take 
1.
John "Z-Bo" Zabroski | January 19, 2011 at 1:42 pm
I think gauranteeing yourself that you will cover less is the wrong approach.
It is a self-fulling prophecy.
2.
John "Z-Bo" Zabroski | January 19, 2011 at 1:44 pm
Self-fulfilling* — hit enter too soon.
What I mean is that you are making a number of logic mistakes here. You are automatically implying your students can’t program and that you are screwed. This is what is wrong with your inference. You have indicated no previous experience that indicates the chain of inferences you are making hold true.
3.
Mark Guzdial | January 19, 2011 at 2:11 pm
John, I mean that I’m going to use more in-class questions, and I’ll spend time discussing the right answers. That’s going to take time that I might have used for exposition of more topics. I am making no inference about their ability to get a program working. In fact, the connection raises a fascinating empirical question. Even those who got the question wrong might be able to write programs dealing with similar concepts, simply through trial-and-error — but that takes time. Do students who get more of these questions RIGHT spend LESS time coding (or rather, debugging)? If so, then there is an interesting implication that learning more of the content in peer instruction may lead to more efficient programming sessions.
4.
John "Z-Bo" Zabroski | January 19, 2011 at 3:03 pm
I understand.
I am saying that predicting peer instruction will cover less material is a self-fulfilling prophecy. I also think the context in which you asked this question, showing iteration with FOR, and then asking about mutation of the array being iterated, probably wasn’t what the audience was expecting. Notice how on your first slide you provide the total. It is probably not clear to the students what objective you have in mind.
The first slide also doesn’t cover the notation for array indexing, which students may have forgotten.
Forgive me if I think that your conclusion presented in this blog post is misleading, since I was not in your class.
5.
Greg Wilson | January 19, 2011 at 1:55 pm
I embarrassed to admit it, but I answered ‘D’ as well, because I only glanced at the question, and my brain, having pattern-matched the loop to a “sum values up” pattern, “knew” that you were going to be asking about the value of “total”. Yes, I have email and Twitter open on my desktop—which makes me wonder, how many of your students do, and how many of them pattern-matched incorrectly (hastily) the way I did?
6.
Mark Guzdial | January 19, 2011 at 1:57 pm
Greg, would you have made that choice a second time, after discussing it with the people next to you?
7.
Greg Wilson | January 19, 2011 at 2:12 pm
Don’t know, but if it was just two of us, the odds of us both mis-reading are nonzero. What happens if you repeat the test and simply ask, “Have the values in the array changed (yes/no)?”
8.
Francisco Garau | January 20, 2011 at 3:43 am
I’ve also misread and answered D.
9.
Alfred Thompson | January 19, 2011 at 2:16 pm
In my own teaching I saw many cases of students taking a shortcut by reading part of the question and making assumptions about the rest of it. I wonder if our placing so much importance of answering a lot of multiple choice questions in a short period of time – think SAT, AP exams and more – if we are not forcing students to take shortcuts that on occasions backfire on them.
Now checking the answers after a discussion among peers and getting results like this are more scary than if this was just the first go round. Could it be that people rationalized their incorrect answers after the fact and were more convincing? I don’t know.
But I sure do agree that you have reason for concern.
10.
Beth Simon | January 19, 2011 at 2:34 pm
All,
I agree with everyone that misreading may account for a good portion (though likely not all) of the D votes. But the issue still stands: 47% of the A, B, or C answerers got it wrong. Still a big concern for moving forward with examples in the classroom that assume knowledge of indexing….
Beth
11.
Stephen Downes | January 19, 2011 at 3:27 pm
It was a trick question.
I answered 424 as well. Because I looked at the function, saw the result, then looked at the question, worded `what would be the value of`, and so gavew the result.
Had I looked on the line *after* the question, I would have seen gradearray[1]. But as it was, I didn’t even know that it was a part of the question. The formatting put it all by itself on another line.
What you *actually* tested was, “can students be distracted by the form of the question?” The answer is, unequivocally, yes.
But what you are saying here is that students do not know how to find the value of an array. That doesn’t follow at all from the actual quiz you gave.
12.
Mark Guzdial | January 19, 2011 at 3:35 pm
You are probably all correct, and it was a “trick question.” I’m going to keep using this approach, and I’ll try to do better at coming up with good questions.
13.
Rob St. Amant | January 19, 2011 at 7:56 pm
I agree with the commenters who think it’s a “trick question”. On the plus side, even if the students aren’t paying enough attention to the details of the question, they seem to have learned one important lesson: First try to figure out what the goal of a program is. The ones who answered D seem to have done that.
14.
Cynthia Lee | January 19, 2011 at 6:17 pm
Interesting read, Mark.
At the urging of Beth Simon, I adopted PI last year in an upper-division computing theory class. Running a PI classroom was surprisingly easy–students readily took to the format. But I found it incredibly time-consuming and mentally challenging to design questions that were non-trivial, but straightforward (not “trick” questions), that tested the particular aspects of the material I wanted emphasized, and that would spark some interesting disucssions.
Looks like you’ve found the same thing!
Good luck, and I’m interested in following your updates on your progress with the methodology.
15.
Rion Dooley | January 19, 2011 at 6:21 pm
Not to nit-pick, but Java is a case-sensitive language. Technically you don’t have an array variable named gradearray, so the actual answer should be none of the above.
16.
Tricky | January 19, 2011 at 6:35 pm
gradearray[] doesn’t exist
17.
Victor Eijkhout | January 19, 2011 at 6:41 pm
Java is case sensitive, so “none of the above”.
18.
Owen Astrachan | January 19, 2011 at 9:09 pm
The headline and the post are a kind of hybrid between sowing fear, uncertainty and doubt along with attraction via sensationalism.
Unless your students are drastically different this year than in previous years you should have a reasonable expectation of what they know and don’t know from teaching the material to students with similar backgrounds in previous years (unless, of course, this is the first time you’ve taught the course) and testing them even if it was by standard exams rather than using peer instruction. So starting with a headline of
My Students Know Far Less Than I Ever Expected
after missing one question is completely antithetical to understanding what students know. Peer Instruction may be a huge help (I think it is), but you’re generalizing from one example to what your students know and don’t know. My first thought would have been “there’s something wrong with the questions”, and not “it’s amazing what my students don’t know”
But that would have been my first thought based on having taught the stuff before. Or maybe because I enjoy pulling wool over my own eyes.
I think it is reasonable to wonder what students know *cold* and what they figure out by debugging using trial-and-error so that they don’t really know it.
On a related note, do you think your students read the blog? If they do, will they think less of themselves even though this isn’t something really under their control?
Misconceptions Students Have on Day One in CS 2
is less sensationalistic and (to me) a better take on what might have transpired.
Owen
19.
Mark Guzdial | January 19, 2011 at 10:15 pm
I haven’t taught this course in four years. When I was teaching it regularly, I was often surprised at the mistakes that students made. In the last four years, I’ve been reading about peer instruction, especially as the physics educators have used it. I do believe that it provides better insights into how students think than the exams and homework assignments I’ve used previously. Yes, I was surprised at the response from my students, and the title conveys my reaction. Now, I am also learning that it’s hard to write good peer instruction questions, and I expect to get better insight into what my students know (and what they don’t) in the following weeks.
If you came upon my blog expecting to find peer-reviewed quality writing with objective and academic-sounding titles, then I offer my apologies but you’re in the wrong place. This is my blog. I report on things I read and experiences I have, and reflect on them. If I was writing this up for a conference or journal publication, I’d say it differently. I wrote this post within the first hour after I finished the course, and it reflects my honest and immediate reaction.
I do think that we in CS need to do more to better measure what students are learning. I think that we’re fooling ourselves about how well we’re doing, and the research results support that belief. I do think Peer Instruction is a better way to measure learning than what I’ve used previously in my classes. I will learn to use Peer Instruction better this semester, and in so doing, I’ll come away with a better understanding of what my students know. I do encourage other CS instructors to try Peer Instruction, and they are welcome to use my experiences, reported in this blog, to avoid my mistakes.
20.
Joe | January 19, 2011 at 10:08 pm
Nitpick: 38% isn’t a majority. It’s a plurality. 56% (23 + 33) chose A or B — both reasonable answers to the question as you mention, and that is a majority.
21.
Jared Bruff | January 19, 2011 at 10:11 pm
Just another suggestion for multiple choice answers – remember that outliers can change how the student reads the question. For example, when I read it (quickly, like most students probably would have), the rusty processes I have in my brain from my Comp Sci undergrad told me to find the sum of the values in the array, because that would be an expected question for this statement. I initially would have fallen for the 424, but before actually performing the calculation, I looked at the answers. I saw that 424 was an oddball and thus very probably placed there for distraction. I took another look at the question to see why you placed values from the array in multiple answers, and I found my mistake. Otherwise, I would have fallen for the eager choice of 424. If the answers had been, for instance, values between 380 and 450, I would have actually calculated the total and missed the question. Outliers in answers can be used for multiple reasons; standardized test makers tend to put them in as options that are meant to be easy to weed out, but for your own questions, you could go for the entirely opposite effect. Depends on the goals of the question.
22.
Barry Brown | January 20, 2011 at 2:52 am
I tend to side with the other commenters here that the students answering D may simply have been pattern matching on the code without reading the whole question.
Multiple choice questions of this type (objective, quantitative) tend to fall into two categories:
Those that provide one correct answer and several “almost correct” distractors;
And those that provide one correct answer and several impossible, incorrect answers.
The problem with the first kind is it reinforces incorrect reasoning. I think tests ought to be an opportunity to learn and providing several “correct” answers leads the student into a false sense of security. The student, upon working problem, looks through the choices, sees one that matches his or her answer, selects it, and moves on to the next question without another thought.
On the other hand, if the question had provided three plausible but incorrect answers, then he or she would work the problem, see that their solution was not among the available choices, and be forced to re-evaluate whether their thinking was correct. The plausible distractors will appeal to the students who are merely guessing, but will steer the ones who are genuinely trying to answer the question to the right one.
23.
Bonnie MacKellar | January 20, 2011 at 10:47 am
I tend to agree with the conclusions made in this blog posting, simply based on my own anecdotal experience. I teach an upper level course on object oriented programming (my program doesn’t teach OO first, but leaves it to later, which is bad in many ways that I won’t go into here). I do in-class lab assignments, so I can observe students at work. I have seen over and over that students coming to this class, ostensibly with 3 semesters of programming coursework behind them, simply do not get arrays or loops that iterate over arrays. In one of my first labs, which was originally designed to just get them used to the programming environment, I ask them to write a simple program that averages some numbers stored in an array. Semester after semester, about half the class is stumped. I now use that lab as a way to assess the students, and to review (reteach?) arrays and loops. I think arrays and array algorithms are harder for students than we imagine.
24.
Blum | January 20, 2011 at 3:41 pm
Students do not know “less”, they are just too lazy to read the whole question and they answer what they THINK you are asking them…
25.
Ria Galanos | January 20, 2011 at 7:37 pm
I wholeheartedly agree with Blum. As a high school AP teacher, I spend a significant amount of time teaching students how to read the text and multiple choice questions for key information. I would have suggested to my students to read the question first before reading the code. While I don’t have any experience with this question to base my assumptions, I think you would have had different results. I am not an experienced test question creator, but I do not think this is a trick question or that there is any problem with the formatting. Students have to be able to learn to read through less than ideally formatting selections for information. After all, aren’t we preparing them for the real world?
26.
gasstationwithoutpumps | January 21, 2011 at 11:56 am
Everyone here is focusing on one problem (poor question design), but I had a more serious criticism. Getting student feedback is not the same thing as peer instruction. I believe that the anecdote illustrates some value to getting quick feedback from students, but says nothing positive about peer instruction. Indeed, the results *after* peer instruction are poor, so this could be taken as evidence of the failure of peer instruction, not the success.
Of course, one poorly-worded question should not be taken as useful evidence for or against peer instruction. After a few weeks, with many before-and-after questions, then maybe there would be some evidence that peer instruction helps (or simply spreads confusion and misinformation, which is one of my concerns with it).
27.
Mark Guzdial | January 21, 2011 at 1:33 pm
You have a fair concern. I don’t interpret Peer Instruction as “only the peer instruct.” After the second round of voting, I discuss the answers and why the right one is correct. I think that the goal of Peer Instruction is to combine what Roger Schank called “expectation failure” (i.e., learning is more likely to occur right after, “What?!? I was wrong?!?”) and the use of social learning, which is a lot less intimidating and more likely to get students’ attention (e.g., no Facebook nor Twitter) than the teacher just lecturing.
28.
Revisited: My Students Know Far Less « Computing Education Blog | January 21, 2011 at 1:52 pm
[…] 21, 2011 Beth Simon made an excellent recommendation after my report on my first Peer Instruction lesson: Was it really a bad question that students misinterpreted? Why not ask the students? You would […]
29.
Kieran Mathieson | January 22, 2011 at 9:55 am
A more general comment. There’s much research on learning in skills-oriented courses, like programming. A consistent recommendation from this research is to cover fewer topics, in more depth. Give students lots of exercises, and lots of feedback. Not just grades, but “here are the things you did wrong” (formative feedback).
So, there’s strong theoretical and empirical evidence that covering less content is, in fact, the right thing to do to improve skill depth. Covering more material would lead to shallower learning.
Is this a trick question? I’m not sure what “trick” means, but I see nothing wrong with the question. An algorithm some students use for answering multiple choice questions breaks down. That’s a good lesson for students.
Of course, not giving them choices – just ask them what would be in the array element – would make that faulty algorithm unusable. Still, reminding students that “careful thought is a Good Thing” is a Good Thing.
Kieran
kieran@coredogs.com
30.
Mark S | January 23, 2011 at 10:45 pm
I agree with the other commentors. This Q/A leaves a big hole for someone to step in, even people who know the right answer. That’s not to say that your coworkers won’t sometimes do this for you too, however, in the way they write their code.
Neverthelss, as an objective measure of knowledge, I prefer short answer over multiple choice. The power of suggestion is too great. (Although I understand the logistics of grading short answer.)
31.
Alfred Thompson | January 23, 2011 at 11:23 pm
With all do respect to others, I do not see how answer D makes any sense at all to anyone who read the question and has a clue. Sorry. I just can’t see it. I can understand getting it wrong with A but that’s about it. Maybe someone can explain it better or differently from the previous replies. Thanks.
32.
Kieran Mathieson | January 24, 2011 at 9:54 am
Got to agree. The question is not faulty, just because many people got it wrong.
33.
Students in CS like videos of Java coding « Computing Education Blog | May 4, 2011 at 10:17 am
[…] semester, I did a couple of new things in my Media Computation data structures class (as mentioned in a previous blog post). I started using Ubiquitous Presenter so that I could ask in-class questions, let students see the […]
34. Minconceptions and Teaching Them « SChaiken's Blog | August 5, 2011 at 11:37 am
[…] has an interesting earlier blog entry My Students Know Far Less Than I Ever Expected about being surprised at students beginning data structures knowing far less than he expected. […]
35.
Seth Chaiken | August 5, 2011 at 11:59 am
Having taught data structures many times, I am not at all surprised by the outcome of the problem, even after the class discussion.
The main misconception that the students have is about what they must pay attention to when they face examples of programs and questions about them in order effectively learn elements of the discipline. They must often attend to the same things of course when they write or debug their own or other peoples’ code. Even the professionals, who rightly observed that the problem is a “trick question”, fell for the trick, I guess, because they did not realize that they needed to turn on the mode of detailed code and question analysis rather than recognize a familiar pattern.
How can we teach “what it is you must pay attention to”, teach the habit to pay attention to that when it is appropriate, and teach how to recognize the situations where particular things to pay attention to are appropriate?
Should we tell the students that certain in-class questions are the kinds of questions that they must ask themselves while they are engaging in disciplinary work, necessary for that work to be effective?