Student Course Evaluations Can’t Measure Teacher’s Knowledge: But We Could

April 20, 2015 at 8:30 am 33 comments

It’s that time of year when Deans and Chairs start prodding students and teachers about course evaluations. What do the students think about their courses? What do the students think about their teachers?

There is a significant body of evidence that suggests that course evaluations are a stable measure about the teachers themselves. For example, the scores for a teacher are consistent across instantiations of the course over time (see Nira Hartiva’s work). However, they still might not be measuring something that we consider significant about teaching.

For example, it’s a mistake to think that student course evaluations tell us what a teacher knows about teaching. The teacher’s pedagogical content knowledge is invisible to the student. The student only sees what the teacher has decided to do to interact with the students. The student can’t see the knowledge that the teacher used in making that choice.

Three kinds of knowledge that are particularly relevant to a CS teacher are:

  • Knowledge about how to teach. A good teacher knows more than one way to teach a particular subject, and knows to change methods for a given student or to change the pace of a class. When I see students driving away in the back of my class, I know that it’s time for a Peer Instruction activity.
  • Knowledge about misconceptions. As was shown in Phil Sadler’s exceptional study (see blog post), a characteristic of teacher expertise is knowledge about what students typically get wrong. Based on that knowledge, teachers can address those misconceptions, and lead students to discover and correct the misconceptions themselves.
  • Knowledge about how to broaden participation in computing, which is particularly relevant to CS teachers. These include how to teach avoiding stereotype threat and triggering the imposter phenomenon, about how to give everyone a voice in the class and not let the loud and boisterous define the teacher’s perceptions of the course. I can offer a negative example, taken from real life but might have been invented after reading the negative examples in Unlocking the Clubhouse.

Teacher: How many of you students had Python in a previous class?
(Most students raise their hands, since it’s the language used in the pre-requisite class.)
Teacher: Well, you learned a terrible language. You’ll have to forget everything you know if you want to pass this class.
(Every student suffering imposter syndrome at this point decides to drop.)

This teacher actually has quite high course evaluation scores — and double the drop rate of every other teacher of that class.

Pedagogical content knowledge (PCK) is the key difference between novice and expert teachers, but is invisible to students. This is another reason why student evaluations of teaching (aka, Student Reviews of Instruction (SRI)) are inadequate as measures of teaching quality. They can’t measure a key indicator of teacher expertise.

I’ve been wondering how post-secondary teaching might change if we were to take a PCK perspective seriously. The knowledge of good teaching is definable and measurable.

  • We might define courses not just in terms of learning objectives but in terms of what knowledge the teacher should have to teach the class effectively.
  • We could evaluate University and College teachers based on their PCK — literally, taking a test on what they know about teaching the class.
  • PCK tests would finally create an impetus for University and College faculty to pursue professional development — that’s where they’d learn the teaching methods, student misconceptions, and methods for encouraging BPC that they need to answer the PCK tests. One might even imagine teachers taking a class on how to teach a new class that they’ll be offering in the future — preparing for a course by developing expertise in teaching that course. What an interesting thought that is, that higher education faculty might study how to teach.

Entry filed under: Uncategorized. Tags: , , , , .

BBC is giving away 1 million mini computers so kids can learn to code: Prediction — little impact on broadening participation Stanford president refines his vision for MOOCs in education

33 Comments Add your own

  • 1. astrachano  |  April 20, 2015 at 8:58 am

    There’s so much I like in this article. But not this: “We could evaluate University and College teachers based on their PCK — literally, taking a test on what they know about teaching the class.” We don’t need a test that teachers take, we need observation and reflection. I want to know how other teachers view you, not whether you know the latest results from having read research papers — if we had a validated test? Don’t see that, but I’ve been blind for years 🙂

    • 2. Mark Guzdial  |  April 20, 2015 at 10:16 am

      The knowledge of how to teach CS well is available outside of research papers, e.g., How People Learn is accessible and valuable, there are lots of workshops and how-to guides on how to use techniques like Peer Instruction and Peer Led Team Learning, and so on. You can’t see knowledge by observation. Reflection, sure — we could ask higher ed teachers to tell us what they thought about when planning a course, when leading a class, etc. I’d count that as one form of a test. My point is that we should value pedagogical content knowledge, and a focus on that can get us away from evaluating teaching as a student popularity contest.

      • 3. astrachano  |  April 23, 2015 at 3:44 pm

        It’s likely that we can’t see knowledge by observation, but you can see effective teaching by observation. PCK without pedagogical content practice? Maybe better than neither, but not better than being an effective practitioner. I’ve seen many less-than-effective teachers who actually know how (in theory) to teach but aren’t (in practice) effective

  • 4. Philip Sadler  |  April 20, 2015 at 10:13 am

    Agreed that PCK is a very important aspect of teaching to measure. As for those who are opposed to “testing,” remember that using assessments to measure one’s own knowledge can help monitor progress towards the goal of mastering one’s craft.

  • 5. Bri Morrison  |  April 20, 2015 at 10:28 am

    Your real life example brings up a point that I’ve often thought about – a teacher evaluation *before* drop date to include the opinions of those students. Or a separate teacher evaluation sent to students who did withdraw from the class – asking if teacher opinion / attitudes contributed to the drop. Wouldn’t those be interesting results for an administrator…

    • 6. Mark Guzdial  |  April 20, 2015 at 10:52 am

      *LOVE* this, Bri. From your lips (keys?) to God’s ears!

    • 7. chaikens  |  April 21, 2015 at 9:42 am

      Yes, that would be good! I wonder if the survey could distinguish students who drop a course because of a genuine impostor syndrome from those who know they don’t like to think and know that learning new ways and things is hard.

      Anecdotally, I’ve had a few students (all female) who expressed concern about whether they were fit for a course because they detected ambiguities or contradictions due to some simplifications in an introduction.

  • 8. Mike Zamansky (@zamansky)  |  April 20, 2015 at 12:13 pm

    A couple of thoughts — on student evals – they also lack perspective. A students view of a course or teacher can be very different once a few years removed.

    Interesting thoughts on PCK but I wonder about those teachers that are naturals – the best athletes frequently don’t know how or why they do what they do – they just do it. Same for some of the best teachers I know whereas a know some horrible teachers that could ace any PCK test you throw at them.

  • 9. Chris Goedde  |  April 20, 2015 at 12:48 pm

    I’ve been pushing for something similar here at the department level: that demonstrating familiarity with current education research in our field (physics, in my case), should be a requirement for promotion and tenure. We wouldn’t tenure or promote someone who didn’t keep up with the research literature in their field, why should we tenure or promote someone who does’t keep up with the research literature in teaching?

    Of course this doesn’t get to implementation, but small steps are better than none.

  • 10. Bonnie  |  April 20, 2015 at 1:17 pm

    Isn’t this just like the exams that K12 teachers take for certification? Testing on pedagogy doesn’t seem to have done much in that realm…

    • 11. Mark Guzdial  |  April 20, 2015 at 1:21 pm

      I don’t know how close it is to Praxis exams and the like. Having the tests might have some effect. I bet that the average K-12 teacher is a far better teacher than the average higher education teacher.

      • 12. Mike Zamansky (@zamansky)  |  April 20, 2015 at 1:25 pm

        Mark, but most of the best 9-12 teachers I know hold both the exams and ed programs they were in in low regard.

        • 13. Mark Guzdial  |  April 20, 2015 at 2:00 pm

          That may be a sampling bias, or it may be that the best teachers don’t need the exams and methods classes (e.g., the exams and methods classes may help bring up the lower half of the distribution), or it may be that it’s a good idea but a bad implementation.

      • 14. Bonnie  |  April 20, 2015 at 11:56 pm

        I have kids in a pretty elite district. I think my kids teachers are great, but no better than a professor in a good teaching oriented college (note that I do not say R1 institution). In fact, they do pretty similar things. I don’t think it has much to do with the certification exams. Rather, interest and drive, at both elite K12 and elite teaching oriented colleges, are the important thing.

  • 15. dennisfrailey  |  April 20, 2015 at 1:57 pm

    It seems to me that we should also be concerned about how well the students learn with a given teacher. A teacher may know plenty of theory but not be good at helping students learn (I’ve had a few genius professors who couldn’t actually help you learn at all). One measure I use is comments received several years after they graduate. I keep in touch with a lot of my students and I learn plenty after they’ve had a chance to work in a real job.

    I also find that the more mature the students the more insightful the feedback on instructor evaluations. I teach a lot of graduate students and often they are older than the typical college student. They tend to be better at understanding the difference between a class they enjoy and a class where they learn a lot.

    One common denominator I’ve heard is that good student assignments (where they apply what was covered in class) and exam questions that ask them to think rather than repeating what they’ve memorized tend to be the ones that students say helped them actually learn. Also, the quality of grading on student assignments and exams is frequently cited. Not just what was wrong but why it was wrong.

    • 16. Bonnie  |  April 20, 2015 at 11:59 pm

      I give extensive feedback on projects when I grade, and I find it disheatening to hear how unusual that is. How can anyone grade programs without lots of comments. Good programmiing is like good writing.

      • 17. chaikens  |  April 21, 2015 at 9:27 am

        In the last few programming courses I taught, we graded for whether the code was indented to make its structure obvious and whether the variable names were meaningful, and, even more importantly, not misleading. We (following examples from professionals, such as the Linux kernel code) de-emphasized the importance of comments and taught that a misleading comment is worse than no comment at all. I agree with people like Linus Torvalds that good code is understandable without comments.
        (Of course, interface specifications don’t fall in this category of comments!) Comments might serve the purpose of teaching that every line has a specific meaning (what it does) and has a purpose. I tried teaching with emphasizing my conscious purpose for every line of code and asking students to express such purposes in comments.

        • 18. Bonnie  |  April 21, 2015 at 12:08 pm

          I didn’t mean code comments – sorry if it read that way. I meant comments from the instructor – feedback on the student’s work

          • 19. chaikens  |  April 23, 2015 at 9:47 am

            Absolutely! Grading of code in beginning programming courses should be like grading of writing in essay courses, with plenty of comments addressing clarity, understanding, expression and style.

            • 20. astrachano  |  April 23, 2015 at 10:43 am

              This simply doesn’t work at schools with large enrollments. We have 300+ students in Compsci 101, getting our TAs or undergrad TAs to provide useful and cogent comments? It’s a dream, but in reality if the code works, that’s fabulous. We don’t expect students to be good writers when they start, we expect them to convey their thoughts effectively. “Good” programs for those coming at this the first time? They do what they’re supposed to, they don’t need to be extensible or beautiful.

              • 21. dennisfrailey  |  April 23, 2015 at 10:56 am

                I agree that it is hard to do this at schools with large enrollments. I’ve been on the teaching end of this (over 200 students in an operating systems or computer architecture class) and on the hiring end of this (where we hired students from humongous state universities and had to retrain them on how to produce supportable and maintainable code as opposed to code that just works.) This all points to the fallacy of the traditional way of evaluating faculty – they get credit for “contact hours” (meaning lecture hours) with the students rather than for the really valuable activities of grading homework with good quality feedback. This is one reason I’m hoping the concept of “course flipping” will be successful, although the barriers to this are substantial. For example, when I tried it I ran into the “not enough lecture hours” problem (recorded lectures were not deemed to count, and face time with the students helping them do their assignments correctly was not counted as “contact hours”). I even ran into bureaucratic problems trying to use competent TAs. I had a cadre a former students who could do a good job, but they were not hirable because they had graduated! I was required to utilize existing students. When you are teaching an upper level course, it’s hard to find students who’ve had the course, done well in it, and are still enrolled as students!

              • 22. gasstationwithoutpumps  |  April 23, 2015 at 11:00 am

                Combining the comments by chaikens and astrachano we see the current dilemma: the beginning courses are too big to have enough competent instructors to provide the feedback students need and the instructors for the advanced courses feel that students should have already learned how to write programs properly and that their job is a different part of the education. The result is that students can graduate with a BS having never had a competent programmer look at their code and tell them how to make it readable.

                My suggestion: everyone should do some feedback on programming style.

                In the huge classes, spend a couple of hours at the beginning of the course training the TAs and have them do some feedback. It won’t be great, but it will be a whole lot better than throwing up your hands and using just automatic grading of I/O behavior.

                In smaller classes, spend the time really going through the code pointing out problems and suggesting improvements. Spend some time in class telling students why good variable names are important, and why comments should be telling you how to interpret the variables (rather than trying to teach the programming language, which are about the only type of comments students have ever seen). By showing students that you really care about how they write, you can get them to make enormous improvements, even in only 10 weeks. It is a lot of work, but proper teaching is more about this sort of feedback than it is about delivering polished lectures. Especially in graduate classes, you should really be pounding on the students to write professionally and taking the time to give them the feedback that their previous instructors skimped on.

      • 23. dennisfrailey  |  April 21, 2015 at 12:31 pm

        I haven’t taught a programming course in over 30 years. Most of what I teach is either advanced topics in computer architecture, operating systems or compiler design or else software engineering topics, including management topics. I give extensive feedback when grading in all of these classes. It takes a lot of time but I think it is where a lot of the learning occurs because students pay a lot of attention to why they lost points on exam questions and assignments.

        In many of my classes I give them a chance to turn in a draft of a major assignment and get comments back before they prepare their final version. This simulates what typically happens in industry when you get peers to review your draft documents before you finalize them. Unfortunately, some students fail to take advantage of this opportunity – they turn in a cursory draft version because “it doesn’t count toward their final grade”. Of course, their final product is typically not as good as it could have been.

  • 24. guy  |  April 21, 2015 at 11:04 am

    I could not agree more with this post. I learned so much reading John Bruer’s book Schools For Thought: A Science of Learning in the Classroom. He gives a good overview and then goes into more depth of what we know about teaching math, science, reading, and writing, with a chapter dedicated to each. Oh how I’d love to see a chapter similar to those dedicated to introductory programming.

    • 25. Peter Donaldson  |  April 24, 2015 at 4:18 am

      Reading through some of the comments it appears there’s potentially a bit of confusion between PCK and subject knowledge. They are definitely not the same thing and probably why someone with a deep theoretical knowledge of a particular topic may be an atrocious teacher.

      Experts have a huge wealth of well organised knowledge in a particular area that’s so deeply ingrained it’s become unconscious and automatic. A teacher in some senses has to be able to decompile this expertise in order to provide a series of experiences that will move a novice towards more expert like behaviour. This is not the same as trying to get a novice to immediately act like an expert which unfortunately often results in cognitive overload and very little learning taking place.

      My own opinion is the key knowledge and understanding I need to develop as a teacher is what the developmental sequence for CS concepts is, what misconceptions learners may have or develop and what types of activities can help them develop particular ways of thinking. I also need to have a clear rationale for the wider benefits of the particular concepts I’m teaching and how they connect to the wider world.

      • 26. Mark Guzdial  |  April 24, 2015 at 11:38 am

        Thanks, Peter — you’re making a point that I wanted to raise in response to Owen and Dennis. Who said anything about “theory”? I’m talking about knowledge that informs practical choices. For example, we know that spaced practice is far better than massed practice, so having multiple quizzes or exams or other assessments is far better for learning than one or two major exams. Connecting across topics improves learning and transfer. Encouraging women in undergrad CS classes is critical to get more female graduate CS students.

        While I didn’t introduce the term “theory” in the discussion, I don’t understand pushing back against “theory” (e.g., knowledge about psychology, learning science, cognitive science, situated cognition, etc.) for teachers. Owen and Dennis, I bet that you both argue for the value of theory and discrete mathematics for CS undergraduates. Why is it not similarly useful for teachers to know something about relevant “theory”?

        Owen, I completely agree that observation is better than student opinion surveys for evaluating teachers. I don’t know that observation of teachers is the best way to measure teacher knowledge about teaching, which is the specific part of being a good teacher that I was raising in this post. I’d love to see if there’s some research that addresses that hypothesis.

  • […] language.) Her class was particularly unfriendly to women and members of under-represented groups (a story I told here). She now rejects the CS classroom culture, the “defensive climate” (re: Barker and […]

  • […] relieve faculty who teach from the responsibility to learn more about learning sciences (see my blog post about testing teachers about PCK). Just teaming up subject-matter experts with learning engineers does not inform a […]

  • […] I predict that if we did this study with CS teachers, we’d find the same result.  The belief that CS is for males and not for females is deeply ingrained in the perceptions of our field.  Kahneman would tell us that it’s part of our System 1 thinking (see NYTimes Book Review).  What do you think teachers would draw if asked to “draw a computer scientist“?  I predict that the gender bias that favors males as computer scientists would be greater for post-secondary teachers than for secondary or elementary teachers.  Most secondary school CS teachers that I’ve met are sensitive to issues of gender diversity in computing, and they actively encourage their female students.  Most post-secondary CS teachers with whom I’ve worked are not sensitized to issues of women in computing and have not changed how they teach to improve gender diversity (see example here). […]

  • […] the only feedback that a teacher gets is from student surveys, course-instructor opinion surveys.  That’s not going to help with this problem.  How could students know that the class would be better with peer instruction?  How could […]

  • […] who decide to try CS.  I’ve talked about some of the reasons in past blog posts (see post here about bad teaching practices and here about my daughter’s experience in CS at Georgia […]

  • […] Annie Murphy Paul is talking about inclusive teaching here, but she could just as well be talking about active learning.  The stages are similar (recall the responses to my proposal to build active learning methods into hiring, promotion, and tenure packages). These are particularly critical for computing where we have so little diversity and CS teachers are typically poor at teaching for diverse audiences. […]

  • […] one of our classes that I believed were likely impacting retention of female students (as mentioned in this blog post). I didn’t have any evidence that the practices were impacting retention. I critiqued the […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,184 other subscribers


Recent Posts

Blog Stats

  • 2,053,612 hits
April 2015

CS Teaching Tips

%d bloggers like this: