What Convinces CS Faculty to Change: Authority over Evidence
July 15, 2015 at 7:21 am 27 comments
My Blog@CACM Post for July 2015 is on the Top Ten Myths of Teaching Computer Science. You can go take a take a look at it here.
I wrote that blog post because we really have had a long debate in our faculty email list about many of those topics. I recently saw our Dean at an event, and he told me that he hadn’t read the thread yet (but he planned to) because “it must be 100 messages long.” Most of the references in that blog post came from messages that I wrote in response to that thread. It was a long post because people generally didn’t agree with me. Several senior, well-established (much more famous than me) faculty strongly disagreed with the evidence-based argument I was making. The thread finally ended when one of the most senior, most respected faculty in the College wrote a note saying (paraphrased), “There are probably better teaching evaluation methods than the ones we now use. I’m sure that Mark knows teaching methods that would help the rest of us teach better.” And that was it. Thread ended. The research-based evidence that I offered was worth fighting about. The word of authority was not.
I’ll bet that faculty across disciplines similarly respond to authority more than evidence. We certainly see the role of authority in Physics Education Research (PER). Pioneering PER researchers were not given much respect and many were ostracized from their departments. Until Eric Mazur at Harvard had his students fail the Force Concept Inventory (FCI), and he changed how he taught because of it. Until Nobel laureate Carl Wieman decided to back PER (all the way to the Office of Science Technology and Policy in the White House). Today, the vast majority of physics teachers know research-based teaching methods (even if they don’t always use them). FCI existed before Mazur started using it, but it really started getting used after Mazur’s support. The evidence of FCI didn’t change physics teaching. The voice of authority did.
While we might wish that CS faculty would respond more to evidence than authority (see previous post on this theme), this insight suggests a path forward. If we want CS faculty to improve their teaching and adopt evidence-based practices, top-down encouragement can have large impact. Well-known faculty at top institutions publicly adopting these practices, and Deans and Chairs promoting these practices can help to convince faculty to change.
Entry filed under: Uncategorized. Tags: educational psychology, learning sciences, teachers.
27 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
Computing Ed Research - Guzdial's Take 
1.
dennisfrailey | July 15, 2015 at 9:09 am
None of this is at all surprising. We’re dealing with human nature. It has been known for a long time that human beings tend to ignore, question or reject evidence that disagrees with their established thinking, ideology, religion, or other beliefs and to over-rate evidence that supports their beliefs. Think about how difficult it was for the theory of plate tectonics to gain acceptance. Authority, indeed, often makes the difference. Wallace had proposed the theory of natural selection to widespread disdain. Darwin, on the other hand, had rank and privilege and authority so he had an easier time of it and, in the end, got the lion’s share of the credit.
Scienists are by nature skeptical. It takes a lot to turn hypothesis into accepted theory and a lot more to admit changes to accepted theory. Non-scientists are similar. Computer scientists are humans and, at least in some cases, scientists. Why should our behavior be any different?
2.
gasstationwithoutpumps | July 15, 2015 at 11:42 am
Since I don’t like the ACM requirement that I create yet another user account to comment on blogs, I’ll comment on your article here instead. You gave as evidence that CS high school teaching can be effective “We can use student performance on the AP exams (which the College Board works hard to make roughly equivalent in difficulty across subjects) as one metric. Here’s a statistic that AP CS teachers have been talking about in social media. Worldwide in 2015, there was 1 perfect AP Calculus Level BC (harder one) exam (i.e., one student in the whole world got every question right on the Calc BC exam), 2 perfect AP Statistics exams, and 3 perfect AP Calculus Level AB exams this year. There were 66 perfect AP CS Level A exams.”
There is a myth here—that the College Board tries to make the possibility of a perfect score equally hard on all exams. Even if they were trying to make the AP exams “roughly equivalent in difficulty” (which is demonstrably not the case, as Calculus AB and Calculus BC are deliberately at different levels of difficulty, as are Physics 1 and Physics C), they have no real interest in the extremes of the distribution (all right and all wrong). They are interested in the main bulk of the distribution, not the tails (which are almost impossible to match anyway).
I agree with you that high school teaching can be effective, that many high school students are interested in and capable of learning to program, and that we should be encouraging more CS education in K–12. But the weakness of your evidence in this argument weakens my confidence in the strength of your evidence in the other arguments. There must be better evidence.
I also had some qualms about your pro-clicker argument, which seemed to be based on one anecdote about a grad student being a better liked and better teacher with clickers than a tenured professor without clickers. Again, I’m sure that better evidence is out there—you weaken your argument by bringing up the anecdote. (Note: any study based on one or two classrooms of students is an anecdote—way too much of education research seems to be based on blessing some anecdotes and vilifying others, without gathering enough data to really tell them apart—confirmation bias is a big problem in the field.)
3.
Mark Guzdial | July 15, 2015 at 11:51 am
The pro-clicker argument isn’t based on an anecdote — it’s based on an article in Science. The PNAS meta-review also includes data on clickers.
4.
gasstationwithoutpumps | July 15, 2015 at 4:18 pm
Your description sounds like an anecdote to me. “In a study in Science in 2011 (see summary paper here), Nobel laureate Carl Wieman taught a post-doc and a graduate student some active learning methods, then had them teach one week of a section of Physics while a tenured physics professor taught the comparison section. The post-doc and grad student had better learning and better student motivation (e.g., more students came to lectures) than the one taught by the physics professor.”
That Science published the anecdote doesn’t raise its value for me. You’ve been arguing against appeals to authority anyway.
5.
Mark Guzdial | July 15, 2015 at 4:34 pm
I tried to write it in an accessible way. The Science paper is here: http://www.sciencemag.org/content/332/6031/862.abstract
I said that faculty respond to authority, not evidence.
6.
gasstationwithoutpumps | July 15, 2015 at 8:23 pm
I just read that article. It was a one-shot experiment with no replicates and lots of uncontrolled variables (different teachers of different ages, for example). That strikes me as little more than an anecdote. A few dozen such comparisons would begin to be research-based evidence.
7.
Mark Guzdial | July 16, 2015 at 8:13 am
Okay, now I think you’re just yanking my chain. You know that the whole point of that study was to show that novice teachers could beat out an expert teacher when using active methods, so of course they’re not going to control for the teacher.
You’re also well aware that education research rarely uses randomized control trials. Human subjects research isn’t like physics or engineering research. We do large n studies, or we do meta-analyses where they look across many studies. One of the large n studies for active learning methods was the Richard Hake 6000 person study, and the PNAS study that I referenced is a large meta-analysis of active learning studies with dozens of studies (158). This is science, the best that education knows how.
Or are you just complaining about clickers? I can give you similar studies about clickers, or just peer instruction in CS studies.
I wrote a blog post, Kevin. Not a dissertation. If you want to see all the evidence, follow the links and read the papers. It’s all there.
8.
gasstationwithoutpumps | July 17, 2015 at 12:45 am
If you want to convince skeptics, then pointing them to the Hake study and the PNAS meta-analysis of dozens of studies is likely to be more convincing than the Science article which is a single short trial with lots more going on than just a change in clicker usage.
You don’t need to convince me about the value of active learning—I make all my classes more about the lab work than lectures (my applied electronics class has 6+ hours a week in the lab with me, and the 3.5 hours of lecture are mainly in support of the lab work—like learning to use gnuplot to analyze the data collected the day before in lab).
But if you are trying to convince skeptics, you need to bring out the strong arguments, not the weak ones. And the CACM blog is more likely to be read by skeptics than the Computing Ed blog is.
9.
Mark Guzdial | July 17, 2015 at 8:29 am
You’re right, Kevin. I was relying on people traversing the links — I linked to the Wired article, which is all about the PNAS article, which references the Hake article. The skeptics especially aren’t going to go link-diving. I’ve edited the Blog@CACM posts to include links directly to the Hake and PNAS papers. If this piece gets picked up in CACM, it’ll have the revised form. Thank you!
10.
Ken Bauer | July 17, 2015 at 9:10 am
I was just going to say this is great feedback. I do hope this post and articles in CACM are of the sort that get educators to initially agree or disagree and then choose how far to dive in as part of a reflective exercise about their own teaching methods.
I really need to stop by here more often; thanks to all for good talking points which I am able to bring back into my discussions with faculty here locally.
11.
alfredtwo | July 15, 2015 at 9:37 pm
Some years ago Dartmouth did some studies on clickers (more or less) and came to the same conclusions about how well they worked. I’m sorry I don’t have links to papers as I am remembering this from discussions with the researchers. And it’s been a while.
12.
gasstationwithoutpumps | July 15, 2015 at 11:49 pm
I’m perfectly willing to accept that evidence for the efficacy of clickers exists—I’m just complaining that the evidence Mark provided was very weak, and unlikely to convince any skeptic. Presenting weak evidence is often worse than presenting no evidence, as it can lower the confidence of the listener in the authority of the speaker. If the listener thinks that the proponent is arguing from biases rather than from real evidence, then they’ll reject most of what the proponent says, even if there is real evidence that would have convinced them.
Weak arguments act as vaccination against stronger arguments, and there have been so many weak arguments from the education community that many faculty members are immunized against any arguments from them.
13.
Bonnie | July 15, 2015 at 1:05 pm
I hope you are not saying that deans and chairs should be pushing pedagogical change from the top down. That is a recipe for disaster. In fact, many would say that is one of the central problems in K12 – everytime a new superintendant comes in, a new set of teaching practices get imposed on the teachers, who learn to quietly ignore them.
On the other hand, if what you mean is that deans and chairs should be empowering faculty to make changes, then that is a good idea. But that means changing the environment and structure so that faculty can engage in improved teaching practices. It may mean giving faculty release time to put in the hours required to change a course. It may mean accepting that student evaluations are likely to worsen as faculty try new teaching practices. It may mean getting into the weeds with the students to help them understand why they are going to be expected to do more than listen to a lecture. It may mean spending money to change classroom configurations, or hiring assistants or changing the registration systems. It may mean accepting that publishing output may go down while changes are happening. The administration could really do a lot to bring about change, but it seems like at many schools they are content to simply tell faculty to do better without fighting for the structural changes that would make it possible.
One problem with using evidence instead of authority is that there really are not that many really definitive studies out there clearly saying “approach X is best”, and even the ones that do exist may only be relevant for settings similar to the settings where the studies were carried out. I have seen a number of papers that show that a particular approach works well, based on studies carried out at large research oriented institutions. How well do these approaches translate to smaller schools with less well prepared students? How about community colleges? I think faculty would be more prepared to accept evidence if the evidence was actually strong – studies replicated across a variety of insitutions, with randomized trials, and accounting for the placebo effect.
Finally, you may not realize the extent to which faculty out in the trenches are actually adopting better practices. I teach at a school that emphasizes teaching, with classes that are capped at 25 students. I don’t know of anyone in my department who simply lectures. We are all aware of peer instruction, project based learning, worked examples, code reading exercises, and flipped classrooms, and use a variety of approaches other than simply lecturing. Way back in the 90’s, several of us (I was at another teaching oriented school back then) had an NSF grant to incorporate active learning methods across our department, so even back then, faculty were aware of the need for change. Maybe the problem is mostly at the large research oriented schools?
14.
Mark Guzdial | July 15, 2015 at 1:39 pm
Deans and Chairs should promote evidence-based best practices. That’s not the same as requiring it. I believe that your school adopts best practices, Bonnie, but I suspect that that’s rare. Poor teaching exists at all levels, and is probably pretty common. Over on Facebook, I’m having a battle with a set of high school teachers who are insisting that they are excellent lecturers and that makes them great teachers.
15.
Bonnie | July 15, 2015 at 2:06 pm
No, promoting is not good enough. Administrators need to make the changes that will allow faculty to adopt best practices. Two big barriers are time, and fear of student blowback. Administrators could really help with that, by making the time available, and by promoting the changes to the students, rather than to faculty. I have had colleagues at other teaching oriented institutions tell me that tenure trackers are routinely advised to stick to lecturing until tenure, because a drop in student evaluation scores could be lethal. This is something that chairs and deans can fix. The time issue is also something that administrators can fix by providing adequate release time. It takes a lot of effort to radically change teaching strategies and make the changes work well. Faculty are already reeling with increased assessment paperwork, constant meetings, and of course, getting research out (even at teaching oriented institutions).
16.
Ken Bauer | July 16, 2015 at 9:07 am
Thanks for your comments here Bonnie, I am indeed at a large teaching oriented institution and working (from the bottom) on enabling change. Even with top level promotion of innovation, their is real work to be done at the multiple levels of management between those leaders and the educators. I don’t have the solution here but am working on it. The point I keep making to lead administrators is the risk/benefit balance is off since the risk is very often completely in the educator’s corner with very little benefit to putting innovation into practice.
17.
dennisfrailey | July 15, 2015 at 9:05 pm
I recall some advice from a colleague when I was a new assistant professor. “There are two steps to making changes in the academic world. Step 1: Prove beyond a shadow of a doubt, with solid evidence, that the change should be made. Step 2: Wait for all the tenured faculty members to retire.”
18.
Mark Guzdial | July 16, 2015 at 10:11 am
Love this, Dennis! There’s an engineering education research newsletter whose motto is: “Achieving reform in engineering education. One tombstone at a time.”
This is what Kuhn said — it’s how paradigm change happens.
19.
carpetbomberz | July 16, 2015 at 1:12 pm
Attrition does have it’s benefits.
20.
alfredtwo | July 15, 2015 at 9:42 pm
Do we have recognized authorities in teaching computer science? Well let me rephrase that. We do but are they known in the broader community? Probably not. Plus we don’t have enough research. But as you point out even with the research a lot of people prefer to believe their own, usually highly filtered, observations, theories and ideas over the research. Since we don’t have the same standards for authority in high school CS – nothing like a tenured full professor in high school teaching – and because there is often some friction between HS and university faculty an argument by authority can be hard to make.
21.
slger | July 29, 2015 at 12:00 pm
Speaking of authority, one more skill/knowledge requirement has, at long last, been recognized by the big guys in computing:
http://teachingaccessibility.com/
How will faculty learn to teach accessibility?
Susan
blog AsYourWorldChanges.wordpress.com
22.
Mark Guzdial | July 30, 2015 at 12:20 pm
I saw that, Susan, and am confused by it. I don’t know who at Georgia Tech is teaching accessibility. It shows up in our HCI classes, but when I suggested teaching about universal design in our required courses, I was met with resistance.
23.
Why we are teaching science wrong, and how to make it right: It’s about CS retention, too | Computing Education Blog | August 3, 2015 at 7:50 am
[…] new paper in Nature that makes the argument for active learning in all science classes, which is one of the arguments I was making in my Top Ten Myths blog post. […]
24.
Interesting Pushback Against Incentivizing Active Learning in CS Classes | Computing Education Blog | August 21, 2015 at 8:24 am
[…] faculty talking about it, or should it be done by administrative fiat? I lean toward the latter. As I’ve pointed out, CS faculty tend to respond to authority more than evidence. The administration should do the right […]
25.
Evidence-resistant science leaders: When Authority goes the Wrong Way | Computing Education Blog | September 2, 2015 at 8:55 am
[…] story in the blog post connects to my previous blog post about CS faculty arguing against doing something other than lectures in their classes. Here the […]
26.
Growing evidence that lectures disadvantage underprivileged students | Computing Education Blog | September 18, 2015 at 8:44 am
[…] I wrote a Blog@CACM post over the summer about the top ten myths of computing education, which was the top-visited page at CACM during the month of July (see post here). I wrote that post in response to a long email thread on a College of Computing faculty mailing list, where I experienced that authority was able to sway CS faculty more than research results (blog post about that story here). […]
27. A Year of Beauty | Nick Falkner | January 1, 2016 at 1:02 am
[…] throughout history and is now supported in this ‘gilt age’ of measurement of efficacy. It still took local authority to stop people piling onto him (even under the rather shabby cloak of ‘scientific […]