Computer scientists need to understand education research methods for CE21

At the CE21 meeting earlier this month, I got asked a similar question more than once.  “I have got this great class on X for high school teachers.  I want to ‘evaluate it’.  Um…how many teachers do I need?”  I’m pretty sure I really heard the quote marks around “evaluate it,” because I’m pretty sure that the question-asker really had little idea what that meant.

I used this story as an example in my educational technology class last week.  It’s worth exploring why that’s not answerable as-is.  “How many teachers do I need?” depends on the research question that you’re trying to answer.  There are lots of questions one might ask about a “great class for high school teachers.”  Which one are you trying to explore?

• Maybe you think you’ve solved a particular problem that high school teachers face in learning computer science, like struggling with data structures or fitting the course material into their daily lives.  I’m particularly interested in that latter problem.  To answer that question, you need to talk to the teachers, to get an understanding of whether the teachers faced the problem and if your class helped them get past it.  You’re not going to interview 20 people and do something useful with your data (interview transcripts).  At least 3-5 people, probably no more than 10-12 participants would let you answer your question.
• Maybe you think that your class in X is better than other classes in X.  Then, you need to do a comparison study.  My rudimentary knowledge of statistics suggests that you need 40-50 teachers with about half taking each course so that you can compare them on some learning or performance measure.
• Maybe you think that your class can scale dramatically well, that you really have a solution to the CS10K challenge — your class can educate thousands of teachers in the next four years.  That’s great, but to be convincing, you’re going to show that you can run your class at scale (maybe 100 teachers at once would be convincing) and that you still achieve learning outcomes (against some reasonable measure of learning, like Allison’s test or the outcome measures being developed for CS:Principles).  You don’t need to do a comparison to something else if you’re trying to demonstrate scale, and you certainly aren’t going to interview all those participants.

There are other possible research questions, with other appropriate evaluation mechanisms.  Do you think that your intervention is going to result in systemic change?  Then you need a longitudinal study.  Do you think that you have a class that will draw more teachers into CS teaching?  Then your real target audience is outside your classroom, and you need to do an evaluation that extends outside your classroom.

The greatest challenge facing the CE21 community is that the community is filled with computer scientists. Computer science too rarely asks questions involving human beings, so we have too little practice defining the right kinds of methods. The CE21 meeting had a few education researchers, who they seemed not too comfortable with computer science — and there was way too little collaboration between the two groups.  If we want to do education research that means something, we need to learn how to to ask research questions that involve humans and to figure out the right methods.