## The Learning Process for Education Research

One of the more influential projects in physics education (and learning sciences over all) was the effort by Jill Larkin and Herb Simon to characterize how students and experts solved physics problems. They found that students tend to look at physics problems at a shallow level, while experts see deep structure.  Students tend to look at what variables are present in the problem, match them to the equations given in class, and see what they can compute with those variables.  Experts look at a problem and identify the kind of problem it was, then work out a process to a solution.

My son is currently taking AP Physics, and I’m seeing this same process when he asks me for help.  My dissertation work was about teaching students kinematics by having them build simulations, so I’m familiar with some of the content.  I’m no expert, but am a bit closer than my son.  Matt brought me a problem then started with, “I can figure out delta-Y here, but can’t see why that’s useful.”  He knew the equation that matched the variables.  I drew a picture, then figured out what we needed to compute.  I then remembered the wrong equation (evidence that I’m no expert) and came up with an answer that clearly couldn’t be right. (Kudos to Matt for realizing that!)  Matt then figured out the right equation, and we came up with a more reasonable answer. I worked from the problem situation to an equation, and Matt started by looking for an equation.

I’ve been seeing this same process lately in how people come to understand education research.  I’m teaching an undergraduate and graduate (joint) class on educational technology this semester.  (We just started class last week.)  In the first week, I had them read two chapters of Seymour Papert’s Mindstorms; the paper “Pianos, not Stereos” by Mitchel Resnick, Amy Bruckman, and Fred Martin; and Jeanette Wing’s “Computational Thinking.”  I started the class discussion by asking for summary descriptions of the papers.  A Ph.D. student described Jeanette’s position as “Programming is useful for everyone to understand, because it provides useful tools and metaphors for understanding the world.”  I corrected that, to explain that Jeanette questions whether “programming” is necessary for gaining “computational thinking.”  The student shrugged off my comment with a (paraphrased) “Whatever.”  For those of us who care about computing education, that’s not a “whatever” issue at all — it’s a deep and interesting question whether someone can understand computing with little (or no?) knowledge of programming.  At the same time, the student can be excused for not seeing the distinction. It”s the first week of class, and it’s hard to see deep structure yet.  The surface level is still being managed. It’s hard to distinguish “learning programming” and “learning to think computationally,” especially for people who have learned to program.  “How else would you come to think computationally?”

This last week, we’ve been reviewing the findings from our first year of our Disciplinary Commons for Computing Educators where we had university and high school computer science teachers do Action Research in their own classrooms. Well, we tried to do Action Research.  We found that the teachers had a hard time inventing researchable questions about their own classrooms.  We ended up scaffolding the process, by starting out with experimental materials from others’ studies, so that teachers could simply pick the experiment that he or she felt would be most useful to replicate in his or her classroom.  We then found that the teachers did not immediately see how the results had any implication for their own classrooms.  It took us awhile to get teachers to even ask the questions: “The results show X (e.g., most students in my classroom never read the book).  What does that mean for my students?  Does that mean X is true for all my students?  Should I be doing something different in response?”

These results aren’t really surprising, either — at least in hindsight.  High school and university teachers have their jobs not because they are expert at education research.  University researchers typically are expert at some computing-related research, not computing education research, and a general “research perspective” doesn’t seem to transfer.  Our teachers were looking at the surface level, and it does take some particular knowledge about how to develop researchable questions and how to interpret results into an action plan afterwards.

Education research is a field of study.  I’ve been doing this kind of work for over 20 years, so you’d think I’d have realized that by now, but I still get surprised.  Simply being a teacher doesn’t make you an expert in education research, and being a domain researcher doesn’t make you an expert in education research in that domain.  It takes time and effort to see the deeper issues in education research, and everyone starts out just managing the surface features.

Entry filed under: Uncategorized. Tags: , , .

• 1. Leigh Ann  |  August 24, 2009 at 5:36 pm

This reminds me of the thread on the SIGCSE list about whether we needed CS Education PhD’s.

I think one of the most significant moments in my time here so far has been something small that my psychology advisor (Sharon Carver) said to me. She said that I had a tendency to report the “that” and not the “what”. As a teacher you were always about the that. Did you teach that? Did they understand that? and did you give that assessment and what were the specific results of that?

The larger research question then is tying in the “what”. What are the cognitive principles that underlie the patterns that you are seeing? how does “that” tell you something about education or cs education in general? what are the larger issues at play here in the classroom.

This sounds like the problem that the teachers – and also a little bit of your students were having. They were taking the research or articles at face value and not connecting them to the larger WHAT that we are trying to understand as a community.

My current personal reflective work is on how I can better incorporate the what into my thinking, planning, and writing process. So far it has made me feel like I am writing more professional research proposals and really understanding how I fit into the bigger picture of a discipline.

• 2. Hélène Martin  |  August 25, 2009 at 5:20 pm

I have a strong feeling that the very act of teaching should be an act of research. After all, if we aren’t constantly asking questions about the effectiveness of what we are doing and refining our ways given the answers, then we probably could be replaced by a videotape of ourselves and that’s not very good news.

That being said, I have a hard time making research questions about my teaching explicit. I know I’m constantly adjusting based on various cues but it’s hard to disaggregated all the things that contribute to those cues and say “aha, here is a generalizable principle about how my students best learn to conceptualize and use nested loops appropriately.” Maybe I try a different activity next time or spend more time doing exercises on paper or I have students try to tackle problems in pairs but at the end of the day I never know whether the problem was what I taught, that the students were hungry the first time around, that the room was too hot… I think that extreme uncertainty discourages a lot of teachers from even attempting research. I tend to get bogged down in externalities and then no research question looks very good!

Do you have examples of questions that DCCE teachers ended up tackling? I notice the wiki has empty resource and example categories but maybe that material exists somewhere else? Pointers to the experiments questions were scaffolded from would also be helpful just to get some ideas.

• 3. Mark Guzdial  |  August 25, 2009 at 9:56 pm

There were two rounds of investigations in our DCCE, and two groups in each round. They looked at some code comprehension questions, some array multiple choice questions like in Lister’s 2006 ITiCSE working group, some questions about comfort in CS1, and some attitudinal questions.

I do agree that a good teacher is always trying something new and is always asking whether or not that worked. My concern is what teachers accept as evidence. I think most teachers trust comments from trusted or best students and use their comments as an indicator for the entire class. I’ve been involved in studies where a teacher claims that X is true, based on discussions with students and TA’s, but I can show surveys that show X is false for the majority of the class. The concern then is that the teacher is making decisions that please the vocal minority, not the quiet (but measured) majority. It’s that reliance on measurement, rather than talking with students (teacher-as-probe), that I think is hard for many teachers who try to adopt an action research perspective.

• 4. Hélène Martin  |  August 26, 2009 at 12:55 am

That’s very interesting. I wonder if maybe teachers do this because they don’t know about the right tools? With Google docs surveys, Survey Monkey and the like, getting course-wide feedback should be a joy. I think it’s particularly eye-opening to do a quick survey with the same questions after each assignment: how many hours did this take? How hard was it from 1-10? How frustrating was it from 1-10? How much did you learn from 1-10?

• 5. Leigh Ann  |  August 26, 2009 at 11:45 am

Mark and Helene,

I’m actually taking a course in Social and Decision sciences and from the first readings it seems like education is not the only place where that kind of skewed perception of outcome occurs. Primacy and recency effects have strong impact on our judgment of events, and arousal (in terms of what draws our attention) also plays a key part.

Helene – a lot of what you are talking about sounds like action research – the only real difference between AR and what teachers do on a daily basis is the documentation and continual focus on what action is to be taken next to change what you see.

Sometimes its hard to find the balance between a teacher’s gut instinct and what the appropriate level of evidence in a classroom should be.

• 6. Raymond Lister  |  August 28, 2009 at 5:22 pm

A small and possibly unnecessary correction — Mark wrote above about “Lister’s 2006 ITiCSE working group”. I did have a working group in 2006, but I think in this case Mark is referring to my 2004 working group … Lister R. et al. (2004) A Multi-National Study of Reading and Tracing Skills in Novice Programmers, SIGCSE Bulletin, Volume 36, Issue 4 (December), pp. 119-150. http://portal.acm.org/citation.cfm?id=1041624.1041673&coll=&dl=&CFID=15151515&CFTOKEN=6184618

• 7. Mark Guzdial  |  August 28, 2009 at 7:45 pm

Yes, that’s the paper I meant, Ray. Thank you for the correction, and sorry I missed the year. (Nice of you to come by!)

This site uses Akismet to reduce spam. Learn how your comment data is processed.