The Learning Process for Education Research

One of the more influential projects in physics education (and learning sciences over all) was the effort by Jill Larkin and Herb Simon to characterize how students and experts solved physics problems. They found that students tend to look at physics problems at a shallow level, while experts see deep structure.  Students tend to look at what variables are present in the problem, match them to the equations given in class, and see what they can compute with those variables.  Experts look at a problem and identify the kind of problem it was, then work out a process to a solution.

My son is currently taking AP Physics, and I’m seeing this same process when he asks me for help.  My dissertation work was about teaching students kinematics by having them build simulations, so I’m familiar with some of the content.  I’m no expert, but am a bit closer than my son.  Matt brought me a problem then started with, “I can figure out delta-Y here, but can’t see why that’s useful.”  He knew the equation that matched the variables.  I drew a picture, then figured out what we needed to compute.  I then remembered the wrong equation (evidence that I’m no expert) and came up with an answer that clearly couldn’t be right. (Kudos to Matt for realizing that!)  Matt then figured out the right equation, and we came up with a more reasonable answer. I worked from the problem situation to an equation, and Matt started by looking for an equation.

I’ve been seeing this same process lately in how people come to understand education research.  I’m teaching an undergraduate and graduate (joint) class on educational technology this semester.  (We just started class last week.)  In the first week, I had them read two chapters of Seymour Papert’s Mindstorms; the paper “Pianos, not Stereos” by Mitchel Resnick, Amy Bruckman, and Fred Martin; and Jeanette Wing’s “Computational Thinking.”  I started the class discussion by asking for summary descriptions of the papers.  A Ph.D. student described Jeanette’s position as “Programming is useful for everyone to understand, because it provides useful tools and metaphors for understanding the world.”  I corrected that, to explain that Jeanette questions whether “programming” is necessary for gaining “computational thinking.”  The student shrugged off my comment with a (paraphrased) “Whatever.”  For those of us who care about computing education, that’s not a “whatever” issue at all — it’s a deep and interesting question whether someone can understand computing with little (or no?) knowledge of programming.  At the same time, the student can be excused for not seeing the distinction. It’’s the first week of class, and it’s hard to see deep structure yet.  The surface level is still being managed. It’s hard to distinguish “learning programming” and “learning to think computationally,” especially for people who have learned to program.  “How else would you come to think computationally?”

This last week, we’ve been reviewing the findings from our first year of our Disciplinary Commons for Computing Educators where we had university and high school computer science teachers do Action Research in their own classrooms. Well, we tried to do Action Research.  We found that the teachers had a hard time inventing researchable questions about their own classrooms.  We ended up scaffolding the process, by starting out with experimental materials from others’ studies, so that teachers could simply pick the experiment that he or she felt would be most useful to replicate in his or her classroom.  We then found that the teachers did not immediately see how the results had any implication for their own classrooms.  It took us awhile to get teachers to even ask the questions: “The results show X (e.g., most students in my classroom never read the book).  What does that mean for my students?  Does that mean X is true for all my students?  Should I be doing something different in response?”

These results aren’t really surprising, either — at least in hindsight.  High school and university teachers have their jobs not because they are expert at education research.  University researchers typically are expert at some computing-related research, not computing education research, and a general “research perspective” doesn’t seem to transfer.  Our teachers were looking at the surface level, and it does take some particular knowledge about how to develop researchable questions and how to interpret results into an action plan afterwards.

Education research is a field of study.  I’ve been doing this kind of work for over 20 years, so you’d think I’d have realized that by now, but I still get surprised.  Simply being a teacher doesn’t make you an expert in education research, and being a domain researcher doesn’t make you an expert in education research in that domain.  It takes time and effort to see the deeper issues in education research, and everyone starts out just managing the surface features.

Feynman lectures from Microsoft: A medium for active essays and computing ed

I got to see the Project Tuva videos at the MSR Faculty Summit during the last session of the day Tuesday.  If you haven’t seen them, I recommend them to you.  (Though, as Ian Bogost found out, you have to have the latest version of Microsoft Silverlight to watch them.)  These are Richard Feynman’s “Messenger Lectures” which he delivered at Cornell and recorded by the BBC.  The Project Tuva site enhances the video with the ability to take notes, read others’ notes (synched to the video), see links (that appear at the appropriate points in the video), including links into simulations and even the Worldwide Telescope.  So, when Feynman talks about how stars are formed, you click and go see telescope imagery of new stars being formed.  When Feynman talks about Tycho Brahe, you go to a simulation of planetary orbits so that you can make your own velocity measurements.

While I’m a Feynman fan (as are most scientists, engineers, mathematicians, and computer scientists I know), I was more excited about the medium than I was the Project Tuva lectures themselves.  I’m still looking for the right medium (and authoring tools) to express ideas in computation.  Books don’t cut it, and running programs go too far the other way.

Expressing computation to students is as pedagogically complex as expressing quantum mechanics or electrodynamics to physics students.  Most physics educators whom I’ve asked list those subjects as the most challenging to teach since the phenomena are impossible to directly observe and the behavior is non-intuitive.  All of computing is like that!  I can convince you that Biology is really about cells with a microscope, and most of Physics and Chemistry is about explaining the phenomena that you see every day.

While we see computing everyday, the computation behind those applications is difficult to see.  Just how many for loops are necessary to write that email?  Did you see all those linked lists behind your Powerpoint slide deck?  Applications that we use daily are layers upon layers of computation, such that it’s nearly impossible to see the low-level computation that we want to teach in an introductory course.  This is the problem of not having a microscope for computing.

So, we use books with source code in them.  To imagine the execution of a piece of source code is perhaps the most important and most intellectually challenging goal of an introductory course.  If human intelligence is computable, then the Halting Problem comes into effect, and for some level of complexity, we cannot figure out the execution just by looking at the source code.  Books with source code are a reasonable way of talking about computation, once you have some level of ability to imagine execution.  For rank beginners, it’s almost cruel — it’s like saying, “Let’s have you learn Russian by throwing you into Moscow without a coat in January.  Better figure out what Russian means quickly!”

Alan Kay has argued for “active essays,” a kind of dynamic book with simulations built in.  Ted Kaehler, Mitchel Resnick, and Brian Silverman have all built some reall interesting active essays.  An active essay could expose the source code, with explanation, and allow for execution within the same medium.  As a form of scaffolding, perhaps the source code could be tweaked in some meaningful ways, so that students could see the relationship between changes to the source code and impact on dynamic behavior — which is the most important thing to learn in an introductory class!

The enhanced video mode of Project Tuva could offer a way of doing “active essays” where the base medium is video rather than text.  Active essays have simulations embedded within text explanations.  We could, however, have videos explaining concepts, with simulations embedded within the video.  That’s an intriguing notion.  As I’ve blogged previously, there is psychology evidence that lectures are better for explaining computing concepts than just reading a book. (Yes, I discovered that my old Amazon blog posts are still there, if I can find a direct link to individual posts.)  Maybe with the Project Tuva enhanced video mode, we could finally do the Metaobject Protocol book in a compelling way.

Now, speaking of “books not cutting it,” the post-review-process version of the Data Structures manuscript is due next Wednesday, and I’d better get to it before my co-author discovers I’ve been spending my writing time blogging (again).

Are we measuring what students are learning?

One measure of the success of a talk is how many questions you get in the hallway after the talk.  I got a few yesterday, which suggests that people were still thinking about the points afterwards.

One question I got was about a finding we’ve had in several of the contextualized computing education classes, like robotics and Gameboys for computer organization.  Students report spending extra time on their homework beyond what’s required “just because it’s cool.”  Yet, in some cases, there is no difference in grade distributions or failure rates compared to a comparison class.  What gives?  Isn’t that a bad thing if students spend extra time but it’s not productive time?

Absolutely, that can be the case.  It may also be the case that students are learning things that we don’t know how to measure.  Think about the argument that it takes 10,000 hours of practice to develop expertise (a number that has been recalculated from several sources).  Can we come up with learning objectives for each of those 10,000 hours?  Or is it that we can measure some of those objectives, but others of the items being learned are subtle, or are prerequisite concepts, or are about skills, or even muscle memory?

A famous story in physics education is about how concepts are more complex and have more  facets than we realize.  David Hestenes has developed some sophisticated and multi-faceted assessments for concepts like “force” — a whole test, just addressing “force.”  Eric Mazur at Harvard scoffed at these assessments (as he said at a AAAS meeting I went to a couple of years ago, and quoted in a paper by Dreifus in 2007).  His Harvard students would blow these assessments away!  Gutsy man that he is, he actually tried them in his classes.  His students did no better than the averages that Hestenes was publishing.  Mazur was aghast and became a outspoken proponent of better forms of teaching and assessment.

Building up these kinds of assessments takes huge effort but is critically important to measure what learning is really going on.  For the most part in Computing Education, we have not done this yet.  Grades are a gross measure of learning, and to progress the field, we need fine-grained measures.

Aligning Computer Science with Mathematics by Felleisen and Krishnamurthi

The July 2009 Communications of the ACM has an interesting article by Matthias Felleisen and Shriram Krishnamurthi Why Computer Science Doesn’t Matter with the great subtitle “Aligning computer science with high school mathematics can help turn it into an essential subject for all students.”  The argument that Matthias and Shriram are making is that we can use programming to teach mathematics better, and help students learn both.  In so doing, we prevent marginalization of computer science and support an important goal of the American education system, the teaching of “‘rithmetic.”

It’s a good argument and one that I support.  I am dubious about some of the education claims made in the argument, like “we have already seen our curricular approach…help students raise their alebra scores” and “Formal evaluation shows the extremely positive impact this curriculum has…”  (I’ve been searching through the sites given in the article, but can’t find peer-reviewed, published papers that support these claims.)  But these are really minor quibbles.  Having written one of these pieces, I know that the tyrany of 1800 words is severe, and the authors can be excused for not providing background citations to the studies supporting their claims.  Instead, I’d like to provide some of the background literature that supports their claim.

Can we use programming to help students learn mathematics?  Absolutely!  Probably the most famous study supporting this argument is Idit Harel’s dissertation work on Instructional Software Development Project (ISDP). Idit had fourth graders write software in Logo to teach fractions to third graders.  She found that a real synergy occurred between the concepts of programming and the concepts of mathematics, and her students ended up learning more about both compared to another class.  Yasmin Kafai (who just moved to Penn from UCLA this last year) continued this project, exploring novel collaboration models (e.g., the fourth graders become fifth grade “consultants” as another cohort of fourth graders helps another cohort of third graders) and expanding from mathematics into science.  My own dissertation explored the synergy between physics and programming.  My results weren’t as strong — I had good physics learning, but not good computer science learning.  I suspect the problems were the challenge of making real learning in only a three week summer workshop, and not having the kinds of IDE’s that Matthias and Shriram are calling for.

“Our community must realize that minor tweaks of currently dominant approaches to programming won’t suffice.”  Completely agreed, and the best argument for this point came from Bruce Sherin’s powerful dissertation (with Andy diSessa at Berkeley).  Bruce taught two groups of students lessons in physics, one using programming and one using algebra.  (Andy would probably argue with Matthias and Shriram, “The ideal language and the IDE for imaginative programming are still to be designed.”  Over 20 years ago, Boxer implemented much of what they’re calling for.)  Bruce found some really interesting differences between what was learned via each form of notation.  For example, programming was better for figuring out causality and sequencing. An algebraic formula like x = x0 + vt leaves invisible to the novice that the t is what will typically vary in that equation.  On the other hand, algebra was better for understanding balance and equilibria.  A formula like F=ma works in both directions:  increase the mass or acceleration and the force increases, or if the force declines, then either the mass or the acceleration must have declined.  Most programming languages do not make evident how constraints work in the world.  The media extensions that Matthias and Shriram describe help address some of the challenges Bruce found when students had a single physics concept (e.g., an object moving because it’s location changed) being represented by multiple lines of code.

“As computer science educators, we must also demand a smooth, continuous path from imaginative programming to the engineering of large programs.”  Alan Kay has been making this argument for years.  He refers to the concept of Omniuser who can move from scripting in E-Toys, to changing how the levels close to the metal of the machine work, all with a single system and (hopefully) a single notation.  His STEPS effort is seeking to build such systems. In particular, Alan and his team are exploring “systems math” which is a kind of powerful mathematics that can only really exist in the powerful medium of programming.  Thus, STEPS gives us a way to go beyond just support “‘rithmetic” to support powerful new kinds of mathematics learning.

I’m a big fan of Scheme and consider DrScheme to be one of the finest pedagogical IDE’s ever created.  TeachScheme is a brilliant curriculum.  My guess is that careful studies of the effort would support many of the claims being made by Matthias and Shriram.  More importantly, though, I believe that they’re right that programming could actually improve mathematics learning.  Doing it in such a way that students’ mathaphobia doesn’t drive even more students from computer science is a real challenge.  An even bigger challenge is doing it in such a way that can gain the support of organizations like NCTM and that meets the mathematics standards in our schools. As they say, “Any attempt to align programming with mathematics will fail unless the programming language is as close to school mathematics as possible.”  It’s more than just the programming language — the whole package (curriculum, IDE, language) has to look and feel like mathematics to make it successful with the mathematics education community.