Bret Victor’s “Inventing on Principle,” and the trade-off between usability and learning

I have had several people now send me a link to Bret Victor’s video on Inventing on Principle. It is a really impressive demo!

His system reminds me of Mike Eisenberg’s work on SchemePaint.  Mike wanted the artist to be able to interleave programming and direct manipulation.  In SchemePaint, you could draw something by hand, then store the result in a variable to manipulate in a loop.  Or you could write some code to tesselate some graphical object, then add tweaks by hand.  It was beautiful.  The work that Mike did on SchemePaint led to his wonderful work on HyperGami, a CAD system for origami, which was the start of his Craft Technology group. That’s the group from which Leah Buechley graduated — she did the LilyPad.

People are sending me Bret’s video asking, “Wouldn’t this be great for learners?”  I bet it could be, but we’d have to try it out. At one point in his lecture, Bret says, “Why should I have to simulate the computer in my head?”  Because that’s the point of understanding computer science.  Bret’s system looks like a powerful visualization system, and visualization can be used to lead to real understanding, but it isn’t easy to design the visualization and context such that learning occurs.

The problem is that visualization is about making information immediate and accessible, but learning is about changes in the mind — invisible associations and structures.  Sometimes good usability makes it easier to make these associations and structures.  Tools like Scratch and Alice increase usability in one direction (e.g., syntax) while still asking students to make an effort toward understanding (e.g., variables, loops, and conditionals).

My first PhD student was Noel Rappin, who explored the features of modeling environments that lead to learning.  He had a CHI paper about his work on helping chemical engineers learn through modeling.  Our colleagues in chemical engineering complained that their students couldn’t connect the equations to the physical details of the pumping systems that they were modeling. Noel built a system where students would lay out the physical representation of a pumping system, then “look underneath” to see the equations of the system, with the values filled in from the physical representation (e.g., height difference between tanks).

He ran a pilot study where students would lay out a system according to certain characteristics.  They would then manipulate the system to achieve some goal, like a given flow rate at a particular point in the system.  When Noel asked the pilot students if they gained any new insights about the equations, one student actually said, “What equations?”  They literally didn’t see the equations, just the particular value they were focusing on.  The system was highly usable for modeling, but not for learning.

Noel built a new system, where students could lay out a model, and values from the model were immediately available in an equation space.  To get the flow rate, the student would have to lay out the equations for themselves.  They would still solve the problem by manipulating the physical representation in order to get the right flow rate, and the system would still do all the calculations — but the students would have to figure out how to compute the flow rate.  The system became much harder to use.  But now, students actually did learn, and better than students in a comparison group.

Bret’s system is insightful and may have some terrific ideas for helping learning.  I’m not convinced that they’re new ideas yet, but an old idea in a new setting (e.g., JavaScript) can be powerful.  I worry that we get too entranced by improvements in usability.  In the end, learning is in the student, not in the system.

The syntax problems in high school CS

This is a nice account of personal struggles with syntax getting in the way of playing with computer science ideas.  The stories are believable, about facing a course in Basic that was more about memorizing history, about a course in Pascal that made Basic seem simple in comparison, and about layers of languages leading to layers of complexity.  I also believe the idea that Scheme, with its minimal syntax, lets the learner forego memorizing “rules” of syntax.

However, I don’t buy the subtitle: “Simple language makes learning complex concepts easy.”  Simple language means that you don’t struggle with complex language.  Hard ideas in computer science are hard ideas.  Hard ideas in a complex language may be impossible for some students.  Hard ideas in a simple language means that you can focus on the right things–it’s easier, but I don’t buy that just the language alone makes complex CS ideas easy.

Simple Language Makes Learning Complex Concepts Easy

The thing I liked most about taking this course in Scheme was that I knew that I was learning something. Every concept I had ever tried and failed to understand comprehensively in my Pascal class – searching and sorting procedures, recursion, processing binary trees – was made clear when I studied them in Scheme. These things occur so naturally in Scheme that I couldn’t help but understand. After mastering the concept, I could then go back into my Pascal class and easily master the code. The point here is that concepts like these are universal in computer science. After you understand them, then you can learn the rules of any language in order to encode them. But it doesn’t matter how well you have mastered the syntax of a language if you don’t understand the meaning of what you are typing or the reason why it works.

via High School Computing: The Inside Story | Institute for Mathematics and Computer Science.

Trip report on ITICSE 2011: Robots for girls, WeScheme, and student bugs in Scratch.

ITICSE 2011 was a fun, interesting event in Darmstadt, Germany last week.  Here’s a brief report on my experience of ITICSE, completely biased and centered on the events that I attended, and only highlighting a handful of papers.

Ulrik Schroeder’s opening keynote (slides available) focused on the outreach activities from his group at Aachen University, with some high-quality evaluation.  The most interesting insight for me was on their work with Lego Robotics.  I raised the issue that, in our GaComputes work, we find that girls get more excited about computing and change their attitudes with other robots (like Pico Crickets or Pleo Dinosaurs) more than Lego Robotics.  Ulrik agreed and said that they found the same thing.  But boys still like and value being good at Lego Robotics, and that’s important for their goals.  He wants to find and encourage the girls who do well at the same robots that the boys like.  He wants the girls to recognize that they are good at the same CS that the boys do.  It’s a different goal than ours — we’re more about changing girls’ view of CS, and they’re more about finding and encouraging girls who will succeed at the existing definition of CS.

I went to a paper session on A Tool to Support the Web Accessibility Evaluation Process for Novices.  They had a checklist and rubric, including role playing (what would an older user do with this site? A sight-impaired user? Someone whose native language isn’t English?) to use in evaluating the accessibility of a website.  I liked the tool and was wondering where the same model could be used elsewhere.  I got to thinking during this talk: Could we do a similar tool to support the design of curriculum that encourages diversity?  Could we provide checklists, rubrics, and role plays (How would a female CS student respond to this homework write-up?) to help faculty be more sensitized to diversity issues?

The coolest technology I saw was WeScheme — they’ve built a Scheme-to-JavaScript compiler into a Web page, so that students can hack Scheme live from within the browser.  I was less impressed by the paper presentation.  They’re using WeScheme in a middle school, where the kids are doing code reviews (“which most undergraduate programs in the US don’t do”) and presenting their work to “programmers from Facebook and Google.”  Somebody asked during Q&A, “How do you know that most undergraduate programs don’t do code reviews?”  They had no evidence, just an informed opinion.  I’m worried that the paper as a model for outreach.  Are Facebook and Google programmers willing to visit all middle schools in the US?  If not, this doesn’t scale.  Nonetheless, the technology is amazing, and I expect that this is the future of programming in US schools.

Probably the paper that most influenced my thinking was Orni Meerbaum-Salant’s paper on Habits of Programming in Scratch (same session).  They studied a bunch of students’ work in Scratch, and identified a number of common misconceptions and errors.  What was fascinating was that the bugs looked (to me) a lot like the ones that Elliot Soloway found with the Rainfall Problem, and the issues with concurrency were like the ones that Mitchel Resnick found with Multilogo and that John Pane found with HANDS.  That suggests that changing the environment doesn’t change the kinds of errors students are making.  And since all student programming misconceptions come from our instruction (i.e., students don’t know much about programming before we teach them programming), it means that we’ve been teaching programming in basically (from a cognitive perspective) the same way since Pascal.

The paper reporting on a multi-year professional development effort in Alice was really interesting.  They had lots of great stories and lessons learned.  The most amazing story for me was the school district where, not only were the CD/DVD players disabled, but the IT staff had used glue guns to fill in the USB ports on the school computers.  The IT administration wanted there to be no way for teachers to load new software onto those computers.  How depressing and frustrating!

All the papers in the session on Facilitating Programming Instruction were thought provoking.  Paul Denny’s project measures how much thrash and confusion that students face in figuring out Java — and there’s a lot of it.  Shuhaida Mohamed Shuhidan (“Dina”)’s dissertation work is yet another example of how little students understand about even programs as simple as “Hello, World!”  I really liked that Matt Bower is exploring how learning a second language can influence/interfere with the first language learned, but I was disappointed that they only used self-report to measure the influence/interference.  Without any kind of prompt (e.g., an example program in the first language), can you really tell what you’ve forgotten about a first language?

My keynote went well, I thought (slides available).  I talked about CS for non-majors, for professionals who discover CS late in life, and for high school students and teachers.  After lunch the third day, I headed off to Aachen University to visit with Ulrik’s group, so I didn’t get to see more.  IITICSE was a lot of fun and gave me lots of good ideas to think about.

Matthias and Gordon win SIGCSE 2011 Honors

I’ve been meaning to post this note from Dan Joyce to the SIGCSE Members list.  Congratulations to Matthias and Gordon! Both well-deserved honors!

It is my pleasure to announce the 2011 SIGCSE award winners.

The SIGCSE Award for Lifetime Service to the Computer Science Education Community has been awarded to Gordon Davies, Department of Computing, Open University (retired).

The SIGCSE Award for Outstanding Contribution to Computer Science Education has been awarded to Matthias Felleisen, Trustee Professor at College of Computer Science, Northeastern University. (Ed note: Known for his TeachScheme approach, DrScheme, and the new Racket environment.)

Professor Felleisen will give a plenary address at the Technical Symposium.

Professor Davies will address the First Timer’s luncheon at the Technical Symposium.

These awards will be presented at the 42nd SIGCSE Technical Symposium being held in Dallas, Texas, USA, March 9-12, 2011.

I am sure you will join me in congratulating these outstanding educators.

For more information about these awards, including lists of previous winners, please see http://sigcse.org/programs/awards

Dan Joyce
Vice Chair SIGCSE

 

The Core of Computer Science: Alan Kay’s “Triple Whammy”

My aunt and uncle were in town last week.  My aunt told Barb and me how many of her friends’ computers were “destroyed” by “watching a YouTube video.”  “It almost happened to us, too, but we got a phone call telling us not to watch that video!”  Sure, there is probably a website out there that can trick users into installing a virus that can cause damage to their computer, and it may have a video on the website.  But I have a hard time believing that simply watching a video on a website like YouTube might “destroy” one’s computer (or more specifically from her explanation, erase one’s hard disk).  Belief that that could happen seems like a belief in magic and mythology, like the belief that a chariot draws the sun across the sky.  We ask everyone to take classes in history and biology, because they should understand how their world works, whether or not they will major in those fields.  It’s part of being an informed citizen who does not believe that the world runs by magic and myths.  What does everyone need to know about computer science?

Alan Kay and I were having an email conversation about this question, about what was the core of computer science that everyone ought to know about, even non-majors.  He came up with a “triple whammy” list that I really like.  It may need som re-phrasing, but there’s something deep there.  I’m copy-pasting his notes to me (repeated here with his permission) in italic-bold, with my intepretation and commentary between.

It is all about the triple whammy of computing.
1. Matter can be made to remember, discriminate, decide and do

In his book Pattern on the Stone, Danny Hillis points out that modern day CPU’s are just patterns on stone, essentially the stuff of sand.  We are able to realize YouTube and eBay and natural language translation and Pixar movies all because we can make patterns on stones that can remember things, distinguish between options, act on those distinctions, and do things from playing sounds to actuating robots.  This feels like magic, that matter can do those things, but mechanical engineers would find this first step unsurprising.  They know how to make machines made out of matter that can do these things, even without modern computers.  Whammy #1 is an important step away from magic, but isn’t yet computer science.

2. Matter can remember descriptions and interpret and act on them

In step 2, we get to programs and programming languages.  We can describe processes, and our matter can act on those descriptions.  While we can do this with steam engines and mechanical engineering, it’s complicated and not obvious.  We do this regularly in computer science.

3. Matter can hold and interpret and act on descriptions that describe anything that matter can do.

This third step is amazingly powerful — it’s where we go meta.  We can also describe the matter itself as programs.  Now we can create abstractions on our programming languages.  Now we can point out that any program can be written in any programming language.  This doesn’t directly address my aunt’s misconceptions, but if she understood the third whammy, we could talk about how a badly written media player could interpret a nefariously designed video such that the video could instruct a too-powerful media player to trash a hard disk, but how unlikely that would be.  This third step is where we get to the role that computer science can

The Triple Whammy isn’t all of computer science.  There is a lot more than these three steps.  For example, I think that everyone should know about limits of computability, and about the possibility of digitizing information in any medium (thus allowing for visualization of sound or auralization of stock market data).  I do see the Triple Whammy as part of a core, and that this could fit into any CS1 for any student.

We definitely talk about steps 1 and 2 in the Media Computation CS1, and parts of step 3. For example, we define a simple line-drawing language, then build an interpreter (just does each line-drawing statement) and a compiler (generates the equivalent Python function or Java method) for that line drawing language.  We do that in order to explain (in part) why Photoshop is faster than Python for any image filter we create.  But we definitely do not do this explicitly yet.  As I’m working on the Powerpoint slides for the Python 2ed book now, I’m thinking about building a “Triple Whammy” slide deck, to encourage teachers to have this discussion with their students in a Media Computation context.  I’ll bet that TeachScheme already gets there.

What I really like about this list is that it clearly explains why Computer Science isn’t just advanced use of application software.  We see adults and kids in our studies all the time who tell us that somebody really good at Photoshop is a computer scientist.  We hear from teachers and principals regularly who tell us that they teach computer science, because here’s the book they use for introducing Excel and Access.  The Triple Whammy is about computer science and not about using applications.

Playing the Cards You’re Dealt: A Story of GT and HtDP

A colleague of mine sent me a link to a piece written by Matthias Felleisen with his explanation for why Georgia Tech switched from his text How to Design Programs (HtDP).  Matthias is a visionary educator who is the winner of this year’s ACM Karlstrom Award for contributions to education, for which I congratulated him here.  I wrote a commentary on an article that they published last year in CACM, and I posted here Matthias’s response. I admire HtDP.  The story (as I know it) of why Georgia Tech moved away from HtDP connects to some themes that I want to talk about.

Georgia Tech taught using HtDP from 2000 (or thereabouts) through 2003 (when we started our MATLAB and Media Computation classes) until 2004 or 2005.  We used HtDP in the course that everyone on campus was supposed to take.  We moved away from HtDP for two reasons mostly: (1) Some majors (like all of the Colleges of Liberal Arts, Architecture, and Management) were failing the course at higher than 50% rates. (2) Engineering faculty, particularly, wanted us to teach something that could be used in their classes, and they weren’t willing to learn Scheme.  We had a cheating scandal that led to a Campus Task Force, which recommended that we abandon the one-size-fits-all strategy that we had used previously. (We already knew that were were going to have to abandon that strategy, and been planning on it.  The Task Force forced the issue.)  We developed the new courses after this decision was made. I had been playing with the idea of media in CS1 for awhile, including a Scheme-based version. This happened after our Founding Dean, Peter Freeman, left Georgia Tech, while Ellen Zegura was Interim Dean, and before Dean Rich DeMillo arrived.  The decision to build the new courses was Jim Foley’s, with encouragement from Kurt Eiselt, for which I’m grateful. (I told this story in the May 2009 CACM.)  We explored using Scheme and Squeak in our MediaComp class, but ended up going with Python (discussed in a 2005 SIGCSE paper).

In the original plans for the Media Computation course, I did think that we would reduce the cheating rate.  I was wrong.  Students cheat in computing courses, and Media Computation is no exception.  We did design the Media Computation course explicitly to be relevant, and Lauren Rich led an effort to interview women in the MediaComp course and in the HtDP course, to test whether we achieved that goal.  (I think that’s where the “making up lists of friends” comment came from.)

We did continue to use HtDP for the CS majors until Kurt Eiselt left for a better job at UBC (not fired). The decision to switch to Python made by lecturers who had taken an HtDP workshop, but probably were influenced by the new Dean who did not like Scheme in our intro course. Except for Kurt, I think that all the lecturers who took HtDP training are still here.  When David Smith wrote the MATLAB text for our Engineering CS1, he explicitly aimed to make it “HtDP in MATLAB.”  Today, we teach our CS majors using Python and the robotics approach developed by IPRE. People at Georgia Tech value HtDP, even if we don’t teach it.

For me, here is the critical part of the story.  When we switched from HtDP to Media Computation (for Liberal Arts, Architecture, and Management majors), the pass rate rose dramatically.  (I can’t claim that they learned the same — that’s what Allison Tew is trying to measure.)  When we switched from HtDP to MATLAB (for Engineering majors), the pass rate rose.  And when we switched from HtDP to Python (for CS majors), the pass rate rose again.

Why did that happen?

Lots of things might have influenced our outcomes, and I’m sure that many variables did.  I know that Kurt now feels that we didn’t implement HtDP correctly, and I am sure that he’s right.  What I’m interested in exploring here are the factors influencing the students’ decision to give up on a class or stick with it, which influences those pass rates.

At the end of their Senior year, our Computational Media majors write a long term paper about what they learned over their four years here and what they will carry with them into their career.  Of the 25-30 papers I graded this year, maybe 5-7 were wonderful and insightful.  Two common themes among the best of them: (1) They “got” Ian Bogost’s “Introduction to Computational Media” class for introducing them to different ways of thinking about programming. (2) They appreciated the course where they had to do team software development for the first time, using a language that they didn’t know previously (Smalltalk).

Then there are the rest.  The students who say that appreciate the course where they first used Maya or Adobe After Effects “because I’ll use that a lot in my career.”  My least favorite was the student who said that she most appreciated the “Digital Video Special Effects” course (a really tremendous course, BTW) because the instructor used a Macintosh, and the student only used Windows  Her great insight on the course was that the student was better off knowing how to use a Mac. We are giving Computational Media degrees to students who find the point of a class is to learn to use a Macintosh. These students talked about Smalltalk differently: as something that wasn’t used by anyone and was frustrating.

The one that has me thinking the most talked about CS1315, the Introduction to Media Computation class.  She said that she felt it was useful, led to skills that could be used in future classes.  She felt it gave her insight into how other applications she used worked.  But she also felt that learning “Jython” was a waste of time, because she’s never used it in another class.

On the one hand: Yay!  She totally gets the course and why we do it!  It’s that last part that I have been thinking about.  Jython is Python. Learning Python is useless knowledge for a Computational Media major?  This is the only essay that mentioned Python or Jython.

When I told this story to my colleague, Amy Bruckman, she said that we should really have these students write these essays five years later.  She’s right, of course.  The Seniors really have little idea what’s going to be useful to them later on in their career.

The problem is that students are making decisions now based on what they believe is going to be important to them.  Thus, students abandon classes that they perceive as being irrelevant to them.  When we switched from Scheme to something else, students stuck with the classes and didn’t give up.

Is this a Georgia Tech phenomenon?  Georgia Tech has had a vocational flavor since its creation, often being referred to as “The North Avenue Trade School.”  Maybe our students are particularly attuned to relevance in their future life.  Maybe students at MIT or Harvard are more willing to trust their faculty and pursue classes that will be useful to them, even if the students don’t see the value or relevance.

I think the pursuit of relevance is common today.  The rhetoric of College has changed.  Did you read about President Obama’s Commencement Speech at Hampton College this last weekend?

For much of the 20th century, a high school diploma was a ticket to a solid middle class life. That is no longer the case. Jobs today often require at least a bachelor’s degree, and that degree is even more important in tough times like these. In fact, the unemployment rate for folks who’ve never gone to college is over twice as high as it is for folks with a college degree or more.

Students seek a bachelor’s degree for a job and a middle class life.  That’s why the President is telling them to go to college.  If students perceive that the courses don’t lead to that goal, they will seek other classes.  Yes, it’s short-sighted.  Yes, they don’t know what they need to know.  But those are the students we have, these are the cards we are dealt, and that’s what American leaders are sending us.  We have to make the case for relevancy, because students have choices. They do need their bachelor’s degree, and we are preparing them for successful careers. But our job is harder now, because we have to make the case.

What CS teachers know: PCK for CS Ed

I’m teaching educational technology again this semester (along with a senior design class — it’s a two-fer semester).  We always read chapters from How People Learn which is an amazing book for providing a gentle introduction to a wide swath of learning theory.  We just read a section introducing pedagogical content knowledge (PCK):

Shulman (1986, 1987) argues that pedagogical content knowledge is not equivalent to knowledge of a content domain plus a generic set of teaching strategies; instead, teaching strategies differ across disciplines. Expert teachers know the kinds of difficulties that students are likely to face; they know how to tap into students’ existing knowledge in order to make new information meaningful; and they know how to assess their students’ progress.

via How People Learn: Brain, Mind, Experience, and School.

This section is always a hotbed for discussion among my students.  There are two main strands of discussion that come out of this section.

First, it’s pretty clear that higher education explicitly disbelieves in the existence of PCK.  Instead, we act upon the belief that knowing a lot about your domain (e.g., getting a Ph.D. in a topic, having an on-going research program) makes you qualified to teach in post-secondary education — arguably the most important level of education for addressing social inequality.  It’s an important job, and colleges and universities don’t do much to make sure that the job is done well.  When we get to this part of the chapter, the students launch into tirades (that I try to moderate) about the poor teaching quality that they find in college, and worse, the lack of interest in making it better. That’s not a computing-only issue, of course.

The second theme, which I emphasize more than the students, is the challenge of identifying PCK for computing education.  I’ve spoken to people at several places that are trying to put together degree programs or certification/endorsement programs around computing education.  In all of these efforts, they need “methods courses” — the courses where teachers are taught PCK.  It’s not obvious what the content is for those methods courses.

So, what would you teach a new teacher about teaching PCK for computing education?  Sure, you’d teach that you can use the spring-loaded cafeteria tray/plate holders as a model for a stack — that’s clearly part of our methods, part of the PCK for computing education.  Everybody uses that example.  What else?  I think there is a lot to include in such a class, from pair programming as a learning method, to the work in Commonsense Computing to define baseline understanding, to contextualized computing education as an organizing principle, to tools and IDEs and interactions between them (like Barb’s Alice-integration-with-Java approach).  There is a lot of teaching lore associated with a particular method, such as the procedures and processes of TeachScheme, and Logo teaching methods like “Ask-two-before me” (which is how I learned it in the 1980’s, which I know see as “Ask 3 before me.”)  The challenge is drawing together these disparate pieces into a cohesive whole.

Question Everything: How We Teach Intro CS is Wrong

I’ve been interested in John Sweller and Cognitive Load Theory since reading Ray Lister’s ACE keynote paper from a couple year’s back.  I’ve wanted to learn more, and as all professors will tell you, the best way to learn something (or at least, to find time to learn it) is to teach it. So, I assigned several papers on the topic (first three papers in the References) to my educational technology class.  Those three papers set me on a paper finding-and-reading spree which is having a dramatic effect on how I think about we teach about computing.

Let me start out with a strawman position of how we teach introductory computing.  I know it won’t be how you and your institution teach computing, but I hope that it’s got some elements of how most American schools teach CS1:

We lecture students on how to use a particular element of computing, say variables, assignments, and function calls.  We walk through 2-3 examples of using those elements, and assign reading from a textbook that might present 1-2 of those examples with maybe a couple others.  We then assign a programming assignment that the students have never seen before whose solution will require use of those programming elements, often with authentic tools or IDEs like those used by experts.

The original 1985 Sweller and Cooper paper had five studies with similar set-ups.  There are two groups of students, each of which is shown two worked-out algebra problems.  Our experimental group then gets eight more algebra problems, completely worked out. Our control group solves those eight more problems.  As you might imagine, the control group takes five times as long to complete the eight problems than the experiment group takes to simply read them.  Both groups then get new problems to solve. The experimental group solves the problems in half the time and with fewer errors than the control group. Not problem-solving leads to better problem-solving skills than those doing problem-solving. That’s when Educational Psychologists began to question “learning by doing” and the idea that we should best teach problem-solving by having students solve problems.

The paper by Kirschner, Sweller, and Clark is the most outspoken and most interesting of the papers in this thread of research. Their title states their basic premise: “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.”  They describe the problem like this:

On one side of this argument are those advocating the hypothesis that people learn best in an unguided or
minimally guided environment, generally defined as one in which learners, rather than being presented with essential information, must discover or construct essential information for themselves (e.g., Bruner, 1961; Papert, 1980; Steffe & Gale, 1995). On the other side are those suggesting that novice learners should be provided with direct instructional guidance on the concepts and procedures required by a particular discipline and should not be left to discover those procedures by themselves (e.g., Cronbach & Snow, 1977; Klahr & Nigam, 2004; Mayer, 2004; Shulman & Keisler, 1966; Sweller, 2003).

What exactly is minimal instruction?  And are they really describing us? I think this quote describes how we work in computing education (as modeled above) pretty well:

There seem to be two main assumptions underlying instructional programs using minimal guidance. First they challenge students to solve “authentic” problems or acquire complex knowledge in information-rich settings based on the assumption that having learners construct their own solutions leads to the most effective learning experience. Second, they appear to assume that knowledge can best be acquired through experience based on the procedures of the discipline (i.e., seeing the pedagogic content of the learning experience as identical to the methods and processes or epistemology of the discipline being studied; Kirschner, 1992).

That is, “people should learn to program by constructing program from the basic information on the language, and they should do it in the same way that experts do it.”  The paper then goes on to present all the evidence showing that this “minimally-guided instruction” does not work.

After a half-century of advocacy associated with instruction using minimal guidance, it appears that there is no body of research supporting the technique. In so far as there is any evidence from controlled studies, it almost uniformly supports direct, strong instructional guidance rather than constructivist-based minimal guidance during the instruction of novice to intermediate learners.

Now, in the same issue of Educational Psychologist there were rebuttals to KSC, and there have been more written since then, including one by my colleague Cindy Hmelo-Silver.  What’s striking about these rebuttals is that they basically say, “But not problem-based and inquiry-based learning! Those are actually guided, scaffolded forms of instruction.”  What’s striking is that no one challenges KSC on the basic premise, that putting introductory students in the position of discovering information for themselves is a bad idea! In general, the Educational Psychology community (from the papers I’ve read) says that the model I describe above is an ineffective way to teach introductory students.

So what’s wrong with my strawman?  First, way too few examples.  The original Sweller and Cooper paper used ten examples of just a simple algebraic manipulation.  In the studies where worked examples have been used with computer programming (yes, it has been used with computer programming, and yes, it works there, too) by Pete Pirolli and Mimi Recker, the examples were in the 6-12 range for something like recursion.  2-5 examples are too few.

Second, we should not be asking students to solve something new.  They should practice the same information, demonstrate that they got that.  What KSC shows is that that approach takes too much time and leads to too little learning — it overloads the cognitive ability of the learner.  Only the very best students can succeed in when simply thrown in front of a speeding interpreter or compiler.  (I think that explains the “two humps” pretty darn well — it’s not the discipline, but how we teach it that results in the bimodal distribution.)

What should we do instead?  That’s a big open question.  Lots of options have been explored in this literature, from using tools like intelligent tutors to focusing on program “completion” problems (van Merrie¨nboer and Krammer in 1987 got great results using completion rather than program generation). I think that a lot of TeachScheme with DrScheme language levels has many of the right characteristics.  Media Computation provides the right number of examples with minimal changes during practice, but still deals with the speeding interpreter problem. The bottomline is that asking students to program doesn’t lead to them learning programming — as Richard Mayer (famous educational psychologist put it) says it promotes behavioral activity too early in the learning process, when learners should be cognitively active. (How often have you seen students programming without thinking much?)  We need to figure out what’s the right kind of practice for CS, and how much, and how to motivate students to take seriously examples and practice, as opposed to full programming.

This literature is not saying never program.  Rather, it’s a bad way to start. Students need the opportunity to gain knowledge first before programming, just as with reading.  Later, there is a expertise reversal effect, where the worked example effect disappears then reverses.  Students do learn better with real programming, real problem-solving.  There is a place for minimally guided student activity, including programming.  It’s just not at the beginning.

Finally, it turns out that textbooks are not the best medium for teaching programming.  The Atkinson, Derry et al. article does a wonderful job of surveying what we know about teaching with examples.  One of the findings is the Modality Effect.  Using text to explain something that is visual (like a program) leads to more extraneous cognitive processing than using audio for the explanation.  Given that we work with computers capable of multimedia, we should be able to do much better than textbooks.

Overall, I find this literature unintuitive.  It seems obvious to me that the way to learn to program is by programming.  It seems obvious to me that real programming can be motivating.  But KSC respond to this, too.

Why do outstanding scientists who demand rigorous proof for scientific assertions in their research continue to use and, indeed defend on the bias of intuition alone, teaching methods that are not the most effective?

This literature doesn’t offer a lot of obvious answers for how to do computing education better.  It does, however, provide strong evidence that what we’re doing wrong, and offers pointers to how other disciplines have done it better.  It’s a challenge to us to question our practice. It’s up to us to use these lessons and models to improve our teaching practice.

References

• Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. W. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70, 181–214.
• Kirschner, P. A., Sweller, J., and Clark, R. E. (2006) Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist 41 (2) 75-86
• Sweller, J., & Cooper, G. A. (1985). “The use of worked examples as a substitute for problem solving in learning algebra”. Cognition and Instruction 2 (1): 59–89. doi:10.1207/s1532690xci0201_3.
• Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006) Hmelo-Silver, Duncan, & Chinn. (2007). Educational Psychologist, 42(2), 99–107

Moving the Needle

I’m teaching Educational Technology this semester (joint undergraduate and graduate section). This last week, we read two chapters from Noss and Hoyle’s 1996 Windows on mathematical meanings: Learning cultures and computers which describe what happened to Logo: From Mindstorms, to the backlash against “curriculum-free” education, to the Pea and Kurland papers. I hadn’t realized until I re-read those chapters and visited the LCSI website just how negative the term “Logo” had become. LCSI, the company Seymour Papert co-founded to create Logo implementations, is now a “global leader in constructivist educational technology” and sells “Microworlds.”  The word “Logo” only appears in the footer of the web page, to explain the “L” in “LCSI.”

Since the session topic was the challenge of educational reform, I also told the story of Media Computation and my recent blog post about teachers unadopting (or maybe “differently adapting”?) aspects of Media Computation. After class, one of the students approached me to ask, “If educational reform is so hard to achieve, why do you do it?” It was a great question that took me a couple days to answer, by realizing that there’s a faulty base assumption behind the question.

Logo is not a failure. It’s an enormous success.  There isn’t an educational technology project in the world that hasn’t been influenced in some way by Seymour and Logo. However Noss and Hoyle are completely correct — Seymour’s vision has not been realized.

There are two ways to think about the purpose of educational research. The first is all-or-nothing — you put together a complete intervention, think through as many aspects of the educational ecosystem that you can, and aim for other classrooms to act just as you envision.  If you’re building a reform intervention, you have to aim for this. Teachers have hard jobs, and classrooms are complicated places. Few would try Media Computation if we didn’t have software and textbooks and syllabi and Powerpoint slides.  The second purpose is moving-the-needle, and that’s a much more important goal.

Education is a worthy domain of research and investigation. The goal of educational researchers is to speak to the community, to add ideas to the conversation, and to change how the community thinks about a problem. To change the conversation, to change practice, and to influence people’s thinking about an educational problem — that’s moving-the-needle.  We publish to try to convince people about the value of our all-or-nothing vision, but our success is about moving-the-needle.

It’s damn hard to hear about teachers I respect giving up on some of the ideas that I hold dear, that I am convinced are better practice and for which we have empirical support. As a member of the research community, I will continue to argue for some of those ideas. (To the faculty who are giving up on having students saving their picture artifacts, in favor of focusing on the code: Think again about what the words “media” and “communication” mean!)  On the other hand, I take pride in evidence that our work is moving the needle.

This week, I spent some time visiting the MediaScheme website at Grinnell. Sam Rebelsky, Janet Davis, and colleagues completely buy into Media Computation and the value for their students. They just won’t give up on Scheme. They are creating their own libraries (to control GIMP from Scheme), writing their own textbook, and even creating their own IDE in order to do Media Computation in Scheme.  From an all-or-nothing perspective, this is a complete rejection of the libraries, textbooks, and IDE that we have created.  From a moving-the-needle perspective, I am completely delighted and flattered that they are making such effort around an idea that we could contribute to the conversation.

So finally, I can answer the student’s question: We do it because we believe we have a vision for improving practice.  We measure to convince others, and if we’re honest, to convince ourselves (or accept that we are wrong). But others are going to take our ideas, agree with some and disagree with others, and adopt what they see are the best ideas and adapt the rest.  Something new will result, hopefully better. It’s not about all-or-nothing. The goal is to be part of that struggle to figure out the best practice, to contribute to the structure of tomorrow’s classrooms and schools.  It’s all about moving the needle.

Aligning Computer Science with Mathematics by Felleisen and Krishnamurthi

The July 2009 Communications of the ACM has an interesting article by Matthias Felleisen and Shriram Krishnamurthi Why Computer Science Doesn’t Matter with the great subtitle “Aligning computer science with high school mathematics can help turn it into an essential subject for all students.”  The argument that Matthias and Shriram are making is that we can use programming to teach mathematics better, and help students learn both.  In so doing, we prevent marginalization of computer science and support an important goal of the American education system, the teaching of “‘rithmetic.”

It’s a good argument and one that I support.  I am dubious about some of the education claims made in the argument, like “we have already seen our curricular approach…help students raise their alebra scores” and “Formal evaluation shows the extremely positive impact this curriculum has…”  (I’ve been searching through the sites given in the article, but can’t find peer-reviewed, published papers that support these claims.)  But these are really minor quibbles.  Having written one of these pieces, I know that the tyrany of 1800 words is severe, and the authors can be excused for not providing background citations to the studies supporting their claims.  Instead, I’d like to provide some of the background literature that supports their claim.

Can we use programming to help students learn mathematics?  Absolutely!  Probably the most famous study supporting this argument is Idit Harel’s dissertation work on Instructional Software Development Project (ISDP). Idit had fourth graders write software in Logo to teach fractions to third graders.  She found that a real synergy occurred between the concepts of programming and the concepts of mathematics, and her students ended up learning more about both compared to another class.  Yasmin Kafai (who just moved to Penn from UCLA this last year) continued this project, exploring novel collaboration models (e.g., the fourth graders become fifth grade “consultants” as another cohort of fourth graders helps another cohort of third graders) and expanding from mathematics into science.  My own dissertation explored the synergy between physics and programming.  My results weren’t as strong — I had good physics learning, but not good computer science learning.  I suspect the problems were the challenge of making real learning in only a three week summer workshop, and not having the kinds of IDE’s that Matthias and Shriram are calling for.

“Our community must realize that minor tweaks of currently dominant approaches to programming won’t suffice.”  Completely agreed, and the best argument for this point came from Bruce Sherin’s powerful dissertation (with Andy diSessa at Berkeley).  Bruce taught two groups of students lessons in physics, one using programming and one using algebra.  (Andy would probably argue with Matthias and Shriram, “The ideal language and the IDE for imaginative programming are still to be designed.”  Over 20 years ago, Boxer implemented much of what they’re calling for.)  Bruce found some really interesting differences between what was learned via each form of notation.  For example, programming was better for figuring out causality and sequencing. An algebraic formula like x = x0 + vt leaves invisible to the novice that the t is what will typically vary in that equation.  On the other hand, algebra was better for understanding balance and equilibria.  A formula like F=ma works in both directions:  increase the mass or acceleration and the force increases, or if the force declines, then either the mass or the acceleration must have declined.  Most programming languages do not make evident how constraints work in the world.  The media extensions that Matthias and Shriram describe help address some of the challenges Bruce found when students had a single physics concept (e.g., an object moving because it’s location changed) being represented by multiple lines of code.

“As computer science educators, we must also demand a smooth, continuous path from imaginative programming to the engineering of large programs.”  Alan Kay has been making this argument for years.  He refers to the concept of Omniuser who can move from scripting in E-Toys, to changing how the levels close to the metal of the machine work, all with a single system and (hopefully) a single notation.  His STEPS effort is seeking to build such systems. In particular, Alan and his team are exploring “systems math” which is a kind of powerful mathematics that can only really exist in the powerful medium of programming.  Thus, STEPS gives us a way to go beyond just support “‘rithmetic” to support powerful new kinds of mathematics learning.

I’m a big fan of Scheme and consider DrScheme to be one of the finest pedagogical IDE’s ever created.  TeachScheme is a brilliant curriculum.  My guess is that careful studies of the effort would support many of the claims being made by Matthias and Shriram.  More importantly, though, I believe that they’re right that programming could actually improve mathematics learning.  Doing it in such a way that students’ mathaphobia doesn’t drive even more students from computer science is a real challenge.  An even bigger challenge is doing it in such a way that can gain the support of organizations like NCTM and that meets the mathematics standards in our schools. As they say, “Any attempt to align programming with mathematics will fail unless the programming language is as close to school mathematics as possible.”  It’s more than just the programming language — the whole package (curriculum, IDE, language) has to look and feel like mathematics to make it successful with the mathematics education community.

Nudging Computing Education

I’m spending Father’s Day reading. Just finished Terry Pratchett’s Equal Rites (the first appearance of Granny Weatherwax, which I had never read before), and have now just started Nudge: Improving Decisions about Health, Wealth, and Happiness by Thaler and Sunstein.  I’d heard of behavioral economics before, especially in the context of how these ideas are influencing the Obama administration.  I’m recognizing implications for computing education as well.

The basic premise of behavioral economics is that people are bad decision makers, and those decisions are easily biased by factors like the ordering of choices.  Consider the choice between a cupcake and a piece of fruit.  The worse choice there only has consequences much later and the direct feedback (“You gained weight because you chose the cupcake!”) is weak.  Thaler and Sunstein promote libertarian paternalism.  The idea is that we want to offer choices to people, but most people will make bad choices.  Libertarian paternalism suggests that we make the default or easiest choice the one which we (paternalistically) define as the best one — that’s a nudge.  It’s not always easy to decide which is the best choice, and we want to emphasize making choices that people would make for themselves (as best as we can) if they had more time and information.

An obvious implication for computing education is our choice of first programming language.  Alan Kay has pointed out many times that people are sometimes like Lorenz’s ducks, who were convinced that the Lorenz was their parent: people “imprint” on the first choice they see.  Thaler and Sunstein would probably agree that the first language someone learns will be their default choice when facing a new problem. We want to make sure that that’s a good default choice.

How do we choose the first, “best choice” language?  If our students are going to become software engineers, then choosing a language which is the default (most common, most popular) in software engineering would make sense: C++ or Java.  But what if our students are not going to become software engineers?  Then we’ve made their first language harder to learn (because it’s always harder as a novice to learn the tool used by experts), and the students don’t have the vocational aspirations to make the extra effort worthwhile.  That choice might then lead to higher failure/withdrawal rates and students regretting trying computer science.  Hmm, that seems familiar…

Another choice might be to show students a language in which the best thinking about computer science is easiest.  For example, Scheme is a great language for pointing out powerful ideas in computer science.  I believe that Structure and Interpretation of Computer Programming by Abelson and Sussman is the best computer science textbook ever written.  It’s power stems, in part, from its use of Scheme for exemplifying its ideas.

The challenge of using Scheme is that it is not naturally the language of choice for whatever problem comes the student’s way.  Sure, you can write anything in Scheme, but few people do, even people who know Scheme.  Libraries of reusable tools that make it easy to solve common problems tend to appear in the languages that more people are using. If students were well-informed (or are/become informed), would they choose Scheme? If the answer to that question is “No,” the teacher appears coercive and constraining, and the course is perceived as being irrelevant. That’s another familiar story.

The ideas of Nudge have implications for teachers, too. I am on the Commission to design the new Advanced Placement (AP) Computer Science exam in “Computer Science: Principles.”  (This exam is in contrast to the existing Level A CS AP exam in computer science programming in Java.) We just met for the first time this last week.  There will be programming in the new APCS exam, and there’s interest in providing teachers with choices of what language they teach.  Providing infinite choice makes it really hard to write a standardized, national exam.  Teachers will likely be offered a menu of choices.  How will those choices be ordered?  How will teachers make these choices?  While there are some wonderful high school teachers, there are too few high school CS teachers.  The new APCS exam will only be successful if most of the teachers offering it are brand new to computer science.  These teachers need help in making these choices, with reasonable default values, because they simply won’t have the experience yet to make well-informed choices.