Are we measuring what students are learning?

July 14, 2009 at 7:53 am 11 comments

One measure of the success of a talk is how many questions you get in the hallway after the talk.  I got a few yesterday, which suggests that people were still thinking about the points afterwards.

One question I got was about a finding we’ve had in several of the contextualized computing education classes, like robotics and Gameboys for computer organization.  Students report spending extra time on their homework beyond what’s required “just because it’s cool.”  Yet, in some cases, there is no difference in grade distributions or failure rates compared to a comparison class.  What gives?  Isn’t that a bad thing if students spend extra time but it’s not productive time?

Absolutely, that can be the case.  It may also be the case that students are learning things that we don’t know how to measure.  Think about the argument that it takes 10,000 hours of practice to develop expertise (a number that has been recalculated from several sources).  Can we come up with learning objectives for each of those 10,000 hours?  Or is it that we can measure some of those objectives, but others of the items being learned are subtle, or are prerequisite concepts, or are about skills, or even muscle memory?

A famous story in physics education is about how concepts are more complex and have more  facets than we realize.  David Hestenes has developed some sophisticated and multi-faceted assessments for concepts like “force” — a whole test, just addressing “force.”  Eric Mazur at Harvard scoffed at these assessments (as he said at a AAAS meeting I went to a couple of years ago, and quoted in a paper by Dreifus in 2007).  His Harvard students would blow these assessments away!  Gutsy man that he is, he actually tried them in his classes.  His students did no better than the averages that Hestenes was publishing.  Mazur was aghast and became a outspoken proponent of better forms of teaching and assessment.

Building up these kinds of assessments takes huge effort but is critically important to measure what learning is really going on.  For the most part in Computing Education, we have not done this yet.  Grades are a gross measure of learning, and to progress the field, we need fine-grained measures.

Entry filed under: Uncategorized. Tags: , , , .

Talk at Microsoft Research on IPRE Feynman lectures from Microsoft: A medium for active essays and computing ed

11 Comments Add your own

  • 1. Leigh Ann  |  July 14, 2009 at 11:19 am

    I’m really interested in this too. I guess it stems a little from trying to just do good CS Ed research – you really need to validate what it is you are measuring.

    I took a very interesting workshop this summer on Item Response Theory which can statistically help validate assessments and also tell you if there are underlying, unaccounted for skills that your assessment is measuring. I have some really interesting results from an old assessment that I gave while I was teaching – will be posting about it soon.

    In the contextualized classes I think its just as important to measure the motivational and emotional aspects of the learning as the actual learning goals. Instructional design includes dispositions as one of the three areas of goals (knowledge, skills, dispositions) and that includes how people feel about the content. Self efficacy and also affect are definitely to be taken into account in any situated learning experience.

  • […] Mark Guzdial: A famous story in physics education is about how concepts are more complex and have more facets than we realize. David Hestenes has developed some sophisticated and multi-faceted assessments for concepts like “force” — a whole test, just addressing “force.” Eric Mazur at Harvard scoffed at these assessments (as he said at a AAAS meeting I went to a couple of years ago, and quoted in a paper by Dreifus in 2007). His Harvard students would blow these assessments away! Gutsy man that he is, he actually tried them in his classes. His students did no better than the averages that Hestenes was publishing. Mazur was aghast and became a outspoken proponent of better forms of teaching and assessment. […]

  • 3. Alan Kay  |  July 16, 2009 at 6:55 am

    Hi Mark

    It’s worth thinking about the latencies in various kinds of learning, including both the idea that it might take longer for important ideas and skills to be acquired than many of the durations between “teaching” and “testing”, and also the even more important die away effects over months and years of time. Even well meaning testing tends to be bogus with respect to what’s really going on, and the oft heard sentiments about “we need to measure the effectiveness of our pedagogy” sound great, but have the actual worth and analogy of searching under the street lamp for the keys inconveniently lost down the street in the dark.

    For example, we’ve found it takes about three years of work to set up and test a curriculum. And that real assessment of the students in computing, math, science, etc., needs to be done very differently than the educational cliches would have it.

    This brings to my mind once again the possible (and likely) analogies to music learning and assessment (I think much of sports learning is very similar also). Some of the main aspects are that:
    (a) the teachers in these areas are usually pretty strong practitioners
    (b) they can work and play with the learners, and do so
    (c) they can tell right away where the learners are operationally
    (d) they know that operational skills are not the only story, temporal effects really make a difference also — students can often show different kinds of knowledge when things are slowed down so the “front of the mind” can help with what “the back of the mind” can’t execute on first
    (d) they know that many kinds of improvements cannot be accomplished in near time to the instruction, but must be practiced for days, months, years
    (e) this makes how they assess in the near term very different (are the students actually doing the practicing? that is, the assessment is on the *process* of the students more than near term results)
    (f) they know that it’s “use it or loose it”, so what really counts is “using it” for long periods and usually outside of instruction
    (g) and so forth

    I’m reflecting on this as I’m attending an intense learning experience at “Jazz Vermont” which is mid to large jazz group playing. I’d assess our coach as being in the top class of educators I’ve ever met, and he’s not even trying to “educate” so much as to “facilitate and open up”.

    The experience is mostly playing very tough arrangements at sight with work ons in one group for 6+ hours a day. I’m in a Tentet and the coach plays one of the two trumpets. He’s really a good player and like most good musicians he can play and listen at the same time, so he’s hearing everything we are doing plus doing well at his own part.

    Part of the key here is that’s what you need to learn to do in jazz ensemble playing, so he’s really helping us learn to be “coaches” rather than just being able to play (which in isolation winds up being *not learning to play*).

    I think this idea is powerful and works for areas other than music (for example, computing).



  • 4. Alan Kay  |  July 16, 2009 at 12:26 pm

    P.S. One thing I forgot to mention is that what counts as “good in school” vs “good in reality” is often so very different as to be unrecognizable. To go back to sports and music learning again, 50-60% field goals (and 40-50% three pointers) are held to be great, anything above a .330 batting average (1 in 3) is very very good (Ichiro is at .332 right now), hitting zillions of clams while practicing music, and making errors while improvising, etc., are all part of standard practice in the real world (there’s lots of risk associated with the “good stuff”).

    Similary, the ARPA/PARC style of high risk research which brought forth much of the breakthrough technologies and ideas and architectures that are used today is no longer funded. Instead most places that used to fund high risk stuff now want in their proposals what is essentially an engineering recipe for “why this project is going to succeed” (and Congress which oversees NSF and DARPA wants “accountability” and does not like “failures”).

    Similarly, in schools, what tends to be taught is stuff that with some work the students can become almost perfect in the short term. There’s something to be said for this, but only if it is balanced off with a hefty taste of the real deal, which is how to handle important tough problems that might take years to solve and won’t be if the system wimps out to safety. An old saying: “Ships are safe in harbor, but that’s not what ships are for”.

    Back to music for a minute … most music camps are set up to *not* wind up with perfect perfomances at the end of a week or two weeks, but instead are set up to make a safe place to be stretched beyond (a) what one can do, and (b) beyond what one can learn to do in the time of the camp. This is true of good sports camps also. The idea is to get a real feel for what the next level is all about in a setting that is very tolerant of glitches. The results — in the form of a final concert, etc. — will have less glitches than at the beginning, but it will still have them, but the learners will really understand just what it is that they need to work on for the next extended period of time.

    In other words, the “course” is not one that has a beginning middle and end, but one that has a beginning and an extended beginning. The middle is carried by the learner through the next phases, and there is no end.



  • 5. Mark Guzdial  |  July 18, 2009 at 1:27 pm

    Hi Alan,

    I find the comparison with music (and sports) education fascinating. The notion of a course as “a beginning and an extended beginning” is powerful, but needs some buy-in from outside education circles. A lot of the 2-year/community colleges and even 4-year colleges and universities that I work with are driven by the demands of industry in the local community. On one hand, that’s a positive thing — it meshes with the local control of our education system, which Dewey praised. Dewey explicitly encouraged connections between the classroom and the needs of the community. On the other hand, it limits the classroom when the industry starts dictating to the faculty as if they were trainers in a jobs preparation program. It’s a hard balance to strike between responsiveness to the local community and maintaining focus on a lifetime of learning and of facing challenging problems.

    Hope your week of jazz in Vermont was great “hard fun”!

  • 6. Alan Kay  |  July 18, 2009 at 3:37 pm

    Hi Mark,

    Well, “do unto others as you would have them do unto you” also needs buy-in. And so do lots of desirable things (like world peace).

    I think first and foremost, it is critical to identify what’s important, even if it is a difficult idea with little buy-in. Otherwise, we wind up with everybody agreeing on something that seems more reasonable and easier, but which actually rots the system.

    The idea that someone is “driven” by the “demands of X” is another oft used figure of speech, where the reality is no such thing.

    They miss the systems aspects of the deadly embrace and downward spiral that results when there is too close coupling between (say) education and industry. It should easily seen to be disastrous to have this tight of a coupling since the eventual employees of industry will come from university. Industry has far narrower goals than those of educational institutions, and thus it is very bad to have these constricted points of view start to constrict educational foundations.

    Instead, it should be crystal clear that it should be the institution with the wider, deeper goals and points of view that should be helping those with narrower focus to avoid slipping into such limited horizons that they can’t even see well enough to make money (witness the pitiful record of US business recently — they can’t even carry out their own limited notion of what they are supposed to do — this is because they have no idea of what world they live in, what the dynamics of systems are like, how to get synergies instead of strip-mining everything in sight.

    Dewey was right, and he was clear about “the needs of the community” not “the wants of the community”. The differences between these are profound, and they are the differences between education and marketing.

    And this difference is one of the things that real education is supposed to help people understand.

    Too bad, that in giving in to the “demands” (and misunderstanding them as a forced choice instead of an actual choice they could make) what used to be education institutions have now let themselves (yes it is their fault) become mere training arms of those of such limited horizons and goals.



    • 7. Mark Miller  |  July 23, 2009 at 8:20 pm

      Hi Alan.

      You make a lot of sense here. I was just recently having a discussion with some IT people about the role of universities in relation to business IT. Students who go into CS or something similar (nowadays it’s more likely CIS/MIS) come in with the expectation that they’re training for a career. That’s the expectation that they and their parents set up. Schools have taken advantage of it. This is the expectation I came in with when I got into it. Ideally what happens is by the time they graduate they’ve learned enough about the field of study itself to realize that they’re actually interested in it for its own sake, because it lends something of value to the way they think about computing, and perceive its role in any endeavor. That can come to the fore and they can realize that it’s not just about a job at the end of it. That’s what happened with me, though I think part of my own mindset contributed to that. I already loved computing by the time I got into CS. So while I had career-minded goals for taking it, I was open to the idea that there was more to it.

      The goals of business have been quite different from the goals of schools. Theirs are centered generally around productivity, and/or efficiency, forming relationships, and adding value (not all businesses succeed at all four or even have them as goals, though forming relationships is essential). There’s also usually a heavy emphasis on obedience to process, because of bureaucracy, particularly in IT.

      There’s long been a sense of rivalry between schools and business, due to their different goals, because each thinks that theirs are paramount, while the other’s are less relevant. I’ve long thought that the positive aspects of education have something to contribute to business processes, but I agree that it’s a constant struggle with business to try to get them to see that. People with more of a business mindset have viewed schools with the same sense of frustration, of course for different reasons.

      In the last 10 years, probably longer, business has been “winning” this struggle between themselves and schools, to the detriment of both. I’ve long wished that this rivalry would end, not in a “victory” for one side or the other, but for both sides to realize that the struggle is unnecessary, that each party’s goals are valuable in their respective contexts, and they could each learn something from each other–probably more from school to business, than from business to school, but that’s my bias. The only way this will happen, I think, is for business to realize that what they think are their complete set of goals are actually quite limited, and that more goals need to be added to the mix in order to accomplish their overall goal of being profitable. I remember that economics professor Lester Thurow used to talk about this publicly.

      I think the opening that’s available in business is in the “adding value” goal. There are businesses who have educating the customer as an important goal, which differentiates them from their competitors. This addition of education to a service offering seems to be something unique to what’s been called the “Information Age”. In order to do this successfully they must internalize a sense that educating themselves is important, which I think is the closest thing that one could ask for as a positive synergy between schools and business.

  • 8. Alan Kay  |  July 18, 2009 at 7:13 pm


    Tim Gallwey of the “Inner Game of Tennis” is an incredibly gifted teacher, and all who strive to teach can learn much from reading (especially) his first book. I often attended his tennis clinics to try to understand more about how he achieved his amazing results.

    One question he asked the students was “Would you like to know how to absolutely win every tennis match you play?” Everyone shouted “Yes, yes!”. He said “Just never play any one who is remotely as good as you are — for example, why not just play 4 year olds?”

    This got this group of Americans to think more about their motivations than they perhaps ever had before. Most realized that they really wanted to play opponents who were a little better than they were, because that would be a nice challenge and they would learn more.

    But then they slowly (and grudgingly in many cases) started to realize that they just weren’t going to win all that often under these conditions. So should they be getting so down on themselves when they didn’t win?

    This went against the American sentiment (voiced by Vince Lombardi as) “Winning is not the best thing, it’s the only thing”. This is crazy, but it is hard for our culture to see it, and harder still to identify what is better that should be sought.

    My favorite philosopher — Linda Ronstadt — was once asked about a recording session (“Songs of my father”) she was about to do with the very best Mexican musicians who had been playing this music for many decades all their life. She said, “Well, of course I will not be up to their abilities, but I will be happy and satisfied if i can *match the quality of their effort*”

    She has a nice way of expressing really important ideas, doesn’t she? “All you can do, ” she once said, “is to follow your instincts and refine your talents to support your instincts”.

    When winning is the main goal — rather than learning, growing, building, helping, etc. — and the ethical environment is weak, then too many corners get cut, and many of these cuttings are disastrous to the quality of human life — they literally cut out and off necessary relationships in the many systems we live inside.

    Using sports as a metaphor is tricky in the US because most Americans think it is about winning, rather than “sport” and “sportsmanship”. So music is a little easier to draw analogies from (though less so each year given how much of music is now portrayed as contests on US TV). On the other hand, if the difficult nature of “sportsmanship” can be grappled with then sports provides many useful parallels that can help us think about much more critical pursuits and issues.



    • 9. Mark Miller  |  July 24, 2009 at 12:46 am

      Referring to your Linda Ronstadt anecdote, I like the example. I’ve heard a few software developers say that it’s best to be “the worst player in the band”, because you learn a lot more that way. That sounds right, but every time I’ve heard that I’ve wondered, “Okay, but will the band accept me if I’m the worst player?” Depending on my abilities I might be a great contributor, or I might be more of a distraction and not be able to contribute anything, despite my best efforts. Again, it’s an issue of matching people’s abilities with challenges they can meet.

      The situation with business has been that the incentives all point towards a motivation to have a lot of wins. I agree it doesn’t have to be this way, but I think a significant part of it is investor behavior. This may have a lot to do with our low savings rate. If people think their retirement depends on making healthy short-term gains in stock values, then that drives this sort of thing. It drives the “quarter-to-quarter” analysis that I’ve heard many complain about for years, because long-term thinking doesn’t come into it at all. Certainly the culture this encourages, combined with ambitious people, can drive this to extremes that are harmful.

      I was talking about this with some people earlier, that Americans have a long history of not understanding credit. I remember a quote I heard years ago where Thomas Jefferson complained about this. We treat credit as a form of wealth, when it’s not, and this I think goes a long way towards explaining our low savings rate. I guess one could say that to our culture credit cards are like television. Both can be used wisely to accomplish positive goals, but they rarely are. Our spending habits with them are driven by our consumerist culture which has been cultivated since the 1920s. This is driven by known psychological factors which are expertly isolated and amplified by some marketers. I don’t mean to bash business, but I think the practice of PR is dangerous to our society, though with conditions being the way they are it is a methodology that gets people on a societal scale to listen to messages, both constructive and destructive. Until people are able to reason at a high level as well as feel this is the reality.

  • 10. Mark Guzdial  |  July 20, 2009 at 7:51 pm

    We’re reviewing the reviews from our latest rejected NSF proposal and we’re definitely feeling your comment: ‘Instead most places that used to fund high risk stuff now want in their proposals what is essentially an engineering recipe for “why this project is going to succeed” ‘ If it was *obvious* that it would work, it wouldn’t be *research* would it?!?

  • 11. Alan Kay  |  July 21, 2009 at 8:53 am

    Hi Mark

    I think today it basically comes down to that they would rather feel 100% in control of mediocre processes than feel even a little out of control on possibly great processes.

    They essentially confuse “responsibility” with “control”. Because they are responsible they think they have to be in control even if this is not a good idea and they are not competent to control. The classic ARPA and ONR 60s funders did not make that error. They knew they were responsible, but they also knew that the folks they funded were going to have to make the big choices.

    And I think there is some “luck of the draw” also. Not every program manager at NSF and the other funding agencies is timid. They do have the power to override the peer review process (which I think tends to be very orthodox and timid in general). Sometimes the PMs do override, but then they are in the line of fire for congressional wrath.

    Also, as you know, CISE has added a new funding area called “Expeditions”, which are supposed to be more like the classic old ARPA and ONR funding in the 60s.

    There’s no question that things have been very different and tough in the “good funding” areas since the 70s when ARPA changed to DARPA and forced (among other things) the foundation of Xerox PARC in order for the next phases of “Lick’s Dream” to get invented. Despite the success of PARC, it was not a great idea to have to do this work inside a company.

    The commercialization of the personal computer and its coinvented networks (Ether and Inter) created a huge verdant valley ripe for strip mining, and most commercial funding has been to exploit the 70s inventions rather than to foster the next wave of real invention. It is missed by almost all that there have been no big inventions in the last 25 years – no qualitatively better ways to program, do networking, architect CPUs, etc.

    The democratization of the 70s technology has led to a pop culture, which like most pop music, etc., does not make use of the best ideas and inventions but finds “air guitar” and now “Guitar Hero” versions of what used to be “real guitar”. This is sometimes called the “reinvention of the flat tire”, because one gets *less* of a vehicle (to put it mildly) than one had before.

    We started this set of comments with how the universities are wrongly “driven” by the commercial interests. The admixture of the pop culture elements in computing and commercialism plus the all too willing to be “driven” universities results in universities actually teaching quite a bit of the pop culture instead of the real deal.

    This is a field day era for Anthropologists and Sociologists and Psychologists, but for computer scientists has all the fascination and horror of a slow motion train wreck.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,052 other followers


Recent Posts

Blog Stats

  • 2,030,874 hits
July 2009

CS Teaching Tips

%d bloggers like this: