Fighting Engineers: Robots, Anecdotes, and Data

February 26, 2010 at 11:43 am 9 comments

Got into a fight yesterday with an engineer.  We were both at an advisory board meeting for a new project here at Georgia Tech about teaching middle school science with Lego robotics.  It’s a great project — very careful in assessment, using Janet Kolodner’s Learning By Design classroom structures and rituals, with a national advisory board including several engineering faculty who work with Lego robots in outreach programs.  I raised the issue of girls and Lego at several points during the day, and at the end of the day, I asked, “Is it too late to consider something other than Lego?  Or are we bought in now?  Could we use another form of robotics?”

At that point, one of the engineers said, “Wait, I have to call him on this.” Then turned to me, “You seem to be under the misperception that girls don’t like Lego robots.  You’re wrong!”

Something in “misperception” and “wrong” got my dander up.  I snapped back, “And I’ve got an n of 1500 that says that you’re wrong!”  At this point, people between us literally stepped back to avoid the cross-fire.

Janet inserted a comment here, “Well, wait a minute, it all depends on what you do and your approach…” I interrupted her and explained.  “We’ve been doing Lego Robots with girls for four years now with Georgia Computes! We find that Lego Robots don’t change girls attitudes about computing, doesn’t make them more interested in STEM fields, and doesn’t make them see themselves as doing computing.  We find that PICO Crickets and Scratch do. ” It’s all about increasing the odds of success.

After the exchange, several engineers came up to me to tell me how their outreach program really works, and how their FIRST Robotics team has 30% females.  I asked how they knew it was working.  “You can just tell!” or “The girls keep coming back!”  How can you tell?  Why do they keep coming back?  Maybe it has nothing to do with the robots?

As I mentioned earlier, I’m the School of Interactive Computing.  We do a lot of qualitative analysis around here.  A commonly heard phrase in our hallways is, “The plural of anecdote is not data.”  We’re pretty careful with our data collection in “Georgia Computes!” We do surveys before and after each activity.  We use an external evaluator, to try to avoid the kids telling us what they think we want to hear.  We regularly use focus groups to spot check our findings.

Janet’s right, of course.  Some of the engineers are probably right, and their programs are really working.  From what I can tell, none of them that I spoke with know that for sure because they are not trying to evaluate things carefully.  They have a bunch of anecdotes.

I do believe that one can create an excellent Lego Robotics outreach program that engages girls.  Our data suggest that it doesn’t happen automatically with the default curriculum.  You have to do something special, and it would be great if someone would do the studies to figure out what that something special is so that we could all be doing it.  Let’s change the default curriculum.

The question I’m raising here about Lego Robotics for girls is the same one that I raised earlier about CS1.  How would you know if it’s a bad idea? How do you know that it’s just not working, given any random set of students, teacher, classroom, textbook, and curriculum? The data on CS1 support the claim that any particular offering of CS1, with any popular and well-used textbook and curriculum, will probably result in a 30-50% failure rate.  You have to do something unusual to beat those numbers. Further, I think that the data support the claim: “Given the number of studies, and the large number of subjects, the probability of engaging girls with Lego Robotics is low.”  Is it possible? Sure!  It would be great to do it right at scale!  But any one person’s experience with their outreach program or FIRST Robotics team is not going to change that statement.

Entry filed under: Uncategorized. Tags: , , , .

A Computing Education Debate in CACM The economic benefits of reducing drop-outs

9 Comments Add your own

  • 1. ibogost  |  February 26, 2010 at 1:04 pm

    My experience with Lego Robotics is purely anecdotal, but it is real: working with one of my kid’s teams for a couple years. I think there are huge problems with the system and the competitions as a motivator of any kind, and it’s not surprising to hear your results on girls and Lego robotics. But I wonder, does your research show a similar distaste among all students, no matter their gender? That was certainly our experience.

    Incidentally, just because I can’t help myself, I’m super confused about using robotics as early CS education tools anyway, precisely because they tend not to work properly. For example, the Lego Mindstorms systems behave erratically based on battery levels and other seemingly random factors. If anything, my gut tells me these robots make kids think computers are frustrating, capricious creatures that they won’t ever master.

    • 2. Mark Guzdial  |  February 26, 2010 at 1:20 pm

      It’s a great question, Ian — I don’t know. Because we are about “broadening participation in computing,” we’re not making much effort to reach the general audience. Through the Girl Scouts, we get a chance to try lots of curricula with middle school girls (sometimes 100 a Saturday!). We have other efforts with members of under-represented minorities, but we haven’t tried Lego Robotics so much there. We tend to take the things that work best with the Girl Scouts to use in other settings — not that everyone is like middle school girls, but it’s a reasonable measure of “this works pretty well, we know how to do this so that students are successful.”

  • 3. Raymond Lister  |  February 27, 2010 at 4:06 pm

    Mark, you wrote …

    … several engineers came up to me to tell me how their outreach program really works … “You can just tell!”

    When these people are doing their own (non-education) research, I doubt very much that they are content to justify their work with “You can just tell”. Certainly, the people who signed off on their dissertations, and the people who referee their papers, are not going to be happy with the justification “You can just tell!”

    We academics tend to leave double lives. In our our research lives, we read the research literature, we compile evidence for our beliefs, we publish, and the cycle repeats. In contrast, most of us lead a private teaching life. Most of us are not guided by any teaching literature, nor do we systematically collect evidence, but instead we follow our gut feel.

    Why the double life?

    Not only do we lead a double life, but we *celebrate* it. Most academics are very deeply emotionally committed to their folk pedagogies, and their anecdotal evidence. Why?

    I’ll end with a couple of quotes that I like, on this same theme …

    Why do outstanding scientists who demand rigorous proof for
    scientific assertions in their research continue to use and,
    indeed defend on the bias of intuition alone, teaching methods
    that are not the most effective?

    Kirschner, Sweller, and Clark (2006)
    Why minimal guidance during instruction does not work.
    Educational Psychologist 41 (2) 75-86


    All over the country these groups of scholars, who would not make a decision about the shape of a leaf or the derivation of a word or the author of a manuscript without painstakingly assembling the evidence, make decisions about admission policy, size of universities, staff-student ratios, content of courses and similar issues, based on dubious assumptions, scrappy data and mere hunch.

    Ashby, E (1963), “Decision making in the academic world”,
    in Halmos, P (ed), Sociological Studies in British University Education, University of Keele.

  • 4. uberVU - social comments  |  February 27, 2010 at 8:48 pm

    Social comments and analytics for this post…

    This post was mentioned on Twitter by alfredtwo: Found this on @guzdial blog ‘The plural of anecdote is not data. “(Frank Kotsonis)

  • 5. Maureen Tumenas  |  February 28, 2010 at 9:51 am

    Have you looked at Alice as an entry for girls? Or Storytelling Alice? I know that part of their research has been to bring more students to CS, and Storytelling Alice has been successful with girls. Are they also documenting their results by gender? Wanda Dunn @ CMU may be able to answer these questions. Anecdotally, I find the girls to be as engaged as the boys, because they can use Alice to build in the ways that they design.

    • 6. Mark Guzdial  |  February 28, 2010 at 10:09 am

      Hi Maureen. Check out the papers from “Georgia Computes!” to see how we’re using Alice, Storytelling Alice, and other tools (like Scratch), and how these tools are impacting student perceptions of computing in our workshops.

  • 7. Frank McCown  |  March 6, 2010 at 9:27 am

    I’m fascinated by our field’s continual push to increase female participation (a goal I personally share). From the amount of attention this subject gets in ACM’s News, CACM and other outlets, this is probably issue #1 on our list of hot CS education topics.

    Mark, do you know if other fields, say nursing or elementary education, are putting as much effort as we are into recruiting men?

    Secondly, how will we know when we have fully succeeded in recruiting women? Is a 50/50 makeup our goal? Since women now out-number men in university graduation rates, would success be 55/45 instead?

    I’m sure someone has addressed both these questions elsewhere, but I have yet to stumble across it.

    • 8. Mark Guzdial  |  March 6, 2010 at 10:05 am

      NCWIT ( has a lot of statistics and explicit goals about this. Since we were at 40/60 in the 70’s, that’s a reasonable starting target.

  • […] of our audience, and not others.  When we use video games or robots in examples, for example, we tend to get the boys more engaged than the girls.  I’ve found that it’s hard to be culturally neutral in my own assignments.  One […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,185 other subscribers


Recent Posts

Blog Stats

  • 2,060,393 hits
February 2010

CS Teaching Tips

%d bloggers like this: