Fighting Engineers: Robots, Anecdotes, and Data

Got into a fight yesterday with an engineer.  We were both at an advisory board meeting for a new project here at Georgia Tech about teaching middle school science with Lego robotics.  It’s a great project — very careful in assessment, using Janet Kolodner’s Learning By Design classroom structures and rituals, with a national advisory board including several engineering faculty who work with Lego robots in outreach programs.  I raised the issue of girls and Lego at several points during the day, and at the end of the day, I asked, “Is it too late to consider something other than Lego?  Or are we bought in now?  Could we use another form of robotics?”

At that point, one of the engineers said, “Wait, I have to call him on this.” Then turned to me, “You seem to be under the misperception that girls don’t like Lego robots.  You’re wrong!”

Something in “misperception” and “wrong” got my dander up.  I snapped back, “And I’ve got an n of 1500 that says that you’re wrong!”  At this point, people between us literally stepped back to avoid the cross-fire.

Janet inserted a comment here, “Well, wait a minute, it all depends on what you do and your approach…” I interrupted her and explained.  “We’ve been doing Lego Robots with girls for four years now with Georgia Computes! We find that Lego Robots don’t change girls attitudes about computing, doesn’t make them more interested in STEM fields, and doesn’t make them see themselves as doing computing.  We find that PICO Crickets and Scratch do. ” It’s all about increasing the odds of success.

After the exchange, several engineers came up to me to tell me how their outreach program really works, and how their FIRST Robotics team has 30% females.  I asked how they knew it was working.  “You can just tell!” or “The girls keep coming back!”  How can you tell?  Why do they keep coming back?  Maybe it has nothing to do with the robots?

As I mentioned earlier, I’m the School of Interactive Computing.  We do a lot of qualitative analysis around here.  A commonly heard phrase in our hallways is, “The plural of anecdote is not data.”  We’re pretty careful with our data collection in “Georgia Computes!” We do surveys before and after each activity.  We use an external evaluator, to try to avoid the kids telling us what they think we want to hear.  We regularly use focus groups to spot check our findings.

Janet’s right, of course.  Some of the engineers are probably right, and their programs are really working.  From what I can tell, none of them that I spoke with know that for sure because they are not trying to evaluate things carefully.  They have a bunch of anecdotes.

I do believe that one can create an excellent Lego Robotics outreach program that engages girls.  Our data suggest that it doesn’t happen automatically with the default curriculum.  You have to do something special, and it would be great if someone would do the studies to figure out what that something special is so that we could all be doing it.  Let’s change the default curriculum.

The question I’m raising here about Lego Robotics for girls is the same one that I raised earlier about CS1.  How would you know if it’s a bad idea? How do you know that it’s just not working, given any random set of students, teacher, classroom, textbook, and curriculum? The data on CS1 support the claim that any particular offering of CS1, with any popular and well-used textbook and curriculum, will probably result in a 30-50% failure rate.  You have to do something unusual to beat those numbers. Further, I think that the data support the claim: “Given the number of studies, and the large number of subjects, the probability of engaging girls with Lego Robotics is low.”  Is it possible? Sure!  It would be great to do it right at scale!  But any one person’s experience with their outreach program or FIRST Robotics team is not going to change that statement.