The Inverse Lake Wobegon Effect in Learning Analytics and SIGCSE Polls

I wrote my Blog@CACM post this month about the Inverse Lake Wobegon effect (see the post here), a term that I coin in my new book (link to post about book).  The Inverse Lake Wobegon effect is where we observe a biased, privileged/elite/superior sample and act as if it is an unbiased, random sample from the overall population.  When we assume that undergraduates are like students in high school, we are falling prey to the Inverse Lake Wobegon effect.

Here’s an example from The Chronicle of Higher Education in the quote below. Looking at learning analytics from MOOCs can only tell us about student success and failure of those who sign up for the MOOC.  As we have already discussed in this blog (see post here), people who take MOOCs are a biased sample — well-educated and rich.  We can’t use MOOCs to learn about learning for those who aren’t there.

“It takes a lot of mystery out of why students succeed and why students fail,” said Robert W. Wagner, executive vice provost and dean at Utah State, and the fan of the spider graphic. “It gives you more information, and when you can put that information into the hands of faculty who are really concerned about students and completion rates and retention, the more you’re able to create better learning and teaching environments.”

Source: This Chart Shows the Promise and Limits of ‘Learning Analytics’ – The Chronicle of Higher Education

A second example: There’s a common thread of research in SIGCSE Symposium and ITICSE that uses survey data from the SIGCSE Members List as a source of information.  SIGCSE Members are elite undergraduate computer science teachers.  They are teachers who have the resources to participate in SIGCSE and the interest in doing so.  I know that at my own institution, only a small percentage (<10%) of our lecturers and instructors participate in SIGCSE.  I know that no one at the local community college’s CS department belongs to SIGCSE.  My guess is that SIGCSE Members represents less than 30% of undergraduate computer science teachers in the United States, and a much smaller percentage of computer science teachers worldwide. I don’t know if we can assume that SIGCSE Members are necessarily more expert or higher-quality.  We do know that they value being part of a professional organization for teaching, so we can assume that SIGCSE Members have an identity as a CS teacher — but that may mean that most CS teachers don’t have an identity as a CS teacher. A survey of SIGCSE Members tell us about an elite sample of undergraduate CS teachers, but not necessarily about CS teachers overall.