Posts tagged ‘MOOCs’

How the Pioneers of the MOOC Got It Wrong (from IEEE), As Predicted

There is a sense of vindication that the predictions that many of us made about MOOCs have been proven right, e.g., see this blog post where I explicitly argue (as the article below states) that MOOCs misunderstand the importance of active learning. It’s disappointing that so much effort went wasted.  MOOCs do have value, but it’s much more modest than the sales pitch.

What accounts for MOOCs’ modest performance? While the technological solution they devised was novel, most MOOC innovators were unfamiliar with key trends in education. That is, they knew a lot about computers and networks, but they hadn’t really thought through how people learn.

It’s unsurprising then that the first MOOCs merely replicated the standard lecture, an uninspiring teaching style but one with which the computer scientists were most familiar. As the education technology consultant Phil Hill recently observed in the Chronicle of Higher Education, “The big MOOCs mostly employed smooth-functioning but basic video recording of lectures, multiple-choice quizzes, and unruly discussion forums. They were big, but they did not break new ground in pedagogy.”

Indeed, most MOOC founders were unaware that a pedagogical revolution was already under way at the nation’s universities: The traditional lecture was being rejected by many scholars, practitioners, and, most tellingly, tech-savvy students. MOOC advocates also failed to appreciate the existing body of knowledge about learning online, built over the last couple of decades by adventurous faculty who were attracted to online teaching for its innovative potential, such as peer-to-peer learning, virtual teamwork, and interactive exercises. These modes of instruction, known collectively as “active” learning, encourage student engagement, in stark contrast to passive listening in lectures. Indeed, even as the first MOOCs were being unveiled, traditional lectures were on their way out.

Source: How the Pioneers of the MOOC Got It Wrong – IEEE Spectrum

February 17, 2017 at 7:17 am 2 comments

OMS CS graduates a different kind of student: Work from Harvard Graduate School of Education

This is the work that most impresses me about OMSCS — that it attracts a different group of students that might get a face-to-face MS in CS. I’m not sure that I buy “equivalent in all ways to an in-person degree,” but I do see that it’s hard to measure and the paper makes a good effort at it.

Previous research has shown that most users of online education look fairly similar to the average college graduate — suggesting that digital learning isn’t yet the great educational equalizer it has the potential to be. But in a study of Georgia Tech’s hugely successful online master of science in computer science (OMSCS) program, educational economists Joshua Goodman and Amanda Pallais and public policy expert Julia Melkers found that digital learning can tap into a new market of students by offering an online degree that is equivalent in all ways to an in-person degree, at a fraction of the cost.

Source: The Digital Bridge | Harvard Graduate School of Education

November 16, 2016 at 7:14 am 4 comments

Helping Adults to Learn while Saving Face: Ukulele and MOOCs at Dagstuhl

I played ukulele every night while at the Dagstuhl seminar on CS learning assessment. Most nights, there was a group of us — some on guitars from the music room, one on piano, and several singers. It was wonderful fun! I don’t often get a chance to play in a group of other instruments and other singers, and I learned a lot about styles of play and synchronizing. The guitar players were all much more experienced, but we were all playing and singing music seen for the first time. We weren’t performance-quality — there were lots of off-key notes, missed entrances/exits. We were a bunch of amateurs having fun. (Thanks to Ben Shapiro, Jan Erik Moström, Lisa Kaczmarczyk, and Shriram Krishnamurthi for sharing these photos.)


We were not always a popular group. Some participants groaned when the guitars and ukulele came in to the room. One commenter asked if the singing was meant to drown out the playing.  Another complained that our choice of songs was “wrong” for the instruments and voices. Clearly, some of the complaints were for humorous effect, and some were pretty funny.

Here’s the thought experiment: Imagine these were kids playing music and singing. I predict the result would be different. I doubt the listeners would criticize the players and singers in the same way, not even for humorous effect. While adults certainly criticize children when in a teacher-student or mentoring relationship, casual criticism by passerby adults of a child playing or practicing is unusual.

Why is it different for adults?

I’ve talked before about the challenges of adult learning. We expect adults to have expertise. We expect quality. It’s hard for adults to be novices. It’s hard for adults to learn and to save face.  My colleague Betsy DiSalvo points out that we typically critique people at a near-peer level of power — we don’t casually critique those with much less power than us (children) because that’s mean, and we don’t casually critique our bosses and managers (to their faces) because that’s foolish.  Getting critiqued is a sign that you’re recognized as a peer.

After her work at Xerox PARC, Adele Goldberg helped develop learning systems, including systems for the Open University in the UK. She once told me that online systems were particularly important for adult learners. She said, “It’s hard for people with 20 years of expertise in a field to raise their hands and say that they don’t know something.”

Andy Ko framed MOOCs for me in a new way at the Dagstuhl Seminar on Assessment in CS. In the discussion of social and professional practice (see previous blog post), I told him about some ideas I had about helping people to retrain for the second half of life. We live much longer than people did 30-50 years ago. Many college-educated workers can expect a work life into our 70’s. I’ve been wondering what it might be like to support adult students who might retrain in their 40’s or 50’s for another 20 year career later. Andy pointed out MOOCs are perfect for this.

College-educated professionals currently in their careers do have prior education, which is a population with which MOOCs are most successful. MOOCs can allow well-educated students to retrain themselves as time permits and without loss of face. A recent Harvard study shows that students who participate Georgia Tech’s MOOC-based OMS CS program are in a demographic unlikely to have participated in a face-to-face MS in CS program (see page here). The MOOCs are serving an untapped need — it’s not necessarily reaching those who wouldn’t have access to education otherwise, but it can be a significant help to people who want to re-train themselves.

There are lots of uses of MOOCs that still don’t make sense to me.  Based on the empirical evidence of MOOCs today (in their current forms), I argue that:

  • MOOCs are not going to democratize education.  They have not been effective at motivating novices to learn required content, as opposed to elective or chosen content.
  • MOOCs are unlikely to broaden participation in computing.  Betsy DiSalvo and I ran a study about why women aren’t participating in OMS CS.  Those reasons are unlikely to change soon.
  • MOOCs may not work for adults who are being required to, or are asked to retrain, as opposed to those who choose to retrain.  Motivation matters. I have not yet seen convincing evidence that MOOCs can play a significant role in developing new CS teachers.  It’s hard to convince teachers to learn to be CS teachers — they’re not necessarily motivated to do so. Without the intrinsic motivation of choosing to be there, they may not complete.  A teacher who doesn’t complete doesn’t know the whole curriculum.

Adults will still have to have tough skins when practicing their new skills. We expect a lot of expertise out of the starting gate for adults in our society, even when retraining for a second career. MOOCs might be excellent preparation for adults in their second acts.

March 11, 2016 at 8:02 am 1 comment

The Inverse Lake Wobegon Effect in Learning Analytics and SIGCSE Polls

I wrote my Blog@CACM post this month about the Inverse Lake Wobegon effect (see the post here), a term that I coin in my new book (link to post about book).  The Inverse Lake Wobegon effect is where we observe a biased, privileged/elite/superior sample and act as if it is an unbiased, random sample from the overall population.  When we assume that undergraduates are like students in high school, we are falling prey to the Inverse Lake Wobegon effect.

Here’s an example from The Chronicle of Higher Education in the quote below. Looking at learning analytics from MOOCs can only tell us about student success and failure of those who sign up for the MOOC.  As we have already discussed in this blog (see post here), people who take MOOCs are a biased sample — well-educated and rich.  We can’t use MOOCs to learn about learning for those who aren’t there.

“It takes a lot of mystery out of why students succeed and why students fail,” said Robert W. Wagner, executive vice provost and dean at Utah State, and the fan of the spider graphic. “It gives you more information, and when you can put that information into the hands of faculty who are really concerned about students and completion rates and retention, the more you’re able to create better learning and teaching environments.”

Source: This Chart Shows the Promise and Limits of ‘Learning Analytics’ – The Chronicle of Higher Education

A second example: There’s a common thread of research in SIGCSE Symposium and ITICSE that uses survey data from the SIGCSE Members List as a source of information.  SIGCSE Members are elite undergraduate computer science teachers.  They are teachers who have the resources to participate in SIGCSE and the interest in doing so.  I know that at my own institution, only a small percentage (<10%) of our lecturers and instructors participate in SIGCSE.  I know that no one at the local community college’s CS department belongs to SIGCSE.  My guess is that SIGCSE Members represents less than 30% of undergraduate computer science teachers in the United States, and a much smaller percentage of computer science teachers worldwide. I don’t know if we can assume that SIGCSE Members are necessarily more expert or higher-quality.  We do know that they value being part of a professional organization for teaching, so we can assume that SIGCSE Members have an identity as a CS teacher — but that may mean that most CS teachers don’t have an identity as a CS teacher. A survey of SIGCSE Members tell us about an elite sample of undergraduate CS teachers, but not necessarily about CS teachers overall.

January 18, 2016 at 8:03 am 3 comments

No Rich Child Left Behind, and Enriching the Rich: Why MOOCs are not improving education

When I talk to people about MOOCs these days, I keep finding myself turning to two themes.

Theme #1. Our schools aren’t getting worse.  The gap between the rich and the poor is growing.  We have more poorer kids, and they are doing worse because of everything, not just because of school.

Before we can figure out what’s happening here, let’s dispel a few myths. The income gap in academic achievement is not growing because the test scores of poor students are dropping or because our schools are in decline. In fact, average test scores on the National Assessment of Educational Progress, the so-called Nation’s Report Card, have been rising — substantially in math and very slowly in reading — since the 1970s. The average 9-year-old today has math skills equal to those her parents had at age 11, a two-year improvement in a single generation. The gains are not as large in reading and they are not as large for older students, but there is no evidence that average test scores have declined over the last three decades for any age or economic group.

The widening income disparity in academic achievement is not a result of widening racial gaps in achievement, either. The achievement gaps between blacks and whites, and Hispanic and non-Hispanic whites have been narrowing slowly over the last two decades, trends that actually keep the yawning gap between higher- and lower-income students from getting even wider. If we look at the test scores of white students only, we find the same growing gap between high- and low-income children as we see in the population as a whole.

It may seem counterintuitive, but schools don’t seem to produce much of the disparity in test scores between high- and low-income students. … It boils down to this: The academic gap is widening because rich students are increasingly entering kindergarten much better prepared to succeed in school than middle-class students. This difference in preparation persists through elementary and high school.

Source: No Rich Child Left Behind – The New York Times

Theme #2:  There are definitely tangible effects of MOOCs, as seen in the study linked below. They help rich white men find better jobs.  They help educate the rich.  They help a small percentage of the poor.

All the money being poured into developing MOOCs fuels the gap between the rich and the poor.  If you want to improve education generally, nationally or worldwide, aim at the other 90%.  MOOCs aren’t improving education. They enrich those who are already rich.

Using data from MOOCs offered by the University of Pennsylvania, Alcorn, Christensen and Emanuel were some of the first to suggest that MOOC learners were more likely to be employed men in developed countries who had previously earned a degree — countering the early narrative that MOOCs would democratize higher education around the world.

Source: Study finds tangible benefits for learners from Coursera’s massive open online courses | InsideHigherEd


Commenters pointed out that I didn’t make my argument clear.  I’m posting one of my comment responses here to make clearer what I was trying to say:


As Alan pointed out, the second article I cited only once says that MOOC learners are “more likely to be employed men in developed countries.” I probably should have supported that point better, since it’s key to my argument. All the evidence I know suggests that MOOC learners are typically well-educated, more affluent from the developed world, and male.

  • In the original EdX MOOC, 78% of the attendees had already taken the class before. (See full report here.)
  • Tucker Balch released demographics on his MOOC: 91% male, 73.3% from OECD countries, and over 50% had graduate degrees. (See post here.)
  • Still the most careful analysis of MOOC demographics that I know is the 2013 Penn study (see article here) which found, “The student population tends to be young, well educated, and employed, with a majority from developed countries. There are significantly more males than females taking MOOCs, especially in developing countries.”
  • As you know, Georgia Tech’s Online MS (OMS) in CS is 85% domestic (the opposite of our face-to-face MS, which actually serves more students from the developing world). (See one page report here.)

If your MOOCs have significantly different demographics, I’d be interested in hearing your statistics. However, given the preponderance of evidence, your MOOC may be an outlier if you do have more students from the developing world.

The argument I’m making in this post is that (a) to improve education, we have to provide more to the underprivileged, (b) most MOOC students are affluent, well-educated students from the developing world, and (c) the benefits of MOOCs are thus accruing mostly to people who don’t need more enrichment. Some people are benefitting from MOOCs. My point is that they are people who don’t need the benefit. MOOCs are certainly not “democratizing education” and are mostly not providing opportunities to those who don’t have them anyway.


November 25, 2015 at 8:38 am 21 comments

Providing computing education to the developing world: How do we avoid educational imperialism?

I got an email from CodersTrust, asking me to help promote this idea of developing grants to help students in the developing world learn to code. But the education materials they’re offering is the same CodeAcademy, Coursera MOOCs, and similar developed-world materials. Should they be? Should we just be sending the educational materials developed for US and Europe to the developing world? I thought that that was one of the complaints about existing MOOCs, that they’re a form of educational imperialism.

CodersTrust is the brainchild of Ferdinand Kjærulff. As a Captain of the Danish army he served as recovery officer in Iraq after the fall of Saddam. He pioneered a recovery project with the allied forces, bringing internet and e-learning to the citizens of the region in which he was stationed. The project was a massive success and inspired him to eventually create CodersTrust – supported by Danida – with a vision to democratize access to education via the internet on a global scale.

via CodersTrust | About.

September 7, 2015 at 7:50 am 2 comments

ICER 2015 Preview: First CSLearning4U Ebook Paper

ICER 2015 (see website here) is August 9-13 in Omaha, Nebraska. The event starts for me and Barbara Ericson, Miranda Parker, and Briana Morrison on Saturday August 8. They’re all in the Doctoral Consortium, and I’m one of the co-chairs this year. (No, I’m not a discussant for any of my students.) The DC kickoff dinner is on Saturday, and the DC is on Sunday. My thanks to my co-chair Anthony Robins and to our discussants Tiffany Barnes, Steve Cooper, Beth Simon, Ben Shapiro, and Aman Yadav. A huge thanks to the SIGCSE Board who fund the DC each year.

We’ve got two papers in ICER this year, and I’ll preview each of them in separate blog posts. The papers are already available in the ACM digital library (see listing here), and I’ll put them on my Guzdial Papers page as soon as the Authorizer updates with them.

I’m very excited that the first CSLearning4U project paper is being presented by Barbara on Tuesday. (See our website here, the initial blog post when I announced the project here, and the announcement that the ebook is now available). Her paper, “Analysis of Interactive Features Designed to Enhance Learning in an Ebook,” presents the educational psychology principles on memory and learning that we’re building on, describes features of the ebooks that we’re building, and presents the first empirical description of how the Runestone ebooks that we’re studying (some that we built, some that others have built) are being used.

My favorite figure in the paper is this one:


This lists all the interactive practice elements of one chapter of a Runestone ebook along the horizontal axis (in the order in which they appear in the book left-to-right), and the number of users who used that element vertically. The drop-off from left-to-right is the classic non-completion rate that we see in MOOCs and other online education. Notice the light blue bars labelled “AC-E”? That’s editing code (in executable Active Code elements). Notice all the taller bars around those light blue bars? That’s everything else. What we see here is that fewer and fewer learners edit code, while we still see learners doing other kinds of learning practice, like Parsons Problems and multiple choice problems. Variety works to keep more users engaged for longer.

A big chunk of the paper is a detailed analysis of learners using Parsons Problems. Barbara did observational studies and log file analyses to gauge how difficult the Parsons problems were.  The teachers solved them in one or two tries, but they had more programming experience.  The undergraduate and high schools students had more difficulty — some took over 100 tries to solve a problem. Her analysis supports her argument that we need adaptive Parsons Problems, which is a challenge that she’s planning on tackling next.

August 5, 2015 at 7:35 am 3 comments

Older Posts

Recent Posts

February 2017
« Jan    


Blog Stats

  • 1,313,702 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 4,625 other followers

CS Teaching Tips