Completion Rates Aren’t the Best Way to Judge MOOCs, Researchers Say: What is?

February 14, 2014 at 1:26 am 22 comments

I’m completely open to the idea that completion rates are the wrong measures of success for MOOCs.  But I do believe that we need some measure.  What would success for MOOCs mean?  How do we know if it’s being achieved?  Or if it’s a waste of time and money?

In the meantime, the Harvard and MIT researchers said they hoped the new studies would help people understand that technology and scale are not the only things that distinguish MOOCs from other kinds of higher education.

“People are projecting their own desires onto MOOCs,” said Mr. Ho, “and then holding them accountable for criteria that the instructors and institutions and, most importantly, students don’t hold for themselves.”

via Completion Rates Aren’t the Best Way to Judge MOOCs, Researchers Say – Wired Campus – Blogs – The Chronicle of Higher Education.

Entry filed under: Uncategorized. Tags: , .

Shortage in the IT U.S. labor market? Or just a lack of graduates? CSTA Keynote Speakers for 2014: Yasmin Kafai and Michael Kölling

22 Comments Add your own

  • 1. shriramkrishnamurthi  |  February 14, 2014 at 1:50 am

    As Ho says, people are so unfair to these MOOCs, projecting their own desires. After all, it wasn’t Koller or Thrun who bragged and bragged about the enrollment numbers, nor was it Agarwal who said this was “the single biggest change in education since the printing press”. Oh wait…

    I too am open to the metric being something different _once the providers stop talking about enrollment rates_. So long as they keep putting out their enrollment numbers (which, in addition to the odd poor-orphan-in-Peru success story) as their primary talking point, they’re rightly being hoist on their own petard.

    • 2. Mike  |  February 14, 2014 at 3:59 am


  • 3. Mike  |  February 14, 2014 at 3:56 am

    I wonder if we could find a metric which measures how much the students actually learned during the course? Something along the lines of a pre/post test.

    This would at least sift out people taking the course to review topics they’d previously studied, or stuff they already have a strong background in (e.g., I have a degree in CompSci, when I pass a course in compilers I’m learning but not as much as someone going for their undergrad degree would).

    • 4. Mark Urban-Lurain  |  February 14, 2014 at 11:09 am

      There’s the rub. In none of our higher education efforts to we actually “measure” learning. We award credit for seat time. No, instructor created assessments are not an objective measure of learning.

      It’s a great suggestion to certify based on performance on some assessment for ANY course (F2F, MOOC, whatever), but getting agreement on what those assessments should be will cause a firestorm of that makes the current discussion of MOOCs seem like a love-fest.

      • 5. gasstationwithoutpumps  |  February 14, 2014 at 12:11 pm

        Actually, some physics classes do use externally produced “concept inventories” to measure learning. The results are usually pretty discouraging, but they have lead to some improvements in pedagogy (changing results from showing that no learning was happening to merely depressing).

        • 6. Mark Urban-Lurain  |  February 14, 2014 at 12:31 pm

          True, many faculty use concept inventories for pre/post formative feedback, but I am not aware of any that assign a grade (hence the credential) based ONLY on these instruments.

          • 7. gasstationwithoutpumps  |  February 14, 2014 at 12:59 pm

            Measuring learning and assigning grades are not the same thing. Concept inventories are rarely used for assigning grades, because attaching grading to a measurement tends to distort the measurement.

            • 8. Mark Urban-Lurain  |  February 14, 2014 at 1:21 pm

              That was the point of my original reply (#4 above). We rarely measure learning, we “credential” students, which is what most students are paying for: some credential (usually a grade in a course, the sum of which results in a degree) that can be used for other purposes, such as finding a job, admission to grad school, etc. Hence, the surge in interest in “badges” for MOOCs as evidence of an outcome that students can use as their credential.

              There are no external assessment entities that award the credentials independently of the content providers. Traditional educational institutions have a monopoly via the accreditation process. They provide content and they do the credentialing. If credentialing were unbundled from content provision, then there would be competition and disruption could occur. An example of this disruption has been seen in the news business, as content creation has been unbundled from delivery (newspapers, broadcast news, etc.)

        • 9. Mark Guzdial  |  February 14, 2014 at 1:42 pm

          The Physics MOOCs here at Georgia Tech are using Physics concept inventories pre/post. The Force Concept Inventory (FCI) was the explicit motivation for Allison Elliott Tew’s FCS1.

          • 10. Peter Donaldson  |  February 14, 2014 at 7:48 pm

            The other interesting outcome of the use of force concept inventories in physics was the invention of peer instruction by Eric Mazur at Harvard. When he used one and the poor results made him realise many physics undergrads conceptual understanding was flawed it must have been extremely sobering. The fact that he didn’t take the path of least resistance and blame the students speaks volumes.

            The closest we’ve come to something similar is probably the simple problem statements that so many students still can’t create a correct solution for after a semester or more of programming instruction.

      • 11. Mark Guzdial  |  February 14, 2014 at 1:41 pm

        Mark, are your programs at MSU ABET-accredited? To get our Computational Media degree accredited, we had to come up with learning outcomes, and measures of those outcomes (mostly rubric-based), and the measures are applied apart from grades (in some cases, by someone external to the instructor and class). Isn’t that a measure of learning? “None” might be a bit too harsh.

        • 12. Mark Urban-Lurain  |  February 14, 2014 at 2:19 pm

          Hi Mark,

          Yes, MSU’s engineering programs are accredited. BUT, as you may know, ABET has each institution define what they believe is evidence of the a-k outcomes. The usual way this works is that each program prepares a mapping of courses to ABET criteria and provides some samples of student work ON THE INSTITUTION’S ASSESSMENTS that they believe show a range of student performance to meet ABET those criteria. The ABET reviewers examine those and may recommend changes when they do not believe the criteria are met, but each program still decides/revises the criteria based on their own assessments of student outcomes (Note that the ABET language is “outcome” not “learning”.)

          The big advantage of ABET over other accreditation (e.g., North Central) is that ABET does at least require programs to articulate outcomes, as opposed to inputs (e.g., take 4 semesters of calc).

          There is no external assessment of student learning, except for things like the PE exams that some engineers must take to be licensed. But the first thing a PE examinee needs is a degree from an accredited institution. So, even in engineering, the content providers are also the ones who do the credentialing.

          “None” might be harsh but I’ll wait for someone to show me the white crow that refutes it.

          • 13. Bri Morrison  |  February 16, 2014 at 11:28 am

            Although certainly not popular any more, many institutions used the ETS CS Field Test as an outside assessment of the quality of their degree program (which I think would qualify as your white crow). 🙂

            • 14. Mark Urban-Lurain  |  February 16, 2014 at 12:10 pm

              Assessing the program is different than credentialing individual students. Do you know of programs that determine grades in courses for individual students based on assessments created by any entity outside their own depts?

              • 15. Bri Morrison  |  February 16, 2014 at 12:24 pm

                While only anecdotal, some high school courses give large weight to scores received on AP exams as part of the student’s final grade in the course.

                And I remember seeing several schools that required passing the ETS CS Field test as a requirement for graduation – which would be credentialing the individual student (although for the degree, not just a course).

                • 16. Mark Urban-Lurain  |  February 16, 2014 at 12:55 pm

                  Making passing the ETS CS Field Test a requirement for graduation would be an example of an external criteria. Do these same institutions award degrees to any student who can simply pass this exam, or do they require them to FIRST pass all of their courses and local assessments? Logical AND or OR?

                  • 17. Bri Morrison  |  February 16, 2014 at 1:37 pm

                    Logical AND. Although the logical OR cause would certainly be interesting….

  • 18. lizaloop  |  February 14, 2014 at 8:30 pm

    Perhaps a survey questionnaire that captured the registrant’s level of involvement and anticipated purpose for enrolling would help understand what folks want from the course. Those who indicate that they are in it to “learn the material presented” could then be offered a graded “achievement test” that is scored for mastery. (I assume that this is already included in any MOOC that anticipates being used for university credit.) An additional test, similar to an SAT, could be offered to help students, evaluators and instructional designers situate the learner within the field could be offered both pre- and post-MOOC.

    No single test or evaluation instrument can be all things to all people. Giving learners a range of instruments against which to measure their position in the field of endeavor, their gains from participation in the MOOC and their certifiability will provide lots of data for researchers to explore.

  • 19. Show me the money! | Journey of a Chinese MOOCer  |  February 15, 2014 at 8:42 am

    […] is discussed quite often what defines the success of MOOC. Some says completion rates aren’t the best way to judge MOOCs, but then what is the […]

  • 20. Dennis J. Frailey  |  February 19, 2014 at 1:10 pm

    It seems to me that, as much as some of us may dislike this answer, the only way to judge the value of MOOCs is if they cost money and we see how much people are willing to pay for them.

    One model is to have them pay a token amount for each lecture. If, instead of free, a MOOC cost $1 per session after the initial few sessions, my guess is that the number of enrollees would drop significantly. That would immediately show how many of them are not serious and are unlikely to get much out of the MOOC. (Some games on smart phones work this way – the game is free but after a while you must make a token payment ($1) to continue playing or to advance to higher levels. Reportedly there are some Silicon Valley millionaires who made their money in this manner.)

    Perhaps an even better measure would be to increase the cost after each session. Free for the first one, $1 for the second,, $2 for the third, and so forth until, perhaps $10 for the last sessions. It would still be a tremendous bargain but we would see how many are serious.

    Of course the methods above only filter out those who don’t take it seriously. How do we evaluate the effectiveness of the teaching? I think the only way is to require students to complete homework /assignments /term papers and the like. These, of course, must be graded and this, of course, gets back to a fundamental issue, in my mind, which is that in order to be effective you have to evaluate student work that reflects whether or not they learned anything.

    Another model is to charge for grading the work even if the lectures are offered free of charge. And, of course, any degree or certification or other credential should be dependent on good grades in the student work.

    In short, what I am recommending here is that it should cost the student something. The idea is not to make the course developers rich but to use the (token?) payment to determine what value the students ascribe to what they are getting.

    It has been my view for some time that making education free or too inexpensive tends to denigrate its value in the minds of students. I can certainly understand how K-12 education ought to be free or nearly so, because the children don’t have the means to pay. But i think we sometimes make it so inexpensive to get a college education that people see it as an entitlement with little actual value rather than as a precious opportunity that they are fortunate to have available. Human nature works like this. For example, when water is abundant, people don’t think about its value, but when it is scarce, suddenly its importance and value are recognized.

    • 21. shriramkrishnamurthi  |  February 19, 2014 at 1:17 pm

      @Dennis, isn’t this the wrong way around? You want to charge $10 for the first lecture, $9 for the second, and so on. If someone actually has the staying power to make it to week ten, they’re clearly there for the learning, and you stop charging them. (This is setting aside the question of the impression conveyed by charging nothing for education, who’s actually paying for the content, etc.)

      It’s understood in numerous contexts that the way to avoid frivolous signups is to charge a modest registration fee. Some people (we have done this for our own workshops) even reimburse the fee if the person shows up (or can show a legitimate reason for not having been able to make it), when the purpose of charging the fee is not to make money but to avoid frivolity.

      However, charging even a small fee to reduce numbers does not fit the rhetoric of of MOOCs, which retain a significant hype and ego component through these large numbers.

      • 22. Dennis J. Frailey  |  February 19, 2014 at 1:30 pm

        I see your approach as viable in some circumstances. It will certainly filter out those who are not serious about taking the MOOC. I see a few disadvantages of the approach: 1) you filter out the person who would be interested if they knew more about what it is all about – you don’t give them a chance to get a taste of what it is like unless they pay the large, upfront fee; 2) some who decide they want to quit will stay in just to get their money back (or the associated credential). So you will have students who are not really serious but “stay the course” for reasons that have little to do with becoming educated; 3) while your method assesses the student’s interest in the lectures, it does not assess the actual effectiveness of the MOOC in educating the student (but then you did explicitly leave out the question of grading and who pays for that).

        What it probably boils down to is that you have to design the system to fit the culture of the target audience. Some cultures value education highly whereas others do not. So the approach that works for one may not work for others.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,030 other followers


Recent Posts

Blog Stats

  • 1,994,760 hits
February 2014

CS Teaching Tips

%d bloggers like this: