We measure educational productivity wrong: Not numbers-served but learning
September 7, 2013 at 1:22 am 6 comments
The Washington Post series on “The Tuition is Too Damn High” has been fascinating, filled with interesting data, useful insights, and economic theory that I hadn’t met previously. The article linked below is about “Baumol’s cost disease” which suggests an explanation for why wages might increase when productivity does not. It’s an explanation that some have used to explain the rise in tuition, which Post blogger Dylan Matthews rejects based on the data (in short: faculty salaries aren’t really rising — the increase in tuition is due to other factors).
But I actually had a concern about an earlier stage in his argument. It’s absolutely true that our labor intensive methods do not lead to an increase in productivity in terms of number of students, while MOOCs and similar other methods can. However, we can gain productivity in terms of quality of learning and retention. We absolutely have teaching methods, well-supported with research, that lead to better learning and more retention — we can get students to complete more classes with better understanding. In the end, isn’t THAT what we should be measuring as productivity of an educational enterprise, not “millions of customers served” (even if they don’t complete and don’t learn)?
Performing a string quartet will always require two violinists, a violist and a cellist. You can’t magically produce the same piece with just two people. Higher education, for at least the past few millennia, has seemed to fall in this category as well. “What just happened in my classroom is not very different from what happened in Plato’s academy,” quips Archibald. We’ve gotten better at auditorium-building, perhaps, but lecturers generally haven’t gotten more productive.
via The Tuition is Too Damn High, Part V — Is the economy forcing colleges to spend more?.
Entry filed under: Uncategorized. Tags: computing education, economics, MOOCs, public policy, teachers.
6 Comments Add your own
Leave a reply to Mark Guzdial Cancel reply
Trackback this post | Subscribe to the comments via RSS Feed
Computing Ed Research - Guzdial's Take 
1.
Turadg Aleahmad | September 8, 2013 at 11:59 am
Interesting point, Mark. How would you quantify that? And on that metric how much has the quality of teaching improved over, say, 1950?
To compare with the student throughput metric, how much would a professor’s quality have to improve to match the productivity gains of teaching 2, 10x, 100x students in the same amount of time at the baseline quality? What would that look like qualitatively?
2.
Mark Guzdial | September 8, 2013 at 1:43 pm
Turadg, just to challenge an assumption I think you’re making: I don’t believe that MOOCs are currently “producing” students at the same baseline quality as face-to-face instruction. It’s the issue of net gain of learning that came up a bit ago.
There are only a few places where there are quantifiable changes in CS ed due to adoptions of new methods. UCSD and Georgia Tech have both significantly improved retention through curricular changes. We’re just starting to develop reliable measures of learning in CS. My argument is that measures of learning and retention are a better yardstick than numbers-served, but I can’t claim yet that we have all the instruments to make those measurements. In the end, maybe that’s why we measure “butts-in-seats” — it’s way easier to measure.
3.
Turadg Aleahmad | September 8, 2013 at 8:53 pm
I agree that measures of learning and retention of students are important measures of the “productivity” of an educational offering. (I assume you meant retention of students rather than of knowledge, leaving that implicit in “learning”.) My question wasn’t meant to challenge that point so much as to invite a discussion of the numbers that an economist would need in order to compare that kind of measure with the throughput measure.
As you allude to, theories are often built on what’s easiest to measure. I want to see quality measured so that it can be incorporated in these types of analyses. I’d also like to support the public discussion through quantifying the “net gain” as you put it in the myths thread. That will also help us figure out what modalities are most cost-efficient for the different goals of different learner profiles.
So I think we agree on some of the questions. I was just hoping you somehow had the answers. 😉
4.
gasstationwithoutpumps | September 9, 2013 at 9:01 am
I think you missed the point: some desired outcomes are intrinsically hard to quantify. Defining what you want with a quantifiable measure ensures that you will not include inherently unquantifiable outcomes. About the best we can do is come up with quantifiable proxies that approximate what we are really interested in, but optimizing for the proxy may make it a poorer measure of what we are really interested in. This is the danger of the “teach to the test” approach to pedagogy—the test is never a perfect measure of what we want.
Student retention is a excellent example of a desired outcome that can be badly gamed. You can always improve “retention” by not removing students who have failed, cheated, or not shown up—all of which distort the meaning of the measure to the point where it is worse than useless.
5.
Turadg Aleahmad | February 2, 2014 at 3:18 pm
Yes, some desired outcomes are hard to quantify. My point is that they’re important to quantify anyway. Yes, quantifying them through poor proxies can lead to optimizations that are actually detrimental to what we truly care about.
But I think we can and must develop better measures. I don’t think it’s impossible. It will require being more explicit about the responsibilities of different actors in our educational system; it will require difficult science and politics, but it will be worth those efforts.
by the way, why do you post under a pseudonym?
6.
gasstationwithoutpumps | February 2, 2014 at 5:19 pm
I started the blog gasstationwithoutpumps initially sort of anonymously, and have used that name for my blogging and blog commenting. More recently, I’ve been announcing a lot of my blog posts under my own name, so 70–80% of my readers probably know who I am. I’ve still not decided to whether to put my name on the blog itself, though anyone with an ounce of search skills could find out who I am within about 60 seconds.