Worst practice in providing educational technology, especially to developing world

I followed an insightful chain of blog articles to this one.  I started with Larry Cuban’s excellent piece about “No End to Magical Thinking When It Comes to High-Tech Schooling” which cited the quote below, but first when through a really terrific analysis of the explanations that educational technology researchers sometimes make when hardware in dumped in the developing world fails to have a measurable impact.  I highly recommend the whole sequence for a deeper understanding of what real educational reform looks like and where technology can play a role.

1. Dump hardware in schools, hope for magic to happen

This is, in many cases, the classic example of worst practice in ICT use in education.  Unfortunately, it shows no sign of disappearing soon, and is the precursor in many ways to the other worst practices on this list.   “If we supply it they will learn”: Maybe in some cases this is true, for a very small minority of exceptional students and teachers, but this simplistic approach is often at the root of failure of many educational technology initiatives.

via Worst practice in ICT use in education | A World Bank Blog on ICT use in Education.

Living with MOOCs: Surviving the Long Open Learning Winter

One of the positive benefits of MOOCs is that a lot of faculty and administration are exploring educational innovations with technology.  When teachers explore how to facilitate learning, improved teaching and learning is likely to result. One of the problems is that many of these teachers and administrators are deciding that MOOCs and other open learning resources are the best bets for addressing educational problems.  They are buying into the belief that open learning is the best that there is (or, perhaps, the only thing that they found) and into the associated beliefs (e.g., that existing educational systems are ineffective and unsustainable, that “everyone already knows that a college degree means next to nothing“).  Those of us who do educational technology research and don’t do MOOCs are likely in for a stretch where our work will be under-appreciated, or simply ignored.  The AI community talks about their “AI Winter.”  Let’s call this the Open Learning Winter.

Regular readers of this blog (and I’m grateful that you are here!) know that I’ve been doing a good bit of traveling the last few months.  From MIT and Stanford, to Indiana and SIGCSE, I’ve had the opportunity to hear lots of people talk about the educational innovations that they are exploring, why they have decided on MOOCs and other open learning resources, and what they think about those of us who are not building MOOCs.  The below are paraphrased snippets of some of these conversations (i.e., some of the parts of these quotes are literally cut-and-paste from email/notes, while other parts are me condensing the conversation into a single quote representing what I heard):

• “You do ebooks?  Don’t you know about Connexions?  Why not just do Connexions books? Do you think that student interactivity with the ebook really matters?”
•  ”You’re making ebooks instead of MOOCs?  That’s really interesting.  Are you building a delivery platform now? One that can scale to 100K students this Fall?” As if that’s the only thing that counts — when no one even considered that scale desirable even a couple years ago.
• “Ebooks will never work for learning. You can’t ask them to read.  Students only want video.”
• “Anchored Collaboration sounds interesting. Can I do it with Piazza?  No?  Then it’s not really useful to anyone, is it?”
• “Why should we want to provide resources to state universities?  Don’t you know that all of their programs are going to die?”
• NSF Program officer at CCC MROE Workshop, “We better figure out online education.  All the state universities are going to close soon.”

These attitudes are not going to change quickly.  People are investing in MOOCs and other open learning resources.  While I do not believe that the MOOCopalypse will happen, people who do believe in it are making investments based on that belief.  The MOOC-believers (perhaps MOOCopalypse survivalists?) are going to want to see their investments will pan out and will keep pursuing that agenda, in part due to the driving power of “sunk costs” (described in this well done Freakonomics podcast).  That’s normal and reasonable, but it means that it will be a long time before some faculty and administrators start asking, “Is there anything other than MOOCs out there?”

I think MOOCs are a fascinating technology with great potential.  I do not invest my time developing MOOCs because I believe that the opportunity cost is too high.  I have had three opportunities to build a MOOC, and each time, I have decided that the work that I would be giving up is more valuable to me than the MOOC I would be producing.  I do not see MOOCs addressing my interests in high school teachers learning CS, or in end-users who are learning programming to use in their work, or in making CS more diverse. It may be that universities will be replaced by online learning, but I don’t think that they’ll all look like MOOCs.  I’m working on some of those non-MOOC options.

Researchers like me, who do educational technology but don’t do MOOCs, need to get ready to hunker down.  Research funding may become more scarce since there are MOOCopalypse survivalists at NSF and other funding agencies.  University administrators are going to be promoting and focusing attention on their pet MOOC projects, not on the non-believers who are doing something else.  (Because we should realize that there won’t be anything else!)  There will probably be fewer graduate students working in non-MOOC areas of educational technology.  Most of the potential PhD students who contacted me during this last application cycle were clear about how important MOOCs were to them and the research that they wanted to do.

We need to learn to live with MOOCs, even if we don’t do MOOCs.  Here are a couple of the hunkering down strategies I’ve been developing:

• While I don’t want to spend the time to build a MOOC, I am interested in being involved in analysis of MOOC data.  It’s not clear how much data Coursera or Udacity will ever release (and why isn’t edX releasing data — they’re a non-profit!), but I see a great value in understanding MOOCs.  We might also learn lessons that can be applied in other areas of educational innovation with technology.
• My colleagues involved in MOOCs at Georgia Tech have told me that we have the rights to re-use GT MOOC materials (e.g., all the video that has been collected).  That might be a source of interesting materials for my research.  For example, my colleague Jim Foley suggested that I might re-purpose video from a MOOC to create an ebook on the same content that might be usefully contrasted in a study.

I can’t predict just how long the Open Learning Winter might be.  Given the height of the hype curve associated with MOOCs and the depth of the pockets of the early adopters, I suspect that it’s going to be quite a long, cold winter.  Make sure that you have lots of jerky on-hand — and hope that it’s just winter and not an Ice Age.

Of MOOCs and Mousetraps: Making Curricular Decisions in Massive Courses

Karen Head is writing occasional blog posts about her efforts to teach introductory  composition in a MOOC.  I appreciate getting a chance to see into the design choices she needs to make to fit into the context of a massive on-line course.

From the beginning we have had logistical issues getting a large group together on a regular basis. After only three meetings, we decided to break into two main subgroups: one focusing on curricular decisions and the other on technical ones. My partner in this project, Rebecca Burnett (director of our Writing and Communication Program), and I attend all meetings to ensure that the two sides remain coordinated. Some of the key curricular decisions we needed to make immediately were the length and theme of the course, expected student commitment, types of assignments, and appropriate instructional approaches. We decided the course should last eight weeks rather than six to create a framework for students to understand the goals and approaches, and to allow time for more end-of-course reflection. We also decided to have a single “build on” main assignment; each week students will learn new skills and apply these to the continuing project. For our theme, which lends itself to our multimodal course goals, we will have students write and speak about a principle that guides their everyday lives.

via Of MOOCs and Mousetraps – Wired Campus – The Chronicle of Higher Education.

Online Learning Outcomes Equivalent to Traditional Methods: But what about the drops?

This is a great result, if I can believe it.  They took 605 students, some in a traditional course and some in a “hybrid” course, and did pre/post tests.  They found no difference in outcomes.

Here’s what I’m not sure about: What happened to those students who failed or who withdrew?  Other studies have suggested that online courses have higher withdraw/failure rates.  Is that the case here? There is only one footnote (page 18) that mentions withdraw/failure: “(27) Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of students who failed the course because the non-passing group includes students who never enrolled or withdrew from the course without receiving a grade.”  But that’s it.  If you lose more students in one format, and the students you lose are the weaker students (not an unreasonable assumption), then having the same learning gains doesn’t mean for all students.  It means that you’ve biased your sample.

The researchers asked the students to complete a number of tests and questionnaires before beginning the course and again after completing it, and they analyzed and compared the results between the two groups of students. The results revealed no statistical difference in educational outcomes between the two groups of students. In fact, the students in the hybrid course performed slightly better, but not enough to be statistically significant.

via Online Learning Outcomes Equivalent to Traditional Methods, Study Finds — Campus Technology.

CMU’s online efforts focus on sustainable business model – but not MOOCs

CMU’s OLI (Open Learning Inititative) is decades ahead of the MOOCs in terms of quality of offerings and evidence of effectiveness. So why aren’t they hyping their better product? I’ve asked that question several times of colleagues at CMU. This article explains it — basically, they’re focusing on sustainable business models, rather than launching into MOOCs without a clear revenue model.

Kamlet said Carnegie Mellon is working on an approach that encourages financially solvent business plans from the get-go.

“To be honest, I think there are some challenges – edX and Coursera would be the first to admit that they are going to see how things go over time in terms of a business model,” Kamlet said.

via Carnegie Mellon’s online efforts include spinoffs and subsidiaries but not MOOCs | Inside Higher Ed.

Using Education Theory to Frame CER: HCIL Invited Lecture

I gave a talk on 19 February at HCIL at U. Maryland-College Park.  I was pleased with how it turned out.  One of the things I learned when I gave my Indiana talks was that I ought to frame my talk with how I define learning and what theoretical frameworks I’m drawing on (e.g., learning sciences, constructionism, situated learning, community of practice, and authenticity).  This was my first talk where I tried to do that, and I liked how I could keep referencing back to the theory as I went along.  The talk gave me a chance to connect my work in computing education research (CER) to a broader education theory.

HCIL 30th Anniversary Distinguished Lecturer Series: Mark Guzdial, Georgia Tech – YouTube.

Pedagogy trumps Technology: Why All the Fuss about MOOCs?

Nice piece in our C21U newsletter, suggesting that pedagogy is more important than the MOOC technology.  How we teach is much more important for dramatic impacts on learning, than aiming for scale via advanced technology.

We may find that MOOCs work well for self-motivated students who have a lot of technology at their fingertips, have been raised in stimulating intellectual environments all their lives, who have lots of support mechanisms within their grasp to help them learn the material, and who have the wherewithal to spend the time and energy required to learn deeply what is being taught in these MOOCs.

But what about those students who don’t have the resources required to support their learning, who have not been raised in intellectually stimulating environments, who don’t even know how to study well? It is hard to see how MOOCs will work for these students, yet these are the students that it is most important that we reach in order to meet the challenges of 21st-century education.

I would much rather see the resources of Georgia Tech and our nation’s other educational institutions, being used to support the creation of research-based learning environments that can most effectively support the learning of all students, regardless of their background. Learning environments that do not rely on the lecture. Learning environments that make good use of those precious and valuable times when students are in direct contact with their instructors.

via Why All the Fuss about MOOCs? | Center for 21st Century Universities.

Taming the Monolith: Refactoring for an open source HyperCard

LiveCode had an earlier blog piece on how they want to implement “Open Language” so that the HyperTalk syntax could be extended.  This piece (linked below) goes into more detail and is an interesting history of how LiveCode evolved from HyperCard, and how they plan to refactor it so that it’s extensible by an open source community.

LiveCode is a large, mature software product which has been around in some form for over 20 years. In this highly technical article, Mark Waddingham, RunRev CTO, takes us under the hood to look at our plan to modularize the code, making it easy for a community to contribute to the project. The project described in this post will make the platform an order of magnitude more flexible, extensible and faster to develop by both our team and the community.

Like many such projects which are developed by a small team (a single person to begin with – Dr Scott Raney – who had a vision for a HyperCard environment running on UNIX systems and thus started MetaCard from which LiveCode derives), LiveCode has grown organically over two decades as it adapts to ever expanding needs.

With the focus on maintenance, porting to new platforms and adding features after all this time evolving we now have what you’d describe as a monolithic system – where all aspects are interwoven to some degree rather than being architecturally separate components.

via Taming the Monolith.

Grades are in for a pioneering free Johns Hopkins online class: Adding more to the public good

Some more statistics from another Coursera course.  The final comments are interesting: Through MOOCs, “everyone can get at least some fraction of what we believe is fundamental knowledge.”  That’s true.  The interesting question is whether MOOCs get more students a fraction that they didn’t have previously (see the edX data about 80% repeating the course) than a similar face-to-face course.  It’s not obvious to me either way — there are certainly results that have us questioning the effectiveness of our face-to-face classes.  While MOOCs lead to few finishing, maybe those that do finish learn more than in a face-to-face class, and maybe overall (amount of learning across number of students), MOOCs contribute more to the public good?

Read on for the final metrics on Caffo’s class and a few thoughts from the associate professor at the university’s school of public health.

Number of students who signed up for Caffo’s class: 15,930.

Number who ordinarily sign up for the class when it is taught solely on campus in Baltimore: a few dozen.

Active users in the final week of the class: 2,778

Total unique visitors who watched Caffo’s video lectures: 8,380

Total who submitted a quiz: 2,882

Total who submitted homework: 2,492

Total who passed the course (averaging 70 percent or better on quizzes): 748

Total who passed with distinction (averaging 90 percent or better): 447

And here is Caffo’s take:

“Regardless of how MOOCs wind up, it is awesome to be a professor in a time where teaching is the hottest topic in higher education at research-driven universities. I also have a lot of sympathy for democratizing education and information. Very few people will have the privilege of a Johns Hopkins University Bloomberg School of Public Health education. But, with these efforts [including free online initiatives such as Open Courseware, iTunes U, Coursera] everyone can get at least some fraction of what we believe is fundamental knowledge for attacking the world’s public health problems.”

via Grades are in for a pioneering free Johns Hopkins online class – College, Inc. – The Washington Post.

The Bigger Issues in Learning to Code: Culture and Pedagogy

I mentioned in a previous blog post the nice summary article that Audrey Watters wrote (linked below) about Learning to Code trends in educational technology in 2012, when I critiqued Jeff Atwood’s position on not learning to code.

Audrey does an excellent job of describing the big trends in learning to code this last year, from CodeAcademy to Bret Victor and Khan Academy and MOOCs.  But the part that I liked the best was where she identified the problem that cool technology and badges won’t solve: culture and pedagogy.

This is a problem. A big problem. A problem that an interactive JavaScript lesson with badges won’t solve.

Two organizations — Black Girls Code and CodeNow — did hold successful Kickstarter campaigns this year to help “change the ratio” and give young kids of color and young girls opportunities to learn programming. And the Irish non-profit CoderDojo also ventured state-side in 2012, helping expand afterschool opportunities for kids interested in hacking. The Maker Movement another key ed-tech trend this year is also opening doors for folks to play and experiment with technologies.

And yet, despite all the hype and hullaballoo from online learning startups and their marketing campaigns that now “everyone can learn to code,” its clear there are still plenty of problems with the culture and the pedagogy surrounding computer science education.

via Top Ed-Tech Trends of 2012: Learning to Code | Inside Higher Ed.

We still do need new programming languages whose design is informed by how humans work and learn.  We still do need new learning technologies that can help us provide the right learning opportunities for individual student’s needs and can provide access to those who might not otherwise get the opportunity.  But those needs are swamped by culture and pedagogy.

What do I mean by culture and pedagogy?

Culture: Betsy diSalvo’s work on Glitch is a great example of considering culture in computing education.  I’ve written about her work before — that she engaged a couple dozen African-American teen men in computing, by hiring them to be video game testers, and the majority of those students went on to post-secondary education in computing.  I’ve talked with Betsy several times about how and why that worked.  The number one reason why it worked: Betsy spent the time to understand the African-American teen men’s values, their culture, what they thought was important.  She engaged in an iterative design process with groups of teen men to figure out what would most appeal to them, how she could reframe computing into something that they would engage with.  Betsy taught coding — but in a different way, in a different context, with different values, where the way, context, and values were specifically tuned to her audience.  Is it worth that effort?  Yeah, because it’s about making a computing that appeals to these other audiences.

Pedagogy: A lot of my work these days is about pedagogy.  I use peer instruction in my classrooms, and try out worked examples in various ways.  In our research, we use subgoal labels to improve our instructional materials.  These things really work.

Let me give you an example with graphs that weren’t in Lauren Margelieux’s paper, but are in the talk slides that she made for me.  As you may recall, we had two sets of instructional materials: A set of nice videos and text descriptions that Barbara Ericson built, and a similar set with subgoal labels inserted.  We found that the subgoal labelled instruction led to better performance (faster and more correct) immediately after instruction, more retention (better performance a week later), and better performance on a transfer task (got more done on a new app that the students had never seen before).  But I hadn’t shown you before just how enormous was the gap between the subgoal labelled group and the conventional group on the transfer task.

Part of the transfer task involved defining a variable in App Inventor — don’t just grab a component, but define a variable to represent that component.  The subgoal label group did that more often.  ALOT more often.

Lauren also noticed that the conventional group tended to “thrash,” to pull out more blocks in App Inventor than they actually needed.  The correlation between number of blocks drawn out and correctness was r = -.349 — you are less likely to be correct (by a large amount) if you pull out extra blocks.  Here’s the graph of number of blocks pulled out by each group.

These aren’t small differences!  These are huge differences from a surprisingly small difference between the instructional materials.  Improving our pedagogy could have a huge impact.

I agree with Audrey: Culture and pedagogy are two of the bigger issues in learning to code.

This changes things: UK universities in online launch to challenge US

MOOCs from Open University UK (with its over 40 years of measurable success) at the lead? With Southampton (home of Dame Wendy Hall and Sir Tim Berners-Lee)? Now this really gets interesting. Hmm — OxBridge isn’t throwing hats into the rings yet.

A partnership of UK universities is launching an online project, challenging US universities that have dominated this emerging market.

They will aim to give the public access to higher education courses via computers, tablets or smartphones.

The partnership will include the Open University, King’s College London, Bristol, Exeter, Warwick, East Anglia, Leeds, Lancaster, Southampton, Cardiff, Birmingham and St Andrews.

Courses will be offered from next year.

via BBC News – UK universities in online launch to challenge US.

Academics and Venture Capitalists: Not close vs open, but evidence vs speculation

Aaron Bady’s essay critiquing Clay Shirky’s take on MOOCs is highly recommended reading (linked below).  I bought in to Shirky’s “Udacity is Napster” argument, and I still think that access may trump quality.  But Bady really highlights why there is such a distinction in views about MOOCs.  The first sentence below is wonderful, both pithy and true.  It’s not that MOOCs won’t be wonderful, it’s just that there’s not enough evidence that they will — but venture capitalists and “education disruptors” run on hope, not evidence.

The key difference between academics and venture capitalists, in fact, is not closed versus open but evidence versus speculation. The thing about academics is that they require evidence of success before declaring victory, while venture capitalists can afford to gamble on the odds. While Shirky can see the future revolutionizing in front of us, he is thinking like a venture capitalist when he does, betting on optimism because he can afford to lose. He doesn’t know that he’s right; he just knows that he might not be wrong. And so, like all such educational futurologists, Shirky’s case for MOOCs is all essentially defensive: he argues against the arguments against MOOCs, taking shelter in the possibility of what isn’t, yet, but which may someday be.

via Essay critiques the ideas of Clay Shirky and others advocating higher ed disruption | Inside Higher Ed.

Comparing MOOCs and books

Massive on-line courses are more comparable to books than face-to-face classes, an issue raised and discussed in the comments to the recent blog post about Larry Cuban and described pointedly in a recent comment by Mark Urban-Lurain on this blog.  A recent Chronicle of Higher Education commentary makes a similar claim:

A set of podcasts is the 21st-century equivalent of a textbook, not the 21st-century equivalent of a teacher. Every age has its autodidacts, gifted people able to teach themselves with only their books. Woe unto us if we require all citizens to manifest that ability.

I just came back from a visit to Stanford where John Mitchell, vice-provost for on-line education at Stanford, explained to me the value of MOOCs over textbooks.  Textbooks don’t provide much of a feedback mechanism to the author — you write the book, and you get feedback from your class and maybe a few teachers who adopt your book and provide you comments.  But MOOCs let you try out ideas at scale, even do A/B testing on how to present something, and get feedback for the next design iteration.  I pointed out to him that that’s true, but only if you can separate out the signal from the noise.  Which MOOC students do you really want to get feedback from?  The 80% of “students” who are re-taking a course they’ve taken before?  The 90% of enrollees who never planned to finish?

In the NYTimes piece linked below, I don’t agree with the claim that poor videos are the “trouble with online education.”  In fact, it paints too broad a stroke — there are lots of things which are online education which aren’t video-based, massive on-line courses.  But the basic claim is fair and reasonable and still interesting.

Not long ago I watched a pre-filmed online course from Yale about the New Testament. It was a very good course. The instructor was hyper-intelligent, learned and splendidly articulate. But the course wasn’t great and could never have been. There were Yale students on hand for the filming, but the class seemed addressed to no one in particular. It had an anonymous quality. In fact there was nothing you could get from that course that you couldn’t get from a good book on the subject.

via The Trouble With Online Education – NYTimes.com.

Certainly, there are differences between MOOCs and books.  I would predict that, in a comparison study, more people would learn more (meaning pre/post learning gain) from books than from MOOCs. Our current best-in-class MOOCs we have are less engaging than best-in-class books for most people.  Whether or not people finish the books they buy, many  people spend good money to purchase top-ranked books.  MOOCs barely get 20% finishing the course (after the first homework), when they don’t charge anything at all.  Sure, there’s not much of a carrot to finish a MOOC (e.g., no credit, no degree), but neither is there for a book.  The challenge is how to build on-line courses that are better than books!

A new MOOC Platform: Class2Go, Stanford’s New Open-Source Platform

MOOCs are still quite new, so it’s not surprising that more platforms are coming out.  It’s a little surprising that this is the third one to come out of Stanford.  Why is Stanford ground-zero for the MOOC movement?  Because Stanford is so entrepreneurial or innovative?  Because of the Silicon Valley culture which encourages exploring use of technology to solve problems, like education?  Because Stanford has deep enough pockets that they can afford to experiment?  I really don’t know, and do find it fascinating.

To unpack that a little: When Class2Go says it’s portable, it means that it wants to be platform agnostic. Its documents are already portable, its videos already live outside its system on YouTube, its assets can be repurposed as professors see fit and the platform’s exercises and problem sets are in the Khan Academy format (meaning they’re not in a proprietary database) and can be used anywhere.

In terms of interoperability, Class2Go’s website reads, “we don’t want to build or maintain more than we have to,” so it stands on the shoulders of, or relies significantly on, other services to run, like Khan, Piazza, YouTube, Python Django, Amazon AWS, Opscode and Github. Furthermore, designing the platform for both teaching and research means that the platform will leverage data to inform and evolve pedagogy, as well as to give them a glimpse into the efficacy of lessons, teaching style, tech tools, etc.

via Class2Go: Stanford’s New Open-Source Platform For Online Education | TechCrunch.

Setting an Agenda for Multidisciplinary Research for Online Education

The Computing Community Consortium (CCC) organizes efforts to focus computing research in the United States.  I’m on the organizing committee for this effort, triggered by MOOCs, to identify research issues in online learning for the computing research community.

Participants will explore and delineate computer science and multidisciplinary research agendas designed to improve formal and informal education. The workshop will build on CCC’s earlier visioning activities on Global Resources for Online Education (GROE), addressing education-relevant research in areas such as intelligent student modeling through data mining, mobile computing for data logging, social networking, serious games, intelligent learning environments, HCI to facilitate educational interactions, computer-supported collaborative learning, interactive visualizations and simulations, and many other areas, to include research at the interface of computing and the social/behavioral sciences.

While the workshop will build on a rich existing landscape of cyber-enabled education research, it also will be informed by very recent developments, such as massively open online courses (MOOCs), that make important dimensions of scale and openness explicit. Throughout the workshop, issues of education and learning quality will be also at the fore; how will the character of education change, and what are the important dimensions and evaluation methodologies for designing online educational instruments of quality at scale for different populations? What computing-relevant multidisciplinary research imperatives will grow to facilitate cyber-enabled transformations in online education?

This CCC visioning workshop will address these and related questions on computing-relevant multidisciplinary research, looking 5-10 years out, for online education. Importantly, the workshop will not address shorter-term concerns such as credentialing and business models for online education ventures, except as these inform the workshop’s focus on longer-term research agendas.

via Multidisciplinary Research for Online Education..