Thought Experiments on Why Face-to-Face Teaching Beats On-Line Teaching: We are Humans, not Econs

With everything moving on-line, I’m seeing more discussion about whether this on-line life might just be better. Amy Ko recently blogged (see post here) about how virtual conferences are cheaper, more accessible, and lower carbon footprint than face-to-face conferences, ending with the conclusion for her “it is hard to make the case to continue meeting in person.” My colleague, Sarita Yardi, has been tweeting about her exploration of “medium-independent classes” where she considers (see tweet here), “Trying to use the block of class time just because that’s how we’ve always taught seems like something to revisit. Less synchronous time; support short, frequent individual/small group interaction, less class time.”

It’s hard to do on-line education well. I used to study this kind of learning a lot (see post on “What I have learned about on-line collaborative learning”). I recently wrote about how we’re mostly doing emergency remote teaching, not effective on-line learning (see post here). I am concerned that moving our classes on-line will hurt the most the students who most need our help (see post here).

It should come as no surprise then that I don’t think that we know how to do on-line teaching or on-line conferences in a way that is anywhere close to the effectiveness of face-to-face learning. I agree with both Amy and Sarita’s points. I’m only focusing on learning outcomes.

Let me offer a thought experiment on why face-to-face matters. How often do you…

• Look at the movie trailer and not watch the movie.
• Watch the first few minutes of a show on Netflix but never finish it.
• Start a book and give up on it.
• Start watching a YouTube video and immediately close it or click away.

Now contrast that with: How often do you…

• Get up from a one-on-one meeting and walk out mid-discussion.
• Get up in the middle of a small group discussion and leave.
• Walk out of a class during a lecture.
• Walk out of a conference session while the speaker is still presenting (not between talks or during Q&A).

For some people, the answers to the first set are like the answers for the second set. I tried this thought experiment on my family, and my wife pointed out that she finishes every book she starts. But for most people, the first set is much more likely to happen than the second set. This is particularly hard for professors and teachers to recognize. We are good at self-regulated learning. We liked school. We don’t understand as well the people who aren’t like us.

There are a lot of people who don’t really like school. There are good reasons for members of minority groups to distrust or dislike school. Most people engage in higher-education for the economic benefit. That means that they have a huge value for the reward at the end, but they don’t particularly want to go through the process. We have to incentivize them to be part of the process.

Yes, of course, many students skip classes. Some students skip many classes. But the odds are still in favor of the face-to-face classes. If you are signed up for a face-to-face class, you are much more likely to show up for that class compared to any totally free and absolutely relevant to your interests lecture, on-campus or on-line. Enrolling in a course is a nudge.

For most people, you are much more willing to walk away from an asynchronous, impersonal event than a face-to-face, personal event. The odds of you learning from face-to-face learning are much higher simply because you are more likely to show up and less likely to walk out. It’s a great design challenge to make on-line learning opportunities just as compelling and “sticky” as face-to-face learning. We’re not there yet.

I would be all in favor of efforts to teach people to be more self-regulated. It would be great if we all were better at learning from books, lectures, and on-line resources. But we’re not. The learners with the best preparation are likely the most privileged students. They were the ones who were taught how to learn well, how to learn from school, and how to enjoy school.

Here’s a second thought experiment, for people who work at Universities. At any University, there are many interesting talks happening every week. For me, at least a couple of those talks each week are faculty candidates, which I am highly encouraged to attend. Now, they’re all on-line. How many of those did you attend when they were face-to-face, and how many do you attend on-line? My guess is that both are small numbers, but I’ll bet that the face-to-face number is at least double the on-line number. Other people see that you’re there face-to-face. There are snacks and people to visit with face-to-face. The incentives are far fewer on-line.

On-line learning is unlikely to ever be as effective as face-to-face learning. Yes, we can design great on-line learning, but we do that fighting against how most humans learn most things. Studies that show on-line learning to be as effective (or even more effective) than face-to-face classes are holding all other variables equal. But holding all other variables equal takes real effort! To get people to show up just as much, to give people as much (or more) feedback, and to make sure that the demographics of the class stay the same on-line or face-to-face — that takes significant effort which is invisible in the studies that are trying to just ask face-to-face vs on-line. The reality is that education is an economic endeavor. Yes, you can get similar learning outcomes, at a pretty high cost. At exactly the same cost, you’re unlikely to get the same learning outcomes.

We are wired to show-up and learn from face-to-face events. I would love for all of us to be better self-regulated learners, to be better at learning from books and from lecture. But we’re not Econs, we’re Humans (to use the Richard Thaler distinction). We need incentives. We need prompts to reflect, like peer instruction. We need to see and be seen, and not just through a small box on a 2-D screen.

So much to learn about emergency remote teaching, but so little to claim about online learning

The Chronicle of Higher Education published an article by Jonathan Zimmerman on March 10 arguing that we should use the dramatic shift to online classes due to Covid-19 pandemic as an opportunity to research online learning (see article here).

For the first time, entire student bodies have been compelled to take all of their classes online. So we can examine how they perform in these courses compared to the face-to-face kind, without worrying about the bias of self-selection.

It might be hard to get good data if the online instruction only lasts a few weeks. But at institutions that have moved to online-only for the rest of the semester, we should be able to measure how much students learn in that medium compared to the face-to-face instruction they received earlier.

To be sure, the abrupt and rushed shift to a new format might not make these courses representative of online instruction as a whole. And we also have to remember that many faculty members will be teaching online for the first time, so they’ll probably be less skilled than professors who have more experience with the medium. But these are the kinds of problems that a good social scientist can solve.

I strongly disagree with Zimmerman’s argument. There is a lot to study here. There is little to claim about online learning.

What we are doing right now is not even close to best practice for online learning. I recommend John Daniels’ book Mega-Universities (Amazon link). One of his analyses is a contrast with online learning structured as “correspondence school” (e.g., send out high-quality materials, require student work, provide structured feedback) or as a “remote classroom” (e.g., video record lectures, replicate in-classroom structures). Remote classrooms tend to have lower-retention and increase costs as the number of students scale. Correspondence school models are expensive (in money and time) to produce, but scales well and has low cost for large numbers. What we’re doing is much closer to remote classrooms than correspondence school. Experience with MOOCs supports this analysis. Doing it well takes time and is expensive, and is carefully-structured. It’s not thrown together with less than a week’s notice.

My first thought when I read Zimmerman’s essay was for the ethics of any experiment comparing to the enforced move to online classes versus face-to-face classe. Students and faculty did not choose to be part of this study. They are being forced into online classes. How can we possibly compare face-to-face classes that have been carefully designed, with hastily-assembled online versions that nobody wants at a time when the world is suffering a crisis. This isn’t a fair nor ethical comparison.

Ian Milligan recommends that we change our language to avoid these kinds of comparisons, and I agree. He writes (see link here) that we should stop calling this “online learning” and instead call it “emergency remote teaching.” Nobody would compare “business as usual” to an “emergency response” in terms of learning outcomes, efficiency, student satisfaction, and development of confidence and self-efficacy.

On the other hand, I do hope that education researchers, e.g., ethnographers, are tracking what happens. This is first-ever event, to move classes online with little notice. We should watch what happens. We should track, reflect, and learn about the experience.

But we shouldn’t make claims about online learning. There is no experiment here. There is a crisis, and we are all trying to do our best under the circumstances.

Indian Supreme Court rules that CS degrees cannot be provided on-line

On-line CS degrees cannot be on par with face-to-face CS degrees, rules the highest court in India.

What makes an online class worse than a face-to-face class?  I think that there’s a good bit of evidence that they are worse for many students, e.g., the NYTimes article about how on-line classes hinder students needing remedial help. I’m interested in the research question of why on-line classes have such differential results from face-to-face classes — it isn’t obvious to me.  But the court’s rules says that on-line classes always are worse, and even that they always will be.  There’s a design space to explore, and it’s short-sighted to rule that on-line is always worse and never on par.

But how should the court have ruled? We need a measure of quality such that we can compare the results of the two degrees.  It’s hard to figure out what such a measure might be — maybe success in work, or employability, or even a measure of skill or content knowledge.  Any reasonable measure would be better than making a decision based on the medium.

(Thanks to Amy Bruckman for sending me this article.)

The Court affirmed the findings of the Punjab and Haryana High Court which had given a similar ruling two years ago, stating that a degree in the subject of ‘computer science’ from a distance learning course could not be considered on par with one attained by attending regular classes.

The verdict came on a batch of appeals challenging the orders of the High Courts of Orissa and Punjab and Haryana by which the former held the degrees in engineering obtained by serving diploma holders through distance learning mode offered by certain deemed universities to be valid, whereas the latter termed such degrees to be invalid.

The top court also directed the University Grants Commission (UGC) to restrain such institutions from using the word ‘University’ within one month from today, observing that commercialisation of education “seriously affects the credibility of standards in education, eroding power and essence of knowledge and seriously affecting excellence and merit”.

Source: Cannot provide technical education through correspondence, rules Supreme Court

Teasing out the meaning of “online classes” — Online Courses Are Harming the Students Who Need the Most Help: NYTimes

The NYTimes published an interesting piece on the state of online education today. Increasingly, online education is being used in schools as a response to students failing in face-to-face, traditional classrooms.  If you’re not making it in the regular class, try it again in the online class.  The article describes how that’s not working. Students who fail in traditional classes need more personal contact and support, not less.

I love that the name of the column where this article appeared is called “The Economic View,” because that’s exactly what it is.  We do now how to teach every student well — give each child a well-educated teacher for their particular subject (Bloom’s two-sigma effect). We can’t afford that, so we make do with less.  But we should aim to do no harm.  Current practice with online classes is clearly doing harm.

The NYTimes article is reporting on empiricism.  We cannot empirically determine what might online classes become. The author, Susan Dynarski of the University of Michigan, is reporting on current practice and on the result of policy.  Can online classes help students?  Absolutely, and the OMS CS is a good example of that.  Can we build online classes that work better for students who struggle with traditional classes?  Maybe — it’s hard to see them in this study. At the ECEP 2018 meeting, Caitlin Dooley (Associate Superintendent for Georgia) said that their online classes do better than face-to-face classes, in part because of caring (“mama bear”) teachers who support the students outside of the online classes.  The online classes that Susan Dynarski is studying are clearly not working well for struggling students.  There may already be models that work well, but they’re swamped in a general study of policy across different kinds of online classes.  Dynarski’s article may just be telling us that the current average practice is insufficient. There may be better models (maybe still in research) that could correct these ills.

Dynarski’s article is fascinating and is sounding an important alarm. It should be even greater motivation for those of us who are working to invent better online education.

Online education helps school districts that need to save money make do with fewer teachers. But there is mounting evidence that struggling students suffer.

In the fully online model, on the other hand, a student may never be in the same room with an instructor. This category is the main problem. It is where less proficient students tend to run into trouble. After all, taking a class without a teacher requires high levels of self-motivation, self-regulation and organization. Yet in high schools across the country, students who are struggling in traditional classrooms are increasingly steered into online courses.

Source: Online Courses Are Harming the Students Who Need the Most Help

Prediction: The majority of US high school students will take CS classes online #CSEdWeek

The Washington Post got it wrong when it announced that Virginia is the first state to mandate CS education for all students.  South Carolina has had that mandate for 30 years.  But they couldn’t prepare enough teachers to teach computer science, so they took classes they were already teaching (like “keyboarding”) and counted those as CS classes.

Virginia could fall into the same trap, but I don’t think so.  Instead, I predict that most Virginia high school students will take CS on-line (and that likely goes for the rest of the US, too).  I was struck by how the Richmond-Times Dispatch described the vote to mandate CS (below quoted from here):

The standards, approved unanimously, but reluctantly, by the state Board of Education on Thursday, are a framework for computer science education in the state. Other states have advisory standards, but Virginia became the first to have mandatory standards.

Board member Anne Holton voiced her concern with the grade level appropriateness of the standards before the vote.

“The standards, they seem ambitious to me,” she said. “These are not meant as aspirational standards, they are meant as a mandate that our teachers need to be able to teach.”

“We’re clearly leading the nation and that puts an extra burden on us to get it right.”

Mark Saunders, the director of the Education Department’s Office of Technology and Virtual Learning, led a presentation of the department’s process in adopting the standards.

The presentation satisfied the board enough to vote on the standards rather than delay action until January.

I’m reading between the lines here, but I’m guessing the process went something like this: Board members balked at a statewide mandate because they knew they didn’t have the teachers to support it. There certainly are CS teachers in Virginia, many of them prepared by CodeVA. But not enough to support a statewide mandate. Then they were assured that the Virtual Learning system could handle the load, so they voted for it (“reluctantly” as the article says).

I don’t know that anybody’s tracking this, but my guess is that it’s already the case that most high school students studying CS in the United States are doing it online.  Since we are not producing enough new CS teachers, the push to grow CS education in high schools is probably going to push more CS students online. This is how schools in Arkansas and other states are meeting the requirements for schools to offer CS — simply make the virtual high school CS course available, and you’ve met the requirement. No teacher hiring or professional learning required.  I know from log file analyses that we are seeing huge numbers of students coming into our ebooks through virtual high school classes.

What are the ramifications of this trend?  We know that not everyone succeeds in online classes, that they tend to have much higher withdrawal and failure rates. We know that most people learn best with active learning (see one of my posts on this), and we do not yet know how to replicate active learning methodologies in online classes.  In particular, lecture-based learning (which is what much of online learning attempts to replicate) works best for the most privileged students.  Our society depends on teachers who motivate students to persevere and learn. Does serving high school CS through online classes increase accessibility, or decrease diversity of those who successfully complete high school CS classes?  Will students still be interested in pursuing CS in the future if their only experience is through a mandated online course?  Does the end result of mostly-online high school CS classes serve the goals of high-quality CS education for all students?

 

Finding that MANY students get lousy returns on online education, but SOME students succeed

The point made below is that online education does work for some students. Our OMS CS succeeds (see evidence here) because it serves a population that has CS background knowledge and can succeed online. Not everyone succeeds in MOOCs.  I don’t like the first sentence in this piece.  “Online education” can be effective.  The models matter.

Despite Hoxby’s troubling findings, it’s hard to say whether online education in and of itself is inherently problematic or whether certain models could be successful. Goodman’s research on a Georgia Institute of Technology online master’s in computer science program indicates that, if done right, an online degree can provide a decent education at a fraction of the cost.“That model doesn’t generalize very well to the broader set of people that are out there,” he said. That’s because the students in the Georgia Tech program have already proved themselves to be successful in higher education (the admissions standards are relatively similar to the school’s elite brick-and-mortar computer science program), which is often not the case for many of the 30-something students that are typical of online education programs.

Source: Damning study finds students get lousy returns on online education – MarketWatch

Friction Between Programming Professionals and Beginners

It’s not obvious that professional programmers are the best people to answer questions for beginners, yet that’s often recommended as a strategy for providing support to CS students when there are too few teachers.  The below article gathers some stories about user experience, and offers advice on how to make the interaction of programming professionals and beginners more successful.

Where is the most obvious place to ask a programming question? Stack Overflow.

Stack Overflow is a community of 4.7 million programmers, just like you, helping each other. Join them; it only takes a minute. Join the Stack Overflow community to: Ask programming questions, […] — Stack Overflow (the front page)

It sounds like exactly the right place to be. It even sounds friendly. But actually asking questions on Stack Overflow is often far from friendly, for a beginner programmer.

“I gave up programming after the last time I asked a question on StackOverflow.”— commenter on reddit

Stack Overflow users ands moderators are quick to downvote and close questions, for a multitude of reasons. These reasons are often surprising to first-time users.

I’m going to pick on Stack Overflow as an example in this article, because it is the most obvious place to ask questions, but the same problems can be seen anywhere that beginners ask questions.

“I must have gone to a couple dozen IRC rooms, whatever online communities I could find. Everywhere I went people shat on me, and I never got an answer to a single question.”— commenter on reddit

Source: Friction Between Programming Professionals and Beginners – Programming for Beginners

Wisdom of massive open online courses now in doubt: Hennessy critiques MOOCs

I thought John Hennessy’s quote below was remarkable, and quite different from his tsunami rhetoric of just last July.  I was also struck by this quote later in the piece:   “MOOCs are basically the 21st-century equivalent of reading a bunch of books and saying you got a degree.”

“Two words are wrong in ‘MOOC’: massive and open,” Stanford President John Hennessy said in a widely noted interview with the Financial Times.

At Tufts University outside Boston, members of the schools of arts and sciences and faculty in the engineering department approved a policyin December that would allow more Web-based classes to be used toward graduation. But Tufts instructors stopped short of joining the world of MOOCs.

“So much of the big conversation around the country is around these massive online courses, and from our perspective, we don’t see evidence that that’s a model that leads to real learning,” Education Policy Committee head David Hammer told The Tufts Daily, the school newspaper. “If I had 750 students, if I had 7,500 students I’m not going to hear and respond to student thinking.”

via Wisdom of massive open online courses now in doubt – Washington Times.

SIGCSE 2014 Preview: How Khan Academy gamifies CS

Briana Morrison and Betsy DiSalvo use theory about gaming and media to analyze how Khan Academy “gamifies” the study of computer science.  What do they get right?  What are they missing?  Thursday from 10:45-12 in Room Regency VI.

Gamification is the buzzword for adding gaming elements such as points or badges to learning experiences to make them more engaging and to increase motivation. In this paper we explore how Khan Academy has incorporated gaming elements into its CS learning platform. By mapping the literature on motivational processes to popular games we critically analyze how successful Khan Academy is at gamifying their site.

via SIGCSE2014 – OpenConf Peer Review & Conference Management System.

Context matters when designing courses, too: Know Thy Learner

In 1994, Elliot Soloway, Ken Hay, and I wrote an article about “learner-centered design.”  We contrasted it with the prevailing paradigm of “user-centered design,” arguing that designing for learners is different than designing for experts (which, we suggested, is really what user-centered design is).

I like the below as pointing toward borrowing ideas from modern UX design for learning design.  The most important lesson that we try to teach undergraduates about human-computer interface design is, “Know Thy User, for the User is not You.”  You have to get to know your user, and they’re not like you.  You can’t use introspection to design interfaces.  That same lesson is what we’re hearing below, but about learning.  “Know Thy Learner, for the Learner is not You.”  Your learner has a different context than you, and you have to get to know it before you can design for it.

“Transferring education from the United States to Africa wouldn’t work,” argued Bakary Diallo, rector of African Virtual University. “Because we have our own realities,” he added, “our own context and culture.”

Naveed A. Malik, founding rector of the Virtual University of Pakistan, echoed that sentiment. “This is something that we learned very early in our virtual-university experience,” he said. “We couldn’t pick up a course from outside and then transplant it into a Pakistani landscape—the context was completely different.”

via Virtual Universities Abroad Say They Already Deliver ‘Massive’ Courses – Wired Campus – The Chronicle of Higher Education.

Benefits of Online, Face-to-Face Professional Development Similar, Study Finds

These are really exciting results.  Done well, on-line professional development is as effective as face-to-face professional development.  These results are promising for our CSLearning4U project. In particular, the benefit that Barry Fishman saw is what we were most hoping for, based on our studies with Klara Benda — it’s all about fitting into the teachers’ lives.

Of course, the devil is in how the teacher training is designed and executed. “There are no shortcuts in professional development,” Fishman stressed.

In the study, teachers who received the online professional development weren’t just plopped in front of YouTube. Instead, the group took a series of self-paced “short courses” via computer. They also interacted online with facilitators who helped them through the units and answered their questions.

Like their counterparts in the face-to-face group, the teachers were expected to become familiar with geographic information system software and how to teach it, as well as how to engage students in a hands-on, iterative learning process. Teachers in both groups had access to the same print materials and computer simulations.

Fishman and his colleagues found that teachers in the online group spent wildly varying amounts of time learning the new curriculum. One teacher cruised through the material in three hours. Another took 52 hours to digest everything. But the classroom results were largely the same.

“One of the benefits of online professional development is that it lets teachers move at their own pace,” Fishman said. “The same thing is probably going on in face-to-face [settings]. You just zone out when you’re sitting in a 40-hour workshop.”

via Benefits of Online, Face-to-Face Professional Development Similar, Study Finds – Digital Education – Education Week.

Online Learning Outcomes Equivalent to Traditional Methods: But what about the drops?

This is a great result, if I can believe it.  They took 605 students, some in a traditional course and some in a “hybrid” course, and did pre/post tests.  They found no difference in outcomes.

Here’s what I’m not sure about: What happened to those students who failed or who withdrew?  Other studies have suggested that online courses have higher withdraw/failure rates.  Is that the case here? There is only one footnote (page 18) that mentions withdraw/failure: “(27) Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of students who failed the course because the non-passing group includes students who never enrolled or withdrew from the course without receiving a grade.”  But that’s it.  If you lose more students in one format, and the students you lose are the weaker students (not an unreasonable assumption), then having the same learning gains doesn’t mean for all students.  It means that you’ve biased your sample.

The researchers asked the students to complete a number of tests and questionnaires before beginning the course and again after completing it, and they analyzed and compared the results between the two groups of students. The results revealed no statistical difference in educational outcomes between the two groups of students. In fact, the students in the hybrid course performed slightly better, but not enough to be statistically significant.

via Online Learning Outcomes Equivalent to Traditional Methods, Study Finds — Campus Technology.

Website that takes your online classes for you

I’m not sure that this is real — I tried to “get a quote” and couldn’t get the submit form to work right. And the “Read More” page is gobbledy gook. Even if satire, it raises a real point. There’s certainly a market in ‘We Take Your Online College Classes for You and Get You an “A”’

You are struggling with your online classes or homework and you want someone to do it for you. We can handle almost any subject and customer service is a priority. Our company culture revolves around making sure you feel safe and satisfied knowing that your work is being done by an expert within your specified deadline. We are here to serve you around the clock by email, live chat, and phone. For all of your academic needs, WeTakeYourClass wants to be the one you turn to time and time again.

via WETAKEYOURCLASS.COM- We Take Your Online Class! We Do Your Homework, Tests, Classes For You!.

What makes for a course that works as a MOOC?

Nice analysis from “Gas station without pumps” on what’s offered via Coursera and why the course offerings are what they are.  I particularly liked his wordplay on MOOC: Massively Over-hyped Online Course.  He suggests offering a Coursera course soon, because it’s unlikely to be around for long.

A lot of the courses that are offered are the “book learning” courses that require no lab facilities, no face-to-face discussions, and no close mentoring.  They are the easiest courses to offer, but the ones least likely to save universities much by switching to an online format (those sorts of lecture classes are already relatively cheap per student).

One exception is computer science classes, since the specialized equipment needed for CS courses is now so cheap that just about anyone who can access on-line courses has the necessary equipment already, and much of the software needed for CS courses is available free (often open-source).  If grading the courses is reduced to low-quality automatic checking of programs (a travesty that has already happened in some brick-and-mortar CS courses), then there is nothing stopping the scaling of fairly advanced courses to MOOCs.

via Coursera Course Catalog « Gas station without pumps.

Yay, College Board! 857 Desks Call Attention to Dropout Problem

How cool that the College Board is being active on this significant problem (that isn’t made better with online education)!  I do understand that increasing the pass rate without maintaining quality is an empty achievement, but the economic cost of the high dropout rate is enormous.

Each desk represents one of the 857 students who drop out of high school in the United States every single hour, every single school day, according to the College Board, which arranged the display to underline its effort to urge presidential candidates to put education at the top of their to-do lists.

The board had nearly a dozen people, iPads in hand, gathering signatures in nearly 100-degree weather for an online petition that said: “If you want my support, I need to hear more from you about how you plan to fix the problems with education. And not just the same old platitudes. I want to know that you have real, tangible solutions, and that once in office, you’re ready to take serious action. I’ll be watching your acceptance speech at your party’s convention.”

via 857 Desks Call Attention to Dropout Problem – NYTimes.com.