Teaching the On-Line Stanford class at UMass Lowell: Guest Post from Fred Martin

December 16, 2011 at 6:58 am 13 comments

160,000 Enroll Stanford’s Online AI Course—Is the University Obsolete?

@aiclass: “Amazing we can probably offer a Master’s degree of Stanford quality for FREE. HOW COOL IS THAT?”—September 23, 2011

Mark blogged about Stanford’s online Artificial Intelligence course in August. I’ve been leading a group of UMass Lowell students, who are following the course and will receive regular course credit under my supervision.

Sebastian Thrun and Peter Norvig’s online course, Introduction to Artificial Intelligence, was announced via email to a AAAI list early last summer. The idea went viral. Articles were published in the New York Times, the Chronicle of Higher Education, and many media outlets.

The course was advertised as equivalent to the Stanford University undergraduate AI course. As Thrun posted on Twitter:

@aiclass: “Advanced students will complete the same homeworks and exams as Stanford students. So the courses will be equal in rigor.”—September 28, 2011

The course launched the first week of October. 160,000 students had signed up.

At UMass Lowell

I had taught my department’s AI course in Fall 2010. Students started asking me if they could take the Stanford course as a directed study. In August, I decided to open two full course sections (grad and undergrad, meeting jointly). I wanted to experience this first-hand.

I told students they would be responsible for a final project on top of the Stanford requirements.

I ended up with 16 students—12 grad and 4 undergrad. We meet once weekly for 75-minutes, seminar-style, and talk about the Stanford material after each week’s assignment is already due.

How It Works

The Stanford course consists of weekly lectures—two or three 45-minute topics that are broken up into 15 or 20 short videos. Many of the individual videos have embedded questions (multiple-choice or fill-in-the-value). Thrun’s colleague at the Stanford AI Lab, Prof. Daphne Koller, is a pioneer of this approach, and discussed it in a recent NYT essay.

At the end of these mini-lessons, the video image transforms web form where you fill in the answers. You’ve already logged into the class server, and it grades you immediately. After you submit, you are presented with a link for an explanation video.

The lectures themselves are inspired by Khan Academy’s approach. Occasionally, Thrun and Norvig will train the camera at themselves, but the core content is with the camera on a piece of paper, with Thrun or Norvig talking and writing in real time. Some people don’t like this format, but I find it relaxing and engaging. I like seeing equations written out with verbal narration in perfect sync.

There are weekly problem set “homeworks” with the same format. The homeworks track the lecture material very closely. If you paid attention and did the problems embedded in the lectures, the problem sets are usually easy.

There has been a midterm with the same format as the homeworks. It had 15 questions. There will be a final, and I expect the same format.

The homeworks, midterm, and final have hard deadlines and are only viewable to pre-registered, logged-in students who chose the “Advanced” track. The lectures are openly viewable by anyone. The server backend keeps track your scores on the homeworks, midterm, and final, which will count to your “grade”—a ranking within the active student cohort. There is a “Basic” track which consists only of the lectures.

Of the 160,000 people who initially registered, it was reported in early December that 34,000 students from the Advanced cohort are still actively participating.

On the Course

Thrun and Norvig are great teachers. Thrun is always visibly excited about teaching the material. Norvig is not as effusive, but you still know he really cares about the ideas. They’ve thought through excellent ways of explaining the ideas and quizzing the in-lecture comprehension checks. They often bring fun props or show research projects in the videos recordings.

Thrun and Norvig are only a week or so ahead of the course delivery, and they’re paying close attention their students’ progress. There is a lot of activity on the web forums. Students are completing assignments.

Thrun and Norvig have recorded several “office hours,” where students submit questions and vote on their favorite ones, and then they pick questions and answer them on camera.

In this way, the course is like a usual class—it’s not “canned.” Thrun’s and Norvig’s enthusiasm is infectious.

This is the old-fashioned emotional connection between teachers and their students.

These things make it seem more like a conventional course than you would expect.

My Role as a Teacher

I don’t have to lecture the material. When we meet, my students have (largely) worked through the lectures and homeworks.

So I don’t have to explain things to students for the first time. Instead, we use in-class time to have an interesting conversation about the parts of the material that people found confusing or disagreed upon. We’ve had some great arguments this semester.

This is a lot like the approach suggested in 2006 by Day and Foley in their HCI course at Georgia Tech. They recorded web lectures, and then used classroom time for hands-on learning activities.

Koller calls this “the flipped classroom.” She reports higher-than-usual attendance in her Stanford courses that are taught this way: “We can focus precious classroom time on more interactive problem-solving activities that achieve deeper understanding—and foster creativity.”

One fun thing for me is that Thrun and Norvig are really in charge, and I get to dig into a role as a learning coach. If something is confusing, I get to really be on the side of my students in helping figure it out. I personally really enjoy this. Others might not.

Overall—this is an exciting experiment. Thrun and Norvig have created a fantastic set of interactive lectures and some good quiz problems. They’ve put in a lot of work, and it shows.

What Does It Mean?

I don’t think this threatens the university in any profound way.

Thrun and Norvig are renowned scientists, and charismatic and thoughtful teachers. But if in some future semester, they, or others comparable, are not participating in real time, I wonder how many the 34,000 remaining students would be carrying forward with the determination that they now are.

Furthermore, in order to learn how to “do AI”—and not just learn “about AI”—students need to do significant implementation and research projects. They need individual, time-intensive guidance from a faculty member who can encourage them and help them sharpen their focus.

In Koller’s essay, she extols the new, deeper value of in-class time.

So, the flipped classroom doesn’t put us out of business. It makes us more valuable—but only if we take advantage of the opportunity.

Entry filed under: Uncategorized. Tags: , , , .

Computer Professor Illuminates Concepts for Blind Students Ladies Learning Code seeks to close gender tech gap: How the teacher matters

13 Comments Add your own

  • 1. Stanford AI classes by simulacron3 - Pearltrees  |  December 16, 2011 at 2:29 pm

    […] Teaching the On-Line Stanford class at UMass Lowell: Guest Post from Fred Martin « Computing Educat… 160,000 Enroll Stanford’s Online AI Course—Is the University Obsolete? @aiclass: “Amazing we can probably offer a Master’s degree of Stanford quality for FREE. HOW COOL IS THAT?” […]

  • 2. PhilG  |  December 17, 2011 at 2:04 am

    As one of the students in both the AI and Machine Learning course, I can say that while the AI class provided a broad exposures to the material, I felt like the structure of the ML class and specifically the requirement to complete programming assignments made the material easier to absorb. I suspect ultimately, the AI class will do what was intended; ‘be an intro’ to the field, whereas the ML class and others offered next semester will provide more application in my projects.

    From a pedagogical point of view, I found Andrew Ng’s teaching style more detailed and comprehensive. The whiteboard style videos of the ML class worked better for me than the hand written ones in the AI class. Additionally, it’s easier to focus and absorb material when the videos are a bit longer. The ML videos were about 10 minutes each, could be downloaded, and came with PDFs of Dr. Ng’s notes written on the slides. The AI class used 1 to 3 minute YouTube videos that couldn’t be downloaded (easily) and often, due to their ‘live’ nature, contained material errors.

    Dr. Thrun and Dr. Norvig are without a doubt very well known in the field, famous even. While it was a privilege to take a class from them, they both suffer a bit from the inherent problem of teaching ‘intro’ material when you’re so far ahead of it. Concepts which to them seem obvious or self-explanatory were often not to those with no exposure to the field. I did well in the class, but based on the questions and comments on the discussion sites, it seemed to me that this was at the root of a large number of the problems. I think they’d both be better suited teaching an advanced class on a specific subject and leave the teaching of intro classe to more junior lecturers. But hey, they do this at their leisure, so I really don’t want to sound like I’m complaining.

    • 3. Fred Martin  |  December 18, 2011 at 10:39 am

      Phil — thanks for this comment.

      One of my frustrations with the AI course was that it was touted as equivalent to the Stanford one. But it turns out the Stanford course has required programming assignments! (They’re using the excellent UC Berkeley Pac-Mac material.)

      I am not sure if Thrun is just being a showman, or if he just doesn’t appreciate the significance of doing implementation projects in learning.

      But either way it’s insidious. His message is that his online class is equivalent to Stanford — so why would you pay to go to Stanford? (Or pay big bucks to a school with a lesser reputation?)

      There is a lot of price pressure on all forms of education, especially higher ed.

      It is decidedly not not helpful to have respected scientists/teachers arguing that on-line mass delivery is the same as what happens in a Stanford undergraduate course.

      (Maybe I should downgrade my model of what Stanford is doing in their own classes? I doubt it….)

  • 4. Kieran Mathieson  |  December 17, 2011 at 11:26 am

    Flipped is nice. It lets instructors focus on helping students build skills, rather than repeat information that’s available in other channels.

    But look at things from a broader perspective. Instead of trying different things (flipped, etc.) and seeing what works, why not start with what existing research has shown to work?

    Take the goal of building intro programming skills. The learning science literature suggests the following might help:

    * Deep learning. Students learn a few core ideas, then learn how to use the ideas to create things. Instead of more concepts, more practice with the core concepts.

    * Outcome-based learning. When designing a course, start with the skills you want students to have, then work backwards to decide on content. Be realistic about what average students can learn deeply – it takes lots of practice. Be disciplined – do not cover your favorite ideas, unless they are central to the outcome.

    * Frequent, formative feedback. Not “You get a B-.” Instead: “Here’s what you did wrong: …. But I liked ….” If you want mastery learning, add: “Change it, and resubmit.”

    * Metacognition. Do students know how to learn this material? Do they know what to do when they get stuck? Give them strategies. Make them practice those strategies.

    When I tried this a few years ago, I ended up not only with flipped, but with a whole lot more.

    For an in-the-trenches perspective, see:



  • 5. Seb Schmoller  |  December 17, 2011 at 2:11 pm

    Interesting and useful post.

    Two points:

    1. I no longer teach (and I never taught computer science), and I did the AI class out of interest in the subject matter and because I’ve been involved in online distance learning since 1992 i.e. pre Web. I would be very interested in your views on whether the course could be used as “enrichment” material for able K12 pupils. I would have thought it would be very good for this, and looking back on my own time as a “tinkering” teenager, I would have given my eye teeth for exposure to this kind of thing. (Not that this would have been possible in the mid 1960s….).

    2. There is quite a comprehensive comparison of the AI, Machine Learning and Databases courses in a Guest Contribution by Gundega Dekena on my blog, which may be of interest.

    Seb Schmoller

  • 6. Fred Martin  |  December 18, 2011 at 10:40 am

    Kieran — this sounds great!

    Can you post some pointers to your research articles based on this work?


  • 7. Bonnie  |  December 19, 2011 at 1:45 pm

    I really like the idea of deep learning – but find that the available textbooks utterly defeat that strategy. I am utterly baffled by the way that textbooks for introductory CS are organized. We teach in Java, having just switched from C++. The books for both languages make the same mistakes. Why are they covering pre and post increment operators in chapter 2, when most students haven’t yet figure out assignment??? Why, when introducing repetition, do the books have to cover every single loop format out there? Simply doing the while loop would be fine, and would allow us to explore the many ways repetition can be logically structured, while keeping the syntax to a minimum. This has nothing to do with Java vs Python vs C++, in my opinion – it has everything to do with textbook authors who think they have to cover every last shred of syntax in these massive tome-like textbooks. And yes, we can just ignore the parts of the book we aren’t using, but it does get confusing for the students.

  • 8. Bonnie  |  December 19, 2011 at 1:48 pm

    I should click the link before posting! I see your link addresses exactly what I am talking about. Thanks – this looks usefu

  • 9. reestheskin – Learning: you need more than videos  |  January 3, 2012 at 6:13 am

    […] cool model that I see in the Stanford online CS classes grew up around the Stanford classes.  Fred Martin at U. Massachusetts Lowell recently wrote a blog post  about how he used the Stanford classes to create a “flipped classroom.”  The students […]

  • […] not convinced.  First, when I read the comments to posts about the the Stanford classes, or Fred Martin’s post, I’m struck by how many people took the courses who already knew the content.  They were […]

  • […] companies have spun-off of the Stanford on-line CS classes, and Stanford has decided to partner with one of them.  Coursera wants to be a platform and let […]

  • […] Norvig (head of research at Google) who co-taught the 100K student on-line AI course was honest and pragmatic.  He started on this because he wanted to do more than a book.  He felt […]

  • 13. mgozaydin  |  December 17, 2012 at 5:18 am

    Fred MARTIN
    Great application .
    I claim that any student at any college should register for 1 or 2 courses from edx with the knıowledge of his advisor.
    The college should assign a facilitator for each online courses taken from edx . Facilitator should make the midterm and final exam as well.
    At the end the COLLEGE should award a credit toward a degree in their college . This way college saves 10 % per 1 online courses .
    2 online courses make 20 % cost savings + more room for new students . Imagine 5 online courses would make tuition halved .
    Since you have worked with AI course you can start this experiment . Please write to me as well .


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,185 other subscribers


Recent Posts

Blog Stats

  • 2,039,877 hits
December 2011

CS Teaching Tips

%d bloggers like this: