Posts tagged ‘ebooks’

An Ebook Integrating Minimal Manuals with Constructionism, Worked Examples, and Inquiry: MOHQ

Our computing education research group at Georgia Tech has been developing and evaluating ebooks for several years (see this post with discussion of some of them). We publish on them frequently, with a new paper just accepted to ICER 2016 in Melbourne. We use the Runestone Interactive platform which allows us to create ebooks with a lot of different kinds of learning activities — not just editing and running code (which I’ve been arguing for awhile is really important to support a range of abilities and motivations), but including editing and running code.

It’s a heavyweight platform. I have been thinking about alternative models of ebooks — maybe closer to e-pamphlets. Since I was working with GP (see previous post) and undergraduate David Tran was interested in working with me on a GP project, we built a prototype of a minimalist medium for learning CS. I call it a MOHQ: Minimal manual Organized around Hypertext Questions: (Suggestion: Use Firefox if you can for playing with browser GP. WAY faster for the JavaScript execution than either Chrome or Safari on my Mac.)

Minimal Manuals

John Carroll came up with the idea of minimal manuals back in the 1980’s (see the earliest paper I found on the idea). The goal is to help people to use complicated computing devices with the minimum of overhead. Each page of the manual starts with a task — something that a user would want to do. The goal is to put the instruction for how to achieve that task all on that one page.

The idea of minimalist instruction is described here:

The four principles of minimal instruction design are:

  1. Allow learners to start immediately on meaningful tasks.
  2. Minimize the amount of reading and other passive forms of training by allowing users to fill in the gaps themselves
  3. Include error recognition and recovery activities in the instruction
  4. Make all learning activities self-contained and independent of sequence.

There’s good evidence that minimal manuals really do work (see Learners become more productive more quickly with minimal manuals, with surprisingly high scores on transfer and retention. A nice attribute of minimal manuals is that they’re geared toward success. They likely increase self-efficacy, a significant problem in CS education.

The goal of most minimal instruction is to be able to do something. What about learning conceptual knowledge?

Adding Learning Theory: Inquiry, Worked Examples, and Constructionism

I started exploring minimal manuals as a model for designing CS educational media after a challenge from Alan Kay. Alan asked me to think about how we would teach people to be autodidacts. One of the approaches used to encourage autodidactism is inquiry-based learning. Could we structure a minimal manual around questions that they might have or that we want students to ask themselves?

We structure our Runestone ebooks around an Examples+Practice framework. We provide a worked example (typically executable code, but sometimes a program visualization), and then ask (practice) questions about that example. We provide one or two practice exercises for every example. Based on Lauren Margeliux’s work, the point of the practice is to get students to think about the example, to engage with it, and to explain it to themselves. It’s less important that they do the questions — I want the students to read the questions and think about them, and Lauren’s work suggests that even the feedback may not be all that important.

Finally, one of the aspects that I like about Runestone is that every example in an active code area is a complete Python interpreter. Modify the code anyway you want. Erase all of it and build something new if you want. It’s constructionist. We want students to construct with the examples and go beyond them.

MOHQ: Minimal Manual Organized around Hypertext Questions

The prototype MOHQ that David Tran and I built ( is an implementation of this integration of minimal manuals with constructionism, inquiry, and worked examples. Each page in the MOHQ:

  • Starts with a question that a student might be wondering about.
  • Offers a worked example in a video.
  • Offer the opportunity to construct with the example project.
  • Asks one or two practice questions, to prompt thinking about the project.

Using the minimal design principles to structure the explanation:

  1. Allow learners to start immediately on meaningful tasks.

The top page offers several questions that I hope are interesting to a student. Every page offers a project that aims to answer that question. GP is a good choice here because it’s blocks-based (low cognitive load) and I can do MediaComp in it (which is what I wanted to teach in this prototype).

#1: Minimize the amount of reading and other passive forms of training by allowing users to fill in the gaps themselves.

Each page has a video of David or me solving the problem in GP. Immediately afterward is a link to jump directly into the GP project exactly where the video ended. Undo something, redo something, start over and build something else. The point is to watch a video (where we try to explain what we’re doing, but we’re certainly not filling in all the gaps), then figure out how it works on your own.

Then we offer a couple of practice questions to challenge the learner: Did you really understand what was going on here?

#2: Include error recognition and recovery activities in the instruction.

Error recovery is easy when everything is in the browser — just hit the back button. You can’t save. You can’t damage anything. (We tell people this explicitly on every page.)

#3: Make all learning activities self-contained and independent of sequence.

This is the tough one. I want people to actually learn something in a MOHQ, that pixels have red, green, and blue components, and chromakey is about replacing one color with a background image, and that removing every other sample increases the frequency of a sound — and more general ideas, e.g., that elements in a collection can be referenced by index number.

So, all the driving questions from the home page start with, “Okay, you can just dive in here, but you might want to first go check out these other pages.” You don’t have to, but if you want to understand better what’s going on here, you might want to start with simpler questions.

We also want students to go on — to ask themselves new questions, to go try other projects. After each project, we offer some new questions that we hope that students might ask themselves. The links are explicitly prompts. “You might be thinking about these questions. Even if you weren’t, you might want to. Let’s see where we can explore next.”

Current Prototype and What Comes Next

Here’s the map of pages that we have out there right now. We built it in a Wiki which facilitated creating the network of pages that we want. This isn’t a linear book.


There’s maybe a dozen pages out there, but even with that relatively small size, it took most of a semester to pull these together. Producing the videos and building these pages by hand (even in a Wiki) was a lot of work. The tough part was every time we changed our minds about something — and had to go back through all of the previously built pages and update them. Since this is a prototype (i.e., we didn’t know what we wanted when we started), that happened quite often. If we were going to add more to the GP MOHQ, I’d want to use a tool for generating pages from a database as we did with STABLE, the Smalltalk Apprenticeship-Based Learning Environment.

I would appreciate your thoughts about MOHQ. Call this an expert review of the idea.

  • Thumbs-up or down? Worth developing further, or a bad direction?
  • What do you think is promising about this idea?
  • What would we need to change to make it more effective for student learning?

June 15, 2016 at 7:35 am 7 comments

Preview for WiPSCE 2015: Usability and usage of interactive features in an online ebook for CS teachers

Next week in London, at WiPSCE 2015, the 10th Workshop in Primary and Secondary Computing Education, Barbara Ericson is going to present the pilot study of our teacher CSP ebook that Steven Moore (undergrad researcher here at Georgia Tech) ran last year.  This is the pilot that came before our Spring trial (see post here) and which led to our student and teacher ebooks that we recently released (see post here).  The authors of the paper are Barbara, Steven, Brianna Morrison, and me.

Steven’s study had two parts to it.  The first was a usability survey comparing three different ebook platforms: Runestone Interactive, Zyante, and CS Circles.  You may recall the post here where I invited you to participate in that survey.  Runestone did well in the survey, just beating out Zyante, and both were far ahead of CS Circles.

The meat of the paper is the study of 10 teachers who qualified for our study (got less than 40% on a pretest) and read the 8 chapters we had ready for them.  Every two chapters, there was a post-test on those two chapters.  Some of the findings:

  • 50% of the study participants finished all 8 chapters.  That’s pretty good, but isn’t directly comparable to MOOC studies because we did offer them a $50 gift card for completing.
  • As we expected and designed for, teachers read the books in chunks of time when they could fit it in.
  • Those who used the book (e.g., did the Parson’s problems, ran the code, etc.), gained confidence in teaching CS and performed well on the post-tests. This is a big deal!  The teachers are not just writing code.  They are using a variety of different kinds of learning activities (see our ICER ebook paper) — and successfully learning and gaining confidence.  Writing code is not the only way to learn CS.  This has been one of the more controversial hypotheses. Many CS teachers believe that apprenticeship is the only way to learn CS, but we believe that we can successfully use a range of pedagogical practices.

Barbara did an extensive log file analysis of the participants, and we learned a lot from that. We learned where our books were not usable, e.g., when participants skipped over interactive features, or when they used the features wrong (e.g., clicking “Check Me” on a Parson’s problem, without ever moving pieces around). We used these findings in designing the current ebooks.

This paper is exciting for us — it’s the first where we test our design claims.


November 6, 2015 at 8:20 am 1 comment

Seeking teacher input on Dashboard for ebooks

Matt Moldavan is an MS HCI student here at Georgia Tech who is developing a new Instructor Dashboard for the Runestone Interactive and our group’s CSP eBooks (see announcement here). The goal for the dashboard is to offer useful reports, graphs, and analytics accessible by instructors. He aims to provide instructors with useful insights into their students’ activity and progress through their online course(s).

He’s conducting a survey to find out what teachers want to know when overseeing student activity in online learning. We would like to know your previous experiences with similar student progress tracking tools (such as Cengage, Moodle, Desire2Learn, and others). Additionally, we are seeking feedback on several early design prototypes of the dashboard.

Matt has a short survey at If you have taught an online course or have used any other student progress tracking tools, your input would be greatly appreciated.

For more information, contact Matt at


October 9, 2015 at 8:02 am Leave a comment

Student and Teacher CSP Ebooks are now Available

We now have TWO ebooks supporting CS Principles (see website here) now available — one for teachers and one for students.

Our teacher ebook summer study is now ended. (Announcement about launching the study is here.) We’re crunching the data now. We’ve already learned a lot about what teachers want in an ebook. We learned where our user interface wasn’t obvious, and where we needed to explain more. We learned that teachers expect end-of-chapter exercises. We have used what we have learned so far to produce the two new ebooks.

STUDENT CSP EBOOK: About a year ago, we received additional NSF funding (from the Improving Undergraduate STEM Education (IUSE) program) to develop a student version of our CSP ebook. We have been running participatory design studies and gathering usability surveys from students to get input on what a student ebook should look like. We have now released the first version of the student ebook.

The student CSP ebook is available at  It doesn’t require a login, but we recommend that teachers have their students login. Without a login, we store saved answers on the local computer, but if the student logs in, we save the answers by the student’s username.  The course name is StudentCSP.

We recommend that teachers create a custom version of the student ebook for your students.  This allows teachers to customize the ebook, assign homework, and view student’s progress, and even create additional assessments for students.

New Version TEACHER CSP EBOOK: We iterated on our teacher ebook at the same time that we were developing the student ebook. We hypothesize that the student CSP ebook may actually encourage teachers to complete the teacher ebook. We can imagine that teachers who use the student ebook might want to stay one step ahead of the students, e.g., “My students are starting Chapter 3 on Monday, so I better finish Chapter 3 this weekend.”

We have now created a second version of our teacher CSP ebook. This one is in lockstep with the student CSP ebook, includes all the end-of-chapter exercise answers and teacher notes (e.g., on how to teach particular concepts, common student difficulties, etc.). We are not making the second teacher ebook available openly (because it includes answers to the student problems).

Teachers, please contact us at with the name and location of your school, and we’ll send you the URL.

We recommend that teachers create their own course for their students.  See for why a teacher might want to build a custom course and how to do it.

  • You must register on Runestone first at Enter StudentCSP as the course name. Be sure to record your username. We find that users often forget what they entered and assume it was their e-mail address — and it may not have been. You can also choose to sign in with your account on Google Plus, Facebook, Twitter, or several others.
  • Then go to and select “Create your own Course”.
  • Create a unique name for your course (use your school name and StudentCSP and year maybe), add a description, and your institution, and then select “CS Principles: Big Ideas in Programming by Mark Guzdial, Barbara Ericson, and Briana Morrison“.
  • Leave the rest as defaults and click the “Submit” button.  This will build a custom version of the student ebook for your students and it will have a unique URL and course name.  You will be listed as the instructor and can look at the log files and view other information on the instructor page (you can get to this by clicking on the icon that looks like a head and shoulders and the top right of your screen when you are in the ebook).

September 25, 2015 at 8:00 am 8 comments

ICER 2015 Preview: First CSLearning4U Ebook Paper

ICER 2015 (see website here) is August 9-13 in Omaha, Nebraska. The event starts for me and Barbara Ericson, Miranda Parker, and Briana Morrison on Saturday August 8. They’re all in the Doctoral Consortium, and I’m one of the co-chairs this year. (No, I’m not a discussant for any of my students.) The DC kickoff dinner is on Saturday, and the DC is on Sunday. My thanks to my co-chair Anthony Robins and to our discussants Tiffany Barnes, Steve Cooper, Beth Simon, Ben Shapiro, and Aman Yadav. A huge thanks to the SIGCSE Board who fund the DC each year.

We’ve got two papers in ICER this year, and I’ll preview each of them in separate blog posts. The papers are already available in the ACM digital library (see listing here), and I’ll put them on my Guzdial Papers page as soon as the Authorizer updates with them.

I’m very excited that the first CSLearning4U project paper is being presented by Barbara on Tuesday. (See our website here, the initial blog post when I announced the project here, and the announcement that the ebook is now available). Her paper, “Analysis of Interactive Features Designed to Enhance Learning in an Ebook,” presents the educational psychology principles on memory and learning that we’re building on, describes features of the ebooks that we’re building, and presents the first empirical description of how the Runestone ebooks that we’re studying (some that we built, some that others have built) are being used.

My favorite figure in the paper is this one:


This lists all the interactive practice elements of one chapter of a Runestone ebook along the horizontal axis (in the order in which they appear in the book left-to-right), and the number of users who used that element vertically. The drop-off from left-to-right is the classic non-completion rate that we see in MOOCs and other online education. Notice the light blue bars labelled “AC-E”? That’s editing code (in executable Active Code elements). Notice all the taller bars around those light blue bars? That’s everything else. What we see here is that fewer and fewer learners edit code, while we still see learners doing other kinds of learning practice, like Parsons Problems and multiple choice problems. Variety works to keep more users engaged for longer.

A big chunk of the paper is a detailed analysis of learners using Parsons Problems. Barbara did observational studies and log file analyses to gauge how difficult the Parsons problems were.  The teachers solved them in one or two tries, but they had more programming experience.  The undergraduate and high schools students had more difficulty — some took over 100 tries to solve a problem. Her analysis supports her argument that we need adaptive Parsons Problems, which is a challenge that she’s planning on tackling next.

August 5, 2015 at 7:35 am 3 comments

SIGCSE 2015 Week! ECEP BOF and Ebooks and IRB and other CS Ed terms

This week is the SIGCSE 2015 Technical Symposium, the largest computing education conference in the US, perhaps in the world.  About 1300 people will be heading to Kansas City for four days of discussion, workshops, and talks.  See the conference page here and the program here.

Barbara Ericson and I will be presenting at several events:

  • I’m speaking on a panel Thursday afternoon at 3:45 on human-subjects review of experiment protocols (by Institutional Review Boards (IRB)) and the challenges we’ve had in working in high schools and working on cross-institutional projects.
  • Barb and I will be hosting with Rick Adrion a Birds of a Feather (BOF) session on state-level change at 6:30 Thursday.  This is part of our ECEP work.
  • On Friday morning at 10 am, we’ll be showing our electronic book (ebook) for high school teachers interested in learning CS Principles.  The first public showing was at the NSF BPC Community meeting in January, but that was to a small audience.  We’ll be presenting at the NSF Showcase at 10 am on the exhibition floor.
  • Barb is speaking on Friday afternoon in a panel at 3:30 on activities for K-12 CS outreach.
  • On Friday night, Barb is running her famous “How to run a computing summer camp workshop.”

As usual, Georgia Tech is sending several of us (not just Barb and me).  One of my PhD students, Briana Morrison, is on a panel on Flipped Classrooms Thursday 1:45-3 pm in 3501G.  Another of my PhD students, Miranda Parker, is part of a BOF Preparing Undergraduates to Make the Most of Attending CS Conferences 6:30-7:20 on Thursday evening.  Our colleague, Betsy DiSalvo, is speaking Friday morning 10:45-12 on a panel Research, Resources and Communities: Informal Ed as a Partner in Computer Science Education in 2505A.

This is one of my shorter stays at a SIGCSE conference.  I’ll be coming in late Wednesday and leaving Friday afternoon.  I’ve been traveling way too much lately (NSF BPC Community meeting in January, talk at Penn in early February, Tapia conference in Boston two weeks ago, AP CS Principles review meeting in Chicago this last week).  I am fortunate to be teaching Media Computation this semester, and I hate to miss so many lectures.  More, it’s hard on our family when we’re both gone.  Barb will be at SIGCSE all week, from Tuesday night to Sunday morning, so be sure to stop by and say hello to her.

March 3, 2015 at 9:23 am Leave a comment

Help us figure out how to design ebooks to be usable

Like the post I made last week, we’ve been working on a bunch of experiment setups during the summer, and are now looking for participants.  This one is open to most readers of this blog. 

We have found that there is a lot of literature on how to design text to be readable on the screen.  But for interactive ebooks with embedded elements like coding areas, visualizations, and Parson’s problems, we know less about usability.  Steven Moore is an undergraduate researcher working with us, and he’s put together a collection of three different ebooks and a survey on preferences for each.  We’d love to get participants to try out his ebook samples and survey, please.


We are a research group at Georgia Tech developing new approaches to teaching computer science at a distance.  In collaboration with researchers at Luther College, we have created a new kind of electronic book for learning Python. The book is entirely web-based and cross-platform, with special features, including programming within the book, program visualizations, videos, multiple-choice questions, and Parson’s problems (a special kind of programming problem).

We are currently seeking individuals with 6 months or more experience with programming in a textual language.  If you are willing to volunteer, you will need to complete a survey regarding the design and usability of three different interactive computer science e-books and specific components within those e-books.  Links to the e-books will be provided within the survey and the whole study can be completed via most web browsers.  The survey should take roughly forty-five minutes to complete.  We would like you to complete it by September 30th, 2014.

The risks involved are no greater than those involved in daily activities.  You will receive a $15.00 gift card for completing the survey.  Study records will be kept confidential and your participation in this study is greatly valued.

If you are interested in participating, please contact Steven Moore at  If you have any questions, please feel free to contact me, Mark Guzdial at

August 25, 2014 at 8:53 am 6 comments

Older Posts

Recent Posts

February 2017
« Jan    


Blog Stats

  • 1,314,704 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 4,625 other followers

CS Teaching Tips