How students use an electronic book

Our technical report on evaluation of Luther College students’ use of the first generation Runestone Interactive Python ebook is finally available: http://hdl.handle.net/1853/45044. This was a paper that we wrote for ICER 2012 but was rejected. The reviewers’ general argument is that we’re just describing one class using an ebook: No comparisons to other uses of books or ebooks, no particular hypothesis being tested. That is a fair criticism, but the problem is that we don’t know of a comparable study. We don’t know of anyone who has studied how CS students use their normal textbooks and IDE’s, for example, so that we can contrast it with the ebook use.

Here’s an example of one of our findings, which we found surprising. You might recall that the Runestone Interactive ebook has three special kinds of features: Embedded videos, “ActiveCode” segments (where students can actually program in Python from within the pages of the book), and “CodeLens” interactive visualizations of code, which can be run forward or backward. Below is a histogram of the number of ActiveCode events on each day of the first five weeks of class. The red bar is the day of the midterm. Blue bars are days when students had class (and most use on that day was in class), and gray bars are use out-of-class.

So here’s the first surprise: Use in-class swamps use out-of-class. The Luther college students are using the book in-class, and are working on programming activities (directed by the teacher) in-class. Those of us at Georgia Tech expected students to be programming far more out-of-class, maybe two or three times as much as in-class. Not so here. Is that unusual behavior only found at Luther? We don’t know yet.

Here’s the second surprise: Notice that the use on the day before the midterm is not one of the larger spikes. If you had an ebook to help you learn CS, with lots of examples that you could poke with, wouldn’t you study by using those? We are not seeing much of that, and students reported studying by “reading” it — few of them mentioned exploring code.

We at Georgia Tech built quizzes for each of the first five chapters of the book, completely separate from Brad Miller and David Ranum, the teachers at Luther. We also got the students’ midterm scores. So of those three features, which one most closely correlated with better performance outcomes on the quiz and midterms? The visualization tool. Here’s a scatterplot of the midterm score and use of the CodeLens by student.

 

Now, what did the students think was most valuable for their learning? Lectures, by quite a bit. We asked them to rank the various learning affordances in the class, and to score each in value from 1 to 5 (lower scores are better).

 

There are surprises here, too. Videos are not a big win here. Students do value being able to run code in the ebook (and told us about how much they liked that in our surveys), but don’t value it as much for their learning as lecture. Students don’t rank CodeLens very highly at all, but it was the feature that had the greatest measurable effect.

We have a whole bunch more data now: From several classes, and from a group of high school teachers that we can compare to the undergraduates. Christine Alvarado has ended her sabbatical at Georgia Tech, but is still working with us on this analysis. Brad Miller is still graciously allowing us to pester him with questions, and is giving us log data in identity-scrubbed form so that we can dig into it without compromising student identities. I hope that we can produce another paper for a peer-review forum, this time, with comparisons across multiple data sets so that we can start to figure out what is normal or typical use of a CS ebook.