How students use an electronic book

November 8, 2012 at 7:34 am 12 comments

Our technical report on evaluation of Luther College students’ use of the first generation Runestone Interactive Python ebook is finally available: This was a paper that we wrote for ICER 2012 but was rejected. The reviewers’ general argument is that we’re just describing one class using an ebook: No comparisons to other uses of books or ebooks, no particular hypothesis being tested. That is a fair criticism, but the problem is that we don’t know of a comparable study. We don’t know of anyone who has studied how CS students use their normal textbooks and IDE’s, for example, so that we can contrast it with the ebook use.

Here’s an example of one of our findings, which we found surprising. You might recall that the Runestone Interactive ebook has three special kinds of features: Embedded videos, “ActiveCode” segments (where students can actually program in Python from within the pages of the book), and “CodeLens” interactive visualizations of code, which can be run forward or backward. Below is a histogram of the number of ActiveCode events on each day of the first five weeks of class. The red bar is the day of the midterm. Blue bars are days when students had class (and most use on that day was in class), and gray bars are use out-of-class.

So here’s the first surprise: Use in-class swamps use out-of-class. The Luther college students are using the book in-class, and are working on programming activities (directed by the teacher) in-class. Those of us at Georgia Tech expected students to be programming far more out-of-class, maybe two or three times as much as in-class. Not so here. Is that unusual behavior only found at Luther? We don’t know yet.

Here’s the second surprise: Notice that the use on the day before the midterm is not one of the larger spikes. If you had an ebook to help you learn CS, with lots of examples that you could poke with, wouldn’t you study by using those? We are not seeing much of that, and students reported studying by “reading” it — few of them mentioned exploring code.

We at Georgia Tech built quizzes for each of the first five chapters of the book, completely separate from Brad Miller and David Ranum, the teachers at Luther. We also got the students’ midterm scores. So of those three features, which one most closely correlated with better performance outcomes on the quiz and midterms? The visualization tool. Here’s a scatterplot of the midterm score and use of the CodeLens by student.


Now, what did the students think was most valuable for their learning? Lectures, by quite a bit. We asked them to rank the various learning affordances in the class, and to score each in value from 1 to 5 (lower scores are better).


There are surprises here, too. Videos are not a big win here. Students do value being able to run code in the ebook (and told us about how much they liked that in our surveys), but don’t value it as much for their learning as lecture. Students don’t rank CodeLens very highly at all, but it was the feature that had the greatest measurable effect.

We have a whole bunch more data now: From several classes, and from a group of high school teachers that we can compare to the undergraduates. Christine Alvarado has ended her sabbatical at Georgia Tech, but is still working with us on this analysis. Brad Miller is still graciously allowing us to pester him with questions, and is giving us log data in identity-scrubbed form so that we can dig into it without compromising student identities. I hope that we can produce another paper for a peer-review forum, this time, with comparisons across multiple data sets so that we can start to figure out what is normal or typical use of a CS ebook.



Entry filed under: Uncategorized. Tags: , , , .

Please pledge for CSEdWeek 2012 — whenever your event occurs “Florida is killing Computer Science”

12 Comments Add your own

  • 1. Bonnie  |  November 8, 2012 at 9:22 am

    This does not surprise me at all. We are using traditional books, but put lots of sample code on our LMS – all of the examples from class plus more. I model how to experiment with code in class, and also via the use of guided labs. Since my LMS lets me track what is downloaded, I know for a fact that my students rarely download the sample code. They also only view videos if they have to answer questions from the video for a grade. Finally, my students do not study by reading the book. Most study merely by reading the Powerpoint slides. Those are the most frequently downloaded item from my LMS site.

    So, I do not think the students that were followed in the paper were an outlier.

  • 2. gflint  |  November 8, 2012 at 10:56 am

    It is so much easier to veg out in a lecture that to have to read the book and interpret what it is says. The “most useful resource” results should be no surprise. I have been teaching high school programming for 30 years. My biggest pain is getting the kids to open the book, be it paper or electronic. When I myself am learning a new language I would much rather have someone show me than work in a book that cannot answer questions. So many of the texts are just stone cold boring.

    • 3. Bonnie  |  November 8, 2012 at 12:23 pm

      Interesting, because I would take the book any day over a person showing me. People tend to not be as organized as books, and they also tend to “show” rather than explain. I really need organized explanations. Worst of all, for me, are videos.

  • 4. profgarrett  |  November 8, 2012 at 12:22 pm

    Strange, I would have assumed that they’d like the code samples more than anything. I wonder if you’ve looked into Cognitive Load theory, as the idea of worked examples should be relevant to new students. Basically, the idea is that novices need more guidance, but that experts can do without. I wonder if the level of computer aptitude or preexisting experience had an effect on their use of the examples.

    In any case, thanks for sharing. This is very interesting stuff.

  • 7. guy  |  November 9, 2012 at 11:14 am

    Off the top of my head, what all say above matches my experience with teaching programming for a decade at BFOIT’s summer camp. We have had a Logo IDE embedded in the on-line lessons for over a decade. I added rudimentary “watch the code be performed” visualizations in 2004 for three early lessons. What I’ve seen is that these visualizations are rarely watched other than when I brought them up on the projector in class. The summer camp takes place in a CS lab on the UC Berkeley campus. The high-school aged students preferred for me to lecture, a brief ~10 minute overview of a piece of a lesson, and then they wanted to work on the program(s) that were in the lesson.

    I do know of schools that have used the lessons and only used the embedded IDE for writing programs. Since it is a Java applet, they can not copy/paste code out of or into it or save their work – a major limitation. But from what I can tell, this is only when the students are younger or are only working on a few of the lessons, i.e., the first few or the one covering recursion.

    I now have to read your entire paper. One thing you have not mentioned is whether there was any scoring facility in the book. For a long time, I’ve thought that giving students a score that they can improve would boost interest in the on-line lesson contest. It could provide better feedback to students and be a bit more fun for them, more like an on-line game/puzzle/challenge…

  • 8. OTR Links 11/10/2012 « doug – off the record  |  November 10, 2012 at 12:31 am

    […] How students use an electronic book « Computing Education Blog […]

  • 9. George Veletsianos  |  November 10, 2012 at 4:31 pm

    In response to this: “The reviewers’ general argument is that we’re just describing one class using an ebook: No comparisons to other uses of books or ebooks, no particular hypothesis being tested”

    That’s unfortunate. It sounds like the reviewers that were assigned to your paper have a methodology that they value and prefer, and are unable to understand that different research approaches yield different results. In this case in particular, it’s clear that a case study approach that seeks to gain an in-depth understanding of this particular ebook, in this particular context, in this particular school is an appropriate methodology. My suggestion would be to hedge against this type of review by educating reviewers, within the paper, about the value of the appropriate methodology chosen. I do that in my papers that use methodological approaches that are not the norm (e.g., phenomenology, ethnography, etc).

    • 10. Mark Guzdial  |  November 11, 2012 at 9:05 pm

      SIGCSE conference papers are very short. I’m not going to do much “educating” in such a small space, and still tell our story. It’s okay. We’re swimming in data. It’s going to be easy to iterate with more data, so that we can really say what’s common and what’s unusual between different student groups.

  • 11. Eugene Wallingford  |  November 12, 2012 at 4:25 pm

    I’m finding this semester that students with laptops are often spending class time working on their programming assignment and listening to the actual class “in the background”. They rely on other students to ask questions and answer mine. When this replaces time spent programming out of class, which it does for most of them, the results are predictable come exam time.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,005 other followers


Recent Posts

Blog Stats

  • 1,878,933 hits
November 2012

CS Teaching Tips

%d bloggers like this: