Archive for April 5, 2019

Opportunities to explore research questions with Guest post from Baker Franke

I wrote my Blog@CACM column this month as a response to the 60 Minutes segment on with Hadi Partovi, “Five Research Questions Raised by a Pre-Mortem on the 60 Minutes Segment on” Baker Franke, research and evaluation manager at (see Felienne’s blog post about Baker here) responded on Facebook with an invitation to engage with on these (and other) research questions.  He provided me with a longer form of his Facebook post, which I’m sharing here with his permission. Thanks, Baker!

Hi Mark,
These are all of course great questions and ones we really hope the community takes up. Some are things we’re actively tracking or working on with other research partners. So, some questions are actually more at the mid-mortem stage 🙂 or at least we can see trends. If you’d like me to share something I’m more than happy to.
But for anyone out there who would like to dig in a little bit more please consider becoming a research partner. I recently gave a flash talk at the SPLICE workshop at SIGCSE ‘19 about how we do research partnerships. Here is a working document about Research Partnerships at It explains with a bit more detail how we work with researchers and the kinds of data we do and don’t have. If you have ideas for killer research projects that make use of our platform let us (read: me) know.

To offer a little something, I’ll take the bait on Mark’s first hypothesis: “Students with accounts don’t actually use the resources.” Answer: depends how you count! Actually it’s more like: depends where you want to draw the line on a full spectrum of activity we can log. We do try to track the difference between just visiting the site, logging in, doing an activity, starting a course with a teacher, and doing more. (Of course nothing would replace a well-designed implementation study, did I mention our research partnership program?)

For example, in CS Fundamentals (K5), we’ve been looking at students who “start” the course (defined in next paragraph) v. those who reach a basic level of “coding proficiency,” but understanding and defining this is tricky and is an ongoing project (click this link to learn more about how we define it)Mark_Guzdial_Guest_Blog_Post_-_Google_DocsWe also track what we call “course started” v. “fully enrolled”. “Course started” is defined as a student user who (1) completes any single activity within one of our full courses (i.e. CSF, CSD, CSP, but not hour of code) and (2) who is in a teacher-section with 5 or more other students. Some exploratory analysis has shown that >= 5 students filters out a lot of noise that is probably not school-related activity. “Fully enrolled” is a metric that we’re also continuing to refine, and is a bit of misnomer because we can’t actually see what course the student is enrolled in at their school, but it means something like: “system activity data is indicative of this person probably actually going through the course in a classroom with a teacher.” We calculate it based on a student account (1) being in a section of >= 5 students (2) reaching a threshold of activity within each unit of the course — different thresholds for each unit depending on size, amount of unplugged, etc. — (3) over all units in the course. I think the important thing to highlight about these metrics is that they’re focused on activity indicative of students in classrooms, not just any old activity. Here are the numbers for the most recently completed school year.


Right now, our definition for “fully enrolled” is admittedly rough, and we’re going to refine it based on what we’re seeing in actual classrooms and publish a definition later this year with better numbers, but these numbers give a high level idea of what we’re seeing.

So do students with accounts use the resources? Yeah, a lot do. It’s tempting to compare these numbers to, say, MOOC completion rates, but we’re not really a MOOC because our curricula aren’t actually online courses. It would be hard for a student to go through our courses without a teacher and a classroom of schoolmates. The difference between “started” and “fully enrolled” could be a teacher only using one unit of a course or a few activities. So where’s the line? Another way to slice this would be to look at started v. completed for students of teachers who went through our PD program. Wouldn’t that be interesting? Want to find out more? Did I mention our Research Partnership Program?


April 5, 2019 at 7:00 am Leave a comment

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 6,196 other followers


Recent Posts

Blog Stats

  • 1,637,513 hits
April 2019
« Mar    

CS Teaching Tips