Archive for September, 2016

No, Really – Programming is Hard and CS Flipped Classrooms are Complicated: ITICSE 2016 Award-Winning Papers

I only recently started digging into the papers from the ITICSE 2016 conference (see Table of Contents link at ACM Digital Library here).  There were two papers that caught my attention.

First, the best paper award went to one of my former PhD students, Brian Dorn: An Empirical Analysis of Video Viewing Behaviors in Flipped CS1 Courses, by  Suzanne L. Dazo, Nicholas R. Stepanek, Robert Fulkerson, and Brian Dorn.  Brian has this cool piece of technology where students can view videos, annotate them, be challenged to answer questions from specific places, and have discussions.  They used this for teaching a flipped CS1 class, where students were required to watch videos before class and then engage in more active learning opportunities in class.  The real trick, as you might imagine and that the paper goes into detail on, is getting students to watch the video.  I liked both the techniques for prodding students to watch videos and the fascinating results showing the relationship between watching the videos and learning.

ITICSE 2016 recognized two “commended” papers this year.  I haven’t found the listing of which papers they were, but I did learn that one of them is Learning to Program is Easy by Andrew Luxton-Reilly.  I enjoyed reading the paper and recommend it — even though I disagree with his conclusions, captured in the paper title.  He does a good job of exploring the evidence that programming is hard (and even uses this blog as a foil, since I’ve claimed several times that programming is hard), and overall, is a terrific synthesis of a bunch of computing education papers (40 references is a lot for a six page ITICSE paper).

His argument that programming is easy has two parts:

  • First, children do it. As he says in the abstract, “But learning to program is easy — so easy that children can do it.”  That’s a false comparison — what children do in programming is not the same definition of “programming” that is in most of the literature that Andrew cites.  The evidence that programming is hard is coming mostly from higher-ed CS classes.  What is going on in introductory University CS classes and what children do is dramatically different.  We saw that in the WIPSCE 2014 Fields and Kafai paper, and those results were recently replicated in a recent ICER 2016 paper.  These are two different activities.
  • Second, what higher-education CS teachers expect at the end of the first course is too much.  He presents significant evidence that what CS teachers expect is achieved by students, but at the end of the second course.  The paper from Morrison, Decker, and Margulieux supports the argument that students think and work very differently and much more successfully by the end of the second CS course than in the first course.

I see Andrew’s argument as evidence that programming is hard.  The problem is that Andrew doesn’t define the target.  What level of ability counts as “programming”?  I believe that level of ability described by the McCracken Working Group, by the FCS1/SCS1 exams, and by most teachers as the outcomes from CS1 (these are all cited by Andrew’s paper) is the definition of the lowest level of “programming ability.”  That it takes two courses to reach that level of ability is what I would call hard.

I’ve been reading a terrific book, Proust and the Squid: The Story and Science of the Reading Brain by Maryanne Wolf.  It’s the story of how humans invented reading, how we teach reading, and how reading changes our brains (physically and in terms of cognitive function).  Oral language is easy.  We are literally wired for that.  Reading is hard.  We are not wired for that, and much of the invention of reading is about inventing how to teach reading.  Unless you can teach reading to a significant part of your population, you don’t develop a literate culture, and your written language doesn’t succeed.

Much of the invention of written language is about making it easier to learn and teach because learning to read is so hard.  Have you ever thought about why our Latin alphabet is ordered?  Why do we talk about the “ABC”‘s and sing a song about them?  We don’t actually need them to be ordered to read.  Ordering the alphabet makes it easier to memorize, and learning to read is a lot about memorization, about drill-and-practice to make the translation of symbols to sounds to words to concepts effortless (or at least, all System 1 in Kahneman terms).  This makes it easier, but the task of learning to read and write is still a cognitively complex task that takes a significant amount of time to master. It’s hard.

Programming is hard like written language is hard.  It’s not possible to program unless you know how to read.  Programming is particularly hard because the concepts that we’re mapping to are unfamiliar, are not part of our daily experience.  We only see it as easy because we have expert blind-spot.  We have already learned these concepts and made those mappings.  We have constructed understandings of iteration and conditional execution and variable storage.  It is difficult for experts to understand how hard it is to develop those concepts.  The evidence of children programming suggests that most children who program don’t have those concepts.

I remain unconvinced by Andrew’s argument, but I recommend the paper for a great summary of literature and an interesting read.


September 30, 2016 at 7:22 am 8 comments

Making learning effective, efficient, and engaging: An Interview With an Educational Realist and Grumpy Old Man, Paul Kirschner

I am a fan of Paul Kirschner‘s work.  This interview is great with useful insights about education — deep and pragmatic thinking.

I want to fundamentally understand how people can learn in effective, efficient, and enjoyable ways, and how you can teach and design learning materials to achieve this objective. If a learner doesn’t enjoy the learning experience, even if it’s effective and/or efficient, they won’t do it. The same is true for teaching: that is it must also be effective, efficient, and enjoyable for the teacher because if a teacher doesn’t enjoy the teaching process, even if it’s effective and/or efficient, they won’t do it.

Source: GUEST POST: An Interview With an Educational Realist and Grumpy Old Man — The Learning Scientists

September 28, 2016 at 7:59 am 1 comment

Assessing Learning In Introductory Computer Science: Dagstuhl Seminar Report now Available

I have written about this Dagstuhl Seminar (see earlier post). The formal report is now available.

This seminar discussed educational outcomes for first-year (university-level) computer science. We explored which outcomes were widely shared across both countries and individual universities, best practices for assessing outcomes, and research projects that would significantly advance assessment of learning in computer science. We considered both technical and professional outcomes (some narrow and some broad) as well as how to create assessments that focused on individual learners. Several concrete research projects took shape during the seminar and are being pursued by some participants.

Source: DROPS – Assessing Learning In Introductory Computer Science (Dagstuhl Seminar 16072)

September 26, 2016 at 7:26 am Leave a comment

Should Computing be in its own College or School?

Probably my favorite session from the CRA Snowbird conference this last summer (see agenda with links to all talks) was a session on creating Colleges or Schools of Computer Science.  Should we?  Why?

The most compelling two talks in the session were from Randy Bryant and Rich LeBlanc, because they were so similar in structure.  They both argued that you don’t make the argument for a high-level College or School of Computing because you’re big and important.  You make it because you have a driving definition of computing that makes it unique.

  • Randy told the story of how CMU’s School of Computer Science was driven by the original definition of computer science from Newell and Simon, and how that definition was broader than most people’s definition of CS today. I recently blogged on that definition.
  • Rich told the story of how Georgia Tech’s College of Computing was driven by the ACM report The Future of Computing (led by Peter Denning) which showed how Computing crossed science, mathematics, and engineering.  Of course, Rich’s story was particularly powerful for me because I lived that definition — that was the vision that drove the College of Computing when I first got here in 1993.  Rich told the story of how that definition convinced faculty and administrators at Georgia Tech that Computing couldn’t be contained within the Colleges of Engineering or Science.  It needed to be its own entity. (I may also be biased because Rich quoted me from this blog 🙂

Many of the people in the audience wanted to know, “How can I turn my Department into a School or College?”  One audience member said, “My CS department is the biggest one in the College of Engineering. How do I break apart into my own College.”  All the panelists told him, “You can’t.”  No Dean will allow its biggest department to leave — that would be crazy.  Some participants (from U. Michigan and U. Washington, in particular) pointed out why they don’t have a College or School of Computing — they have successful multi-department collaborations that make it unnecessary.  A new College or School is expensive.  Don’t do it unless you have to.


September 23, 2016 at 7:10 am 3 comments

Every University Student should Learn to Program: Guzdial Arguing for CS for All in Higher Education

A colleague recently approached me and said, “It would be useful if Universities got involved in this CS for All effort.  All Universities should offer courses aimed at everyone on campus. There should be a systematic effort to get everyone to take those classes.”

I agree, and have been making this argument for several years now.  I spent a few minutes gathering the papers, blog posts, and book where I’ve made that argument over the last decade and a bit.

In 2002, Elliot Soloway and I argued in CACM that we needed a new way to engage students in intro programming: Teaching the Nintendo Generation to Program.

In 2003, I published the first paper on Media Computation: A media computation course for non-majors.

In 2004, Andrea Forte led the team studying the Media Computation class at GT:Computers for communication, not calculation: Media as a motivation and context for learning and  A CS1 course designed to address interests of women.

In 2005, Andrea Forte and I presented empirical evidence about the courses we’d designed for specific audiences: Motivation and nonmajors in computer science: identifying discrete audiences for introductory courses. I published a paper in CACM about how the courses came to be at Georgia Tech: Teaching computing to everyone.

In 2008, I offered the historical argument for teaching everyone to program: Paving the Way for Computational Thinking.

We’ve published several papers about our design process: Imagineering inauthentic legitimate peripheral participation: an instructional design approach for motivating computing education and Design process for a non-majors computing course.

My 2013 ICER paper was a review of a decade’s worth of research on Media Computation: Exploring hypotheses about media computation

My keynote at VL/HCC 2015 was on how computing for all is a requirement for modern society: Requirements for a computing-literate society

My 2015 book is, to a great extent: an exploration of how to achieve CS for All: Learner-Centered Design of Computing Education: Research on Computing for Everyone.

In blog posts, it’s been a frequent topic of conversation:

I don’t know how to convince University CS departments to do just about anything, but here are my contributions to the dialogs that I hope are happening at Colleges and Universities worldwide about how to prepare students to engage in computational literacy.

September 19, 2016 at 7:15 am 17 comments

Learning Curves, Given vs Generated Subgoal Labels, Replicating a US study in India, and Frames vs Text: More ICER 2016 Trip Reports

My Blog@CACM post for this month is a trip report on ICER 2016. I recommend Amy Ko’s excellent ICER 2016 trip report for another take on the conference. You can also see the Twitter live feed with hashtag #ICER2016.

I write in the Blog@CACM post about three papers (and reference two others), but I could easily write reports on a dozen more. The findings were that interesting and that well done. I’m going to give four more mini-summaries here, where the results are more confusing or surprising than those I included in the CACM Blog post.

This year was the first time we had a neck-and-neck race for the attendee-selected award, the “John Henry” award. The runner-up was Learning Curve Analysis for Programming: Which Concepts do Students Struggle With? by Kelly Rivers, Erik Harpstead, and Ken Koedinger. Tutoring systems can be used to track errors on knowledge concepts over multiple practice problems. Tutoring systems developers can show these lovely decreasing error curves as students get more practice, which clearly demonstrate learning. Kelly wanted to see if she could do that with open editing of code, not in a tutoring system. She tried to use AST graphs as a sense of programming “concepts,” and measure errors in use of the various constructs. It didn’t work, as Kelly explains in her paper. It was a nice example of an interesting and promising idea that didn’t pan out, but with careful explanation for the next try.

I mentioned in this blog previously that Briana Morrison and Lauren Margulieux had a replication study (see paper here), written with Adrienne Decker using participants from Adrienne’s institution. I hadn’t read the paper when I wrote that first blog post, and I was amazed by their results. Recall that they had this unexpected result where changing contexts for subgoal labeling worked better (i.e., led to better performance) for students than keeping students in the same context. The weird contextual-transfer problems that they’d seen previously went away in the second (follow-on) CS class — see below snap from their slides. The weird result was replicated in the first class at this new institution, so we know it’s not just one strange student population, and now we know that it’s a novice problem. That’s fascinating, but still doesn’t really explain why. Even more interesting was that when the context transfer issues go away, students did better when they were given subgoal labels than when they generated them. That’s not what happens in other fields. Why is CS different? It’s such an interesting trail that they’re exploring!


Mike Hewner and Shitanshu Mishra replicated Mike’s dissertation study about how students choose CS as a major, but in Indian institutions rather than in US institutions: When Everyone Knows CS is the Best Major: Decisions about CS in an Indian context. The results that came out of the Grounded Theory analysis were quite different! Mike had found that US students use enjoyment as a proxy for ability — “If I like CS, I must be good at it, so I’ll major in that.” But Indian students already thought CS was the best major. The social pressures were completely different. So, Indian students chose CS — if they had no other plans. CS was the default behavior.

One of the more surprising results was from Thomas W. Price, Neil C.C. Brown, Dragan Lipovac, Tiffany Barnes, and Michael Kölling, Evaluation of a Frame-based Programming Editor. They asked a group of middle school students in a short laboratory study (not the most optimal choice, but an acceptable starting place) to program in Java or in Stride, the new frame-based language and editing environment from the BlueJ/Greenfoot team.  They found no statistically significant differences between the two different languages, in terms of number of objectives completed, student frustration/satisfaction, or amount of time spent on the tasks. Yes, Java students got more syntax errors, but it didn’t seem to have a significant impact on performance or satisfaction. I found that totally unexpected. This is a result that cries out for more exploration and explanation.

There’s a lot more I could say, from Colleen Lewis’s terrific ideas to reduce the impact of CS stereotypes to a promising new method of expert heuristic evaluation of cognitive load.  I recommend reviewing the papers while they’re still free to download.

September 16, 2016 at 7:07 am 4 comments

New Progress and Momentum in Support of President Obama’s Computer Science for All Initiative

Ruthe Farmer at the Office of Science and Technology Policy (OSTP) has been working furiously towards today’s announcements.  The Obama Administration is aiming to achieve the goal of CSforAll, and with only a few months left before the new Administration takes off, they’re showing what they’ve put in place today.  The full details on all the announcements are here. There’s a webcast at 1 pm EDT today here.  The biggest deal to me is the establishment of the CSforAll Consortium (see website here) which is meant to carry on the initiative, no matter who wins in November.

To mark this progress, and celebrate new commitments in support of the President’s initiative, the White House is hosting a summit on Computer Science for All. Key announcements being made today include:

  • More than $25 million in new grants awarded from the National Science Foundation (NSF) to expand CS education;

  • A new CSforAll Consortium of more than 180 organizations, which will connect stakeholders with curriculum and resources, as well as track progress towards the goal of Computer Science for All; and

  • New commitments from more than 200 organizations, ranging from expanded CS offerings within the Girl Scouts of the USA that could reach 1.4 million girls per year, to supporting professional development for 40,000 additional teachers, to new collaborations to bring CS to students in a variety of settings from African-American churches to family coding nights to tribal Head Start programs to students as Chief Science Officers.

Source: Fact Sheet: New Progress and Momentum in Support of President Obama’s Computer Science for All Initiative |

September 14, 2016 at 9:39 am 1 comment

Meeting the need for computing teachers in Australia

ICER 2016 was just held in Melbourne, Australia, so I found the article linked at the bottom (and from which these images come from) particularly relevant and interesting.

Australia is facing a boom in primary school students, which creates additional demand for teachers.  As has been mentioned here previously, there is a shortage of teachers.  The shortage isn’t distributed across fields.  In particular, over 30% of computing teachers in Australia are teaching without qualification (see image below).  When considering other shortages (e.g., declining number of computing teachers in Scotland, as described in the last post), it’s clear that the pipeline of CS teachers is going to be an impediment to CS for all.

But an influx of new students isn’t the only problem our school system needs to address.Shortages in specific subject areas mean that many students are being taught by teachers working outside of their qualifications.

Source: What will school education look like by 2020?

September 14, 2016 at 7:55 am 1 comment

Losing CS Teachers in Scotland: Latest report on CS teacher numbers from Computing At School Scotland

If you can forgive the bias in the graph (what looks like a 90% drop is actually a 25% drop), you will find this to be an important and interesting report. Scotland has one of the strongest computing at schools efforts in the world (see site here), with an advanced curriculum and a large and well-designed professional development effort (PLAN-C).  Why are they losing CS teachers?

When I wrote about this in 2014 (the trend has only continued), I pointed out that part of the problem is teachers refusing to shift from teaching Office applications to computer science.  The current report doesn’t give us much more insight into why. The point I found most interesting was that Scottish student numbers dropped 11%, and teacher numbers in the other disciplines are also declining (e.g., mathematics teachers are declining by 6% over the same period), but at a much slower rate than the CS decline of 25%. That makes sense too — if you’re a teacher and things are getting tough, stick with the “core” subjects, not the “new” one.  It’s worth asking, “How do we avoid this in the US?” and “Can we avoid it?”

We know too little about what happens to CS teachers in the US after professional development. I know of only one study of CS teacher retention in the US, and the observed attrition rate in that study was far worse than 25%. Do we know what US retention rate is for CS teachers? Maybe Scotland is actually doing better than the US?

Today we launch our latest report into the numbers of Computing Science teacher numbers across Scotland.  We have carried out this survey in 2012, 2014 and now 2016 as we are concerned about the decreasing number in Computing teachers in Scottish schools.  Nationally we now have 17% of schools with no computing specialist and a quarter of Secondary schools have only one CS teacher.

Source: Latest report on CS teacher numbers | Computing At School Scotland

September 12, 2016 at 7:26 am 2 comments

Barriers to Stack Overflow Use for Females

Stack Overflow is an often used resource by programmers today. It’s also a barrier to women entering computing. Here’s a blog post summarizing a recent study on why women find Stack Overflow so unwelcoming.

There are many movements to get women into programming, but what about keeping them there? If they don’t feel comfortable using the resources that are available for all programmers then that is a big problem for retention in the field. To do our part in being more proactive in welcoming women into the field, we sought to uncover some reasons for this low participation.

Source: Paradise Unplugged: Barriers to Stack Overflow Use for Females | fordable

September 12, 2016 at 7:08 am 11 comments

Learning CS while Learning English: Scaffolding ESL CS Learners – Thesis from Yogendra Pal

When I visited Mumbai for LaTICE 2016, I mentioned meeting Yogendra Pal. I was asked to be a reader for his thesis, which I found fascinating. I’m pleased to report that he has now graduated and his thesis, A Framework for Scaffolding to Teach Vernacular Medium Learners, is available here:

I learned a lot from Yogendra’s thesis, like what “vernacular medium learners” means. Here’s the problem that he’s facing (and that Yogendra faced as a student). Students go through primary and secondary school learning in one language (Hindi, in Yogendra’s personal case and in his thesis), and then come to University to study Computer Science. Do you teach them (what Yogendra calls “Medium of Instruction” or MoI) in English, or in Hindi? Note that English is pervasive in Computer Science, e.g., almost all our programming languages use English keywords.

Here’s Yogendra’s bottomline finding: “We find that self-paced video-based environment is more suitable for vernacular medium students than a classroom environment if English-only MoI are used.” Yogendra uses a design-based research methodology. He measures the students, tries something based on his current hypothesis, then measures them again. He compares what he thought would happen to what he saw, and revises his hypothesis — and then iterate. Some of the scaffolds he tested may seem obvious (like using a slower pace), but a strength of the thesis is that he develops rationale for each of his changes and tests them. Eventually, he came to this surprising (to me) and interesting result: It’s better to teach with Hindi in the classroom, and in English when students are learning from self-paced videos.

The stories at the beginning of the thesis are insightful and moving. I hadn’t realized what a handicap it is to be learning English in a class that uses English. It’s obvious that the learners would be struggling with the language. What I hadn’t realized was how hard it is to raise your hand and ask questions. Maybe you have a question just because you don’t know the language. Maybe you’ll expose yourself to ridicule because you’ll post the question wrong.

Yogendra describes solutions that the Hindi-speaking students tried, and where the solutions didn’t work. The Hindi-speaking students used English-to-English dictionaries. They didn’t want English-Hindi dictionaries, because they wanted to become fluent in English, but they needed help with the complicated (especially technical) words. They tried using online videos for additional explanations of concepts, but most of those were made by American or British speakers. When you’re still learning English, switching from an Indian accent to another accent is a barrier to understanding.

The middle chapters are a detailed description of Yogendra’s attempts to scaffold student learning. He tried to teach in all-Hindi but some English technical terms like “execute” don’t have a direct translation in Hindi. He selected other Hindi words to represent the technical terms, but the words he selected as the Hindi translation were unusual and not well-known to the students. Perhaps the most compelling insight for me in these chapters was how important it was to both the students and the teachers that the students learn English — even when the Hindi materials were measurably better for learning in some conditions.

In the end, he found that Hindi language screencasts led to better learning (statistically significantly) when the learners (who had received primary and secondary school instruction in Hindi) were in a classroom, but that the English language screencasts led to better learning (again, statistically significantly) when the learners were watching the screencasts self-paced. When the students are self-paced, they can rewind and re-watch things that are confusing, so it’s okay to struggle with the English. In the classroom, the lecture just goes on by. It works best if it’s in Hindi for the students who learned in Hindi in school.

Yogendra tells a convincing story. It’s an interesting question of how these lessons transfer to other contexts. For example, what are the issues for Spanish-speaking students learning CS in the United States? In a general form, can we use the lessons from this thesis to make CS learning accessible to more ESL (English as a Second Language) learners?

September 8, 2016 at 5:50 pm 6 comments

Interview with Sue Sentance on Computing Education Research: CAS TV YouTube Channel

I didn’t realize that Computing at School has their own YouTube channel:

This episode is particularly relevant for this blog — Sue Sentence talking about computing education research:

Some of the articles mentioned by Sue (from Miles Berry on the CAS site):

September 7, 2016 at 7:15 am Leave a comment

Survey for Human-Computer Interaction (HCI) Instructors

From Lauren Wilcox:

Betsy DiSalvo, Dick Henneman and I have designed a survey about a topic that is near and dear to us as HCI faculty:  topics, learning goals, and learning activities in HCI classrooms!

We hope to do an annual “pulse” of HCI instructors across the globe.

The survey takes about 15 minutes. We plan to share the results with the broader academic HCI community.

We are hoping that you can take the survey, and also please share with your colleagues who teach HCI-related classes.

September 5, 2016 at 7:55 am 4 comments

Preview ICER 2016: Ebooks Design-Based Research and Replications in Assessment and Cognitive Load Studies

The International Computing Education Research (ICER) Conference 2016 is September 8-12 in Melbourne, Australia (see website here). There were 102 papers submitted, and 26 papers accepted for a 25% acceptance rate. Georgia Tech computing education researchers are justifiably proud — we submitted three papers to ICER 2016, and we had three acceptances. We’re over 10% of all papers at ICER 2016.

One of the papers extends the ebook work that I’ve reported on here (see here where we made them available and our paper on usability and usage from WiPSCE 2015). Identifying Design Principles for CS Teacher Ebooks through Design-Based Research (click on the title to get to the ACM DL page) by Barbara Ericson, Kantwon Rogers, Miranda Parker, Briana Morrison, and I use a Design-Based Research perspective on our ebooks work. We describe our theory for the ebooks, then describe the iterations of what we designed, what happened when we deployed (data-driven), and how we then re-designed.

Two of our papers are replication studies — so grateful to the ICER reviewers and communities for seeing the value of replication studies. The first is Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment by Miranda Parker, me, and Shelly Engleman. This is Miranda’s paper expanding on her SIGCSE 2016 poster introducing the SCS1 validated and language-independent measure of CS1 knowledge. The paper does a great survey of validated measures of learning, explains her process, and then presents what one can and can’t claim with a validated instrument.

The second is Learning Loops: A Replication Study Illuminates Impact of HS Courses by Briana Morrison, Adrienne Decker, and Lauren Margulieux. Briana and Lauren have both now left Georgia Tech, but they were still here when they did this paper, so we’re claiming them. Readers of this blog may recall Briana and Lauren’s confusing results from SIGCSE 2016 result that suggest that cognitive load in CS textual programming is so high that it blows away our experimental instructional treatments. Was that an aberration? With Adrienne Decker’s help (and student participants), they replicated the study. I’ll give away the bottom line: It wasn’t an aberration. One new finding is that students who did not have high school CS classes caught up with those who did in the experiment, with respect to understanding loops

We’re sending three of our Human-Centered Computing PhD students to the ICER 2016 Doctoral Consortium. These folks will be in the DC on Sept 8, and will present posters to the conference on Sept 9 afternoon.

September 2, 2016 at 7:53 am 17 comments

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,005 other followers


Recent Posts

Blog Stats

  • 1,880,400 hits
September 2016

CS Teaching Tips