What does it mean to assess Computational Thinking?

January 13, 2016 at 8:12 am 15 comments

One of the arguments I develop in my book on learner-centered design of computing education is that computational thinking, using Jeannette Wing’s description, is implausible.  There’s part of her description that talks about computing providing a medium for advancing thinking and learning in other domains — that’s the application part of computing, and that’s quite plausible.  I call that part computational literacy because that’s the name Andrea diSessa gave to that idea years ago. Much of my book is about how to help students (of all kinds, from graphic designers to teachers to undergraduates) develop computational literacy.  Then there’s the part of Jeannette’s description that suggests that learning computing will impact everyday thinking and problem-solving, e.g., people will use ideas about caching when packing for a trip.  There is no evidence to support the belief that that will happen. Many studies investigating this kind of impact have not found that effect. (I’ve reported in the past how educational psychologists find computational thinking implausible.)  Sure, there other definitions of computational thinking (I reference the others in my book), but they all have this same thread — computational thinking is about thinking that helps outside of computing.

So what does it mean to assess computational thinking?  Most computational thinking assessments I’ve seen fail to connect the computing to some other discipline.  For both of Wing’s sets of goals, we need to show that students are learning computing.  That’s a necessary part — if you don’t know computing, you can’t apply that knowledge and you can’t transfer it.  But it’s not sufficient.  Students must be applying, connecting, or transferring the computing knowledge to other domains to be computational thinking.

SRI is developing a set of computational thinking assessments.  From poking through their website, I’m not finding any examples, so I don’t know if they succeed where others have not.  Their process is promising.

As part of the NSF-funded Principled Assessment of Computational Thinking (PACT) suite of projects, SRI Education has been working with curriculum authors and teachers, assessment experts, and computer scientists to develop assessments for ECS.

ECS emphasizes inquiry-based teaching to develop students’ problem solving skills, as well as their abilities to explain, elaborate, and evaluate what they are learning, often using multiple representations of particular solutions. These skills go well beyond recalling facts or giving inputs to a program and predicting its outputs. As a result, the SRI PACT team had to design and develop assessment tasks that elicited students’ problem solving and inquiry skills in authentic contexts and gave them opportunities to represent their skills in their own words and ways.

Applying a principled design method, the team first developed generalized design templates for computational thinking practices. These practices refer to how students design and implement creative solutions and artifacts, how they design and apply abstractions and models, and how they analyze their computational work and the work of others (among other practices). We then used these templates to guide the development of assessment tasks and scoring rubrics aligned with the skills related to the learning goals of the ECS curriculum.

Source: Broadening Student Participation in Secondary Computer Science Through Principled Assessment of Computational Thinking (PACT) | SRI International

Entry filed under: Uncategorized. Tags: , .

Developing a Framework to Define K-12 CS Ed: It’s about consensus not vision The President Wants Every Student To Learn Computer Science. How Would That Work?

15 Comments Add your own

  • 1. nickfalkner  |  January 13, 2016 at 8:31 pm

    We have talked about this before but I had never quite understood some of the subtleties in the point that was being made. This really clearly explained the issues and, now, I agree with the point.

    Thank you!

  • 2. embee  |  January 14, 2016 at 5:49 am

    Thanks for your clear differentiation between computational thinking (CT) and computational literacy (CL). That’s crucial, because it may help us to clarify what CS ed is aiming for. I want to teach kids both CL (because their work and life will be full of computers in whatever disguise) and CT (because creative, iterative problem-solving skills are more necessary than ever in a world where most repetitive, predefined tasks can be done by machines). I see computational thinking happening everywhere and needed even more often, so when you say that it is implausible, I’m a little confused.

    Maybe you can clarify first: Do you mean to say that CT does not exist or that we do not teach it by teaching CS?

    Probably my definition of CT does not exactly coincide with Wing’s, so let me clarify: Many of the tasks we face in life are computational in that they involve developing ad-hoc or carefully planned algorithms, involve decomposition in to subproblems, reduction to other known tasks (and their solutions), analyzing errors and adapting the processes developed (i.e. “debugging” in Papert’s wider sense), etc. They often do not involve computers (though, by their very nature, they lend themselves well to being “computerized”), but they require thinking algorithmically. You do not mean to say that this kind of thinking does not exist, do you?

    So if what you mean is my second interpretation, that CS ed does not teach CT, you are probably right. However, I’d argue this is the case mainly because CS education aims for CL rather than CT in the first place. If CT is only a serendipitous side effect, not a major goal, it’s no wonder many students won’t experience that side effect.

    It’s very similar in Math (at least here in Germany): Thinking about problem solving (e.g. explicitly learning about and applying Polya’s principles) is usually just not done (cf. Lockhart’s Lament), because the focus in on “learning the tools of the trade”. Now, I believe that the need for *thinking* about problem solving skills is even greater in CS than in Math, because in CS you need to “teach” them to a dumb machine. So CS ed could be an ideal subject to teach kids CT. But it will only happen if 1) we make it the (or at least one) goal we’re aiming for and 2) design curricula accordingly.

    And, yes, then assessing whether it really works, is crucial. So I’m looking forward to seeing the SRI templates, assessment tasks and, of course, results.

    • 3. Mark Guzdial  |  January 14, 2016 at 9:25 am

      We have no evidence that that kind of thinking exists. We have no evidence that teaching students Polya’s problem solving strategies will lead to students spontaneously recognizing and apply those strategies is their daily lives. We have no evidence that students recognize algorithms or debugging opportunities in their daily lives. Some students will — high ability students. We don’t know how to teach for that. “Serendipitous side effects” didn’t happen with Latin or geometry, and they don’t happen with programming either.

      • 4. gasstationwithoutpumps  |  January 14, 2016 at 11:42 am

        You are making too strong a claim here, Mark!

        I think that there is plenty of evidence that debugging and algorithmic thinking exist—many of us can give anecdotes from our own experience. The decomposition of problems into subproblems with clear interfaces (sometimes called “systems thinking”) is ubiquitous in engineering fields—some say it defines the engineering approach to problem-solving.

        What hasn’t been shown is that computational thinking is teachable.

        By dismissing those who learn to think algorithmically as “high-ability students”, you seem to be falling into the fixed-mindset way of thinking about students—that their achievements are based on their ability, and not on how they are taught. This seems to be contradictory to your usual attitudes about the value and relevance of teaching.

        • 5. Mark Guzdial  |  January 14, 2016 at 12:44 pm

          I do agree that we all have experiences where we sense that we’re applying knowledge from computing (especially knowledge of debugging and about algorithms) to our daily lives. That’s different than someone watching over us and noting, “They could have applied X here — do they? They could have applied Y here — do they?” We might be fooling ourselves. It might happen very rarely, and we only notice when we do recognize a similarity. In the research on situated learning, researchers found that people compartmentalize their learning. Shoppers who couldn’t do any mathematics story problems actually used quite sophisticated mathematical reasoning in the grocery — but they may have figured it out in the grocery, and there was no sign of transfer between mathematics classes and the shops.

          Here’s the thought experiment that I offer my students: Think about the very best programmers that you know. Do they generally have the best lives, the ones devoid of problems (or problems that they easily solve)? Are they the most organized and planful people? Can you make any generalization about the daily lives of the very best programmers you know?

          I agree that systems thinking probably exists, though I don’t have any evidence to support that belief. Systems thinking explicitly identifies “systems” and teaches generalizing notions of feedback and influences. I bet that students who study systems thinking can apply those principles to a wide variety of systems. Still, I wonder if students trained in systems thinking recognize systems in everyday life, like (say) politics or traffic flows. I don’t know.

          I say “high-ability” because a constant in the literature on transfer is that students who know more (whether taught more, or experienced more, whatever — they do better on tests) are able to transfer more. Super smart people recognize connections and are more likely to transfer. If transfer happens, it happened because students developed a great deal of ability in that field.

          I can’t say that computational thinking is impossible — you can’t use scientific experiments to prove that something can’t happen. All I can say is that I know of no empirical evidence showing that we can teach it (as you say) nor documenting spontaneous application of computational problem-solving techniques in daily life. And, our theories of learning, problem-solving, and transfer do not support the possibility of teaching general computational thinking, as defined by Wing.

      • 6. embee  |  January 14, 2016 at 12:12 pm

        You say “We have no evidence”, but that is different from having evidence of the contrary. I’m asking in all seriousness: Could you point me to some negative results, please? I’m well aware that most evidence of computational thinking is anecdotal and I would love to have hard data, even if it does not support my personal experience. That’s why the SRI work may be important.

        As for the high-ability students: If their high ability is shown by their being able to transfer their skills to other domains, this is a circular argument. And, as gasstationwithoutpumps points out, you deny the possibility that this could be achieved as a result of teaching.

        You are right: Computational thinking won’t appear by serendipity (but that’s what I was saying, too). But there’s so much general generally useful thinking involved in CS (modeling, problem decomposition, planning for contingencies, analyzing data, reasoning, debugging, etc) that, as Simon Peyton-Jones says, “there is good reason to believe that while every subject claims to teach students to think, computer science really does”. Of course, Simon might be wrong as well. So empirical data is needed.

        Btw, the claim my teachers made for Latin was quite a different one: that it could help us when learning any other language later. That claim turned out to be true, for myself at least. How Latin would help me learn to think, I don’t know. How a CS course centered around Papert’s “thinking about thinking” would, I could at least argue for.

        • 7. Mark Guzdial  |  January 14, 2016 at 12:28 pm

          I spend a good bit of time in my book going over the evidence. I recommend David Palumbo’s meta-review of the literature relating programming and problem-solving, as well as the work by Roy Pea, Midian Kurland, and Jan Hawkins. The book on transfer of cognitive skill by Singley and Anderson is probably the most definitive piece on the subject. From Jim Greeno through Lave and Wenger, the most common thread in situated learning research simply acknowledges that transfer of problem-solving knowledge doesn’t happen. At all.

        • 8. Mark Guzdial  |  January 14, 2016 at 12:50 pm

          Here’s a relatively short introduction to the ideas of situated learning: http://www.umsl.edu/~wilmarthp/modla-links-2011/Situated-Cognition.pdf

          I also recommend the Wikipedia page on situated learning which states that:

          Action is grounded in the concrete situations in which it occurs.
          Knowledge does not transfer between tasks.
          Training by abstraction is of little use.

  • 9. David Weintrop  |  January 14, 2016 at 11:32 am

    I agree that diSessa’s notion of Computational Literacy is a clearer and more operationalizable construct than computational thinking, but for better or worse, computational thinking has gained more traction, so is the term were stuck with.

    Towards your comments about assessing CT as it relates to other disciplines, I wanted to point you towards a project at Northwestern University focused on bringing computational thinking into high school math and science classrooms (http://ct-stem.northwestern.edu). As part of the project, we’ve defined what CT means in math and science contexts, developed curricula that blend the two, and, to the topic of the post, have created a set of CT in Math and Science assessments. We’ve published our CT in Math and Science taxonomy (http://link.springer.com/article/10.1007%2Fs10956-015-9581-5) and also have shared lessons and assessments on the project website. We’re in the process of analyzing results that will hopefully address the concerns you raise about a lack of evidence that CT can help students in other disciplines. Preliminary findings are encouraging, so stay tuned.

    • 10. Mark Guzdial  |  January 14, 2016 at 11:36 am

      Thanks, David! Skimming the paper (only the first few papers), it looks like your definition of CT is more like diSessa’s CL than Wing’s CT. In other words, you’re not claiming broad transfer of problem-solving skills from computing to daily lives. You’re focusing on teaching and assessing computing as applied in mathematics and science contexts. Do I have that right?

      • 11. David Weintrop  |  January 14, 2016 at 11:43 am

        Yes, an explicit goal of the project is to ground CT and make it concrete. There are a number of reasons for this both theoretical and practical (we talk about this in the paper), but one reason of relevance here is the fact that in-service math and science teachers will need to teach CT as part of the new NGSS standards. By narrowing the scope of CT from everything down to math and science contexts, our hope is to be able to make CT actionable and useful to teachers (and researchers) as well as to sharpen the construct so it is more theoretically sound.

  • 12. Bonnie  |  January 15, 2016 at 5:33 pm

    “Still, I wonder if students trained in systems thinking recognize systems in everyday life, like (say) politics or traffic flows. I don’t know.”

    You must not be married to an engineer like I am! Everything reduces to a system to be specified and analyzed, in his mind. In truth, I find I do the same thing, as do all our engineering friends. I think that is why we engineers often have trouble relating to every day people.

    Yes, being trained in systems thinking does radically change your approach to everyday life.

  • 13. OTR Links 01/19/2016 | doug --- off the record  |  January 19, 2016 at 12:31 am

    […] What does it mean to assess Computational Thinking? | Computing Education Blog […]

  • […] (“computational thinking”) were unlikely to transfer to everyday situations (see post here).  We see a similar pattern in the recent controversy about “brain training.”  Yes, […]

  • […] see evidence that we can teach computational thinking the way it’s been described (as I discussed here).  For example, is it true that “Computational thinking can also help in understanding and […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

Recent Posts

January 2016
« Dec   Feb »


Blog Stats

  • 1,267,783 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 4,562 other followers

CS Teaching Tips

%d bloggers like this: