Any cognitive benefit of video games? Video-game studies have serious flaws

Do video games provide some kind of cognitive benefit after the game play?  There have been arguments that video games lead to improved attention, quicker responses, and visual skills.  A paper in Frontiers in Psychology has reviewed the past literature and found that they are all flawed with some basic bias errors.  This doesn’t mean that video games don’t have cognitive benefits.  But we don’t have any evidence that they do.

Most of the studies compare the cognitive performances of expert gamers with those of non-gamers, and suffer from well-known pitfalls of experimental design. The studies are not blinded: participants know that they have been recruited because they have gaming expertise, which can influence their performance, because they are motivated to do well and prove themselves. And the researchers know which participants are in which group, so they can have preconceptions that might inadvertently affect participants’ performance.

via Video-game studies have serious flaws : Nature News.

Heading to International Computing Education Research 2011 in Rhode Island: How CS students choose Threads

I’m heading out Sunday for the 2011 International Computing Education Research (ICER) Workshop, hosted by Dr. Kate Sanders at Rhode Island College in Providence.  The schedule is exciting — we have a bunch of speakers from communities who have been doing CS Ed research, but have not been at ICER previously.  (“Workshop” is ACM’s name for a small conference.)  I’m chairing the discussion papers session.  I’m looking forward to Eric Mazur’s keynote (who has a new educational technology that he’s promoting), and his advice from the Physics Education Research community to the much-younger Computing Education Research community.

The second talk of the conference is from my PhD student, Mike Hewner (same student who previously studied what game developers look for in graduates).  Mike’s dissertation research is asking, “How do computer science undergraduates define ’computer science,’ and how does their definition influence their educational decisions?” He’s using grounded theory, which is a demanding social science method.  He’s done about a dozen interviews so-far, and has not yet reached “saturation” (where new interviews don’t contribute to the developing theory), so the current theory is still considered “tentative.”  This paper is one piece of that work.

In most CS degree programs, there are some options for students: Choices between electives, between specialization paths, between Threads.  Mike wanted to know how students made those choices.  Several findings surprised me.  First, students don’t ”begin with the end in mind.”  Students he interviewed had little idea what job they wanted, and if they did, they didn’t really know what the job entailed.  Second, students don’t think that the choice of specialization is all that important — they figure that they’re at a good school, they trust the faculty, so whatever choice they make will turn out fine.  Finally, an engaging, fun class can dramatically influence students’ perception of a field.  A “fun” theory class can convince students that they like theory.  Their opinion of the subject is easily swayed by the qualities of the class and the teacher. “Why are you in robotics (even though it doesn’t have much to do with what you say you want to do for your job)?” “Well, I really liked the robots we used in CS101…”

Hope to see some of you there!

Not “Gamification” — it’s “Exploitationware”

Ian’s call to re-brand “gamification” as “exploitationware” is getting a lot of attention. It was covered in the Wall Street Journal’s blog yesterday. It’s certainly true that the term “gamification” is getting traction, e.g., I was just on an NSF panel where reviewers praised proposals trying to “gamify” educational software.  Ian points out that the language matters.  Consider the different connotations between “global warming” and “climate change,” where both terms are describing the same phenomena but from different political perspectives.  Most of the comments on Ian’s blog seem to be saying, “Give up! It’s too late.”  But I agree with Ian’s strategy. It is possible to change language, by calling attention to it and offering a significant alternative.

Note how deftly Zicherman makes his readers believe that points, badges, levels, leader boards, and rewards are “key game mechanics.” This is wrong, of course — key game mechanics are the operational parts of games that produce an experience of interest, enlightenment, terror, fascination, hope, or any number of other sensations. Points and levels and the like are mere gestures that provide structure and measure progress within such a system.

But as Frank Luntz has shown time and time again, reality matters far less than perception. When people hear “gamification,” it’s this incredible facility that registers, the simplicity, smoothness, and ease with which the wild, magical beast of games can be tamed and integrated into any other context at low cost and high scale.

Margaret Robertson has critiqued gamification on the basis that it takes the least essential aspects of games and presents them as the most essential. Robertson coins the derogatory term pointsification as a more accurate description of this process.

via Gamasutra – Features – Persuasive Games: Exploitationware.

Students more interested in having fun than doing good

There’s a challenging and interesting paper being presented this afternoon at SIGCSE 2011 Exploring the Appeal of Socially Relevant Computing: Are Students Interested in Socially Relevant Problems? by Cyndi Rader, Doug Hakkarinen, Barbara Moskal, and Keith Hellman from the Colorado School of Mines. I’ve worked with Barbara Moskal before, and know her to be a careful and thoughtful evaluator. So, when I read their abstract, especially the bottom line, I was surprised and intrigued.

Prior research indicates that today‘s students, especially women, are attracted to careers in
which they recognize the direct benefit of the field for serving societal needs. Traditional
college level computer science courses rarely illustrate the potential benefits of computer
science to the broader community. This paper describes a curricula development effort
designed to embed humanitarian projects into undergraduate computer science courses. The impact of this program was measured through student self-report instruments. Through this investigation, it was found that students preferred projects that they perceived as “fun” over the projects that were social in nature.

As I expected, the paper is careful and insightful. The authors did create some new socially relevant assignments to put into CS1 and Software Engineering assignments, and they asked students about their experience doing those. They also collected a wide variety of assignment descriptions for students to rank in terms of how interesting the assignment was: “A coding of ’1′ reflected a rating of ‘I definitely would not like to do this project’ and a coding of ’4′ reflected a rating of ‘I definitely would like to do this project.’ In other words, a higher rating reflected greater interest in the given project.”

• The authors found that students preferred the projects building games to those focused on social good. They also found a distinction that another researcher (Buckley et al., SIGCSE 2008) had identified — that students were more motivated by social and personally meaningful: “In other words, students may need to recognize the application of the solution to a problem to their own life.”
• While the Software Engineering assignments worked well, the CS1-level socially-relevant assignments did not — in part, because they were just so hard.  ”Our efforts were successful in Software Engineering, with 88% and 93% responding positively to the SAR and DM projects, respectively. However, only 54% of the studentsin the CS1 course, including 47% of the females, indicated that they found the SAR project appealing.” The authors conclude that, “This [the lack of interest in the socially-relevant projects in CS1] may, in part, be due to the fact that it was difficult to reduce socially relevant problems to a level that beginning students could easily comprehend. This made it difficult to capitalize on the appeal of socially relevant problems in the early computer science courses.”

I’m looking forward to seeing this paper presented this afternoon. There’s a certain cynical similarity to this paper, and work we’ve reported on about teachers. Davide Fossati’s paper on Saturday describes that faculty he interviewed changed their teaching practice for their own reasons, never because of student learning results, and Lijun Ni’s work last year showing that teachers adopt a new approach because they find something fun, not because it’s been shown to be effective.  I wonder if we’d see similar results outside the United States?

Can focus on Video Games and Visual Effects enhance STEM education efficiency?

Is this last thought true, that opportunities in video games are growing?  Last I heard, we already have an over-supply of video game programmers.  Each programmer is actually pretty productive, so a relatively small number of programmers is all that the relatively small number of major game studios really need.  Is that not the case?

An increasing number of schools and teachers now recognize that games can be used to improve mathematics, physics and computer science outcomes in the classroom itself.

Moreover, awareness of opportunities in these industries and the requisite skills will add a modern and exciting flavor to the study of these subjects, normally considered dry and boring, and thus attract more students towards them. These disciplines would then be viewed as leading to creative careers rather than technical ones alone.

Thus, the report suggests  ”We need to set in motion a virtuous circle where video games and visual effects help draw young people into maths, physics and computer science, and improve their learning outcomes, in turn enlarging the talent pool for these industries in the future. Schools should do more to encourage cross-curricular learning. Career guidance needs to reflect the growing employment opportunities in high-tech creative industries like video games and visual effects.”

via Can focus on Video Games and Visual Effects enhance STEM education efficiency?.

The Male-Centric Culture of Gaming

Change the verb “game” to “program,” and “gamer” to “hacker” in the quote below, and I think that this could almost be a transcript from Margolis and Fisher’s Unlocking the Clubhouse. Recall that Margolis and Fisher found that many of the factors that drove away women from CS at CMU were cultural and social, e.g. the male-dominated geek culture, and the bravado of showing-off knowledge in classroom questions.  Maybe it’s for the same reason that so few females take game design and programming classes?  Maybe it’s not about the technical content, or even about games, but about game culture?  As MMPORGs become increasingly dominant, the social aspect of games may become the most visible, especially to women.  It that culture is not welcoming to women, that would be a disincentive to take more classes in the field.

Who I am talking to are the guys in between, and there’s a whole swath of them. They’re the guys who claim they have no problem with “girls who game” but seem to have a problem with “girl gamers.” They’re the ones who probably wouldn’t seem to have an issue with women in their everyday lives but if one shows up on the game server, all rules of normal social decorum go out the window.

I’ve said it before and I’ll say it again: stop assuming that women who game are trying to be this Girl Gamer you keep getting hung up on. There is no such thing.

First of all, when I ask guys like you what you mean by “trying to be a girl gamer,” the definitions are ambiguous and sketchy. “They talk a lot and act all cute.” “They’re too chatty, they just want attention.” “They…you know, act like girls.”

via The Glorious Grazers » Blog Archive » There Is No Such Thing As A Girl Gamer.

Does creating computer games impact girls attitudes toward computing?

This story came across ACM Technews, with the claim that “University of Alberta researchers have found that high school girls become more interested in computer science if video game creation is incorporated into the lesson plans.” That’s a strong and surprising claim, countering what other studies (including ours) have found.

A pre-print of the journal article is available.  The claim is a bit strong.  First, the researchers never asked the kids if they were interested in computer science or game development!  They asked the students to compare how much fun they had short story writing at school, to interactive story writing at the University on a field trip, to interactive story writing at school, and then to compare traditional writing to interactive writing at school.  Kids far preferred interactive story writing at the University to writing a traditional short story –everybody enjoyed the field trip.  However, for the girls, the difference at school was not significant, while it was for boys.  Girls did like the activity, but not as much as the boys, and we don’t know anything about how it influenced the girls’ attitudes towards computing.  To claim that creation of video games might then influence girls’ retention in CS (an explicit claim in the journal article) seems stronger than their evidence warrants.  It might — the evidence just isn’t in this study.

In their study, the researchers wanted to see whether girls would gain as much interest in game development as they boys in the class control group. To facilitate the experiment, they introduced a group of local Grade 10 students to a program called ScriptEase, a tool that allowed them to develop and design their own games. A key factor in the study was having male participants who had more experience than the females in gaming.

Szafron says that there is an inherent creative component to computing science, and that having a student design and construct something using the tool is one way to allow them to investigate that aspect of computing science. “We thought we should have female students create games and see if they are just as excited about making games as male students and see whether it’s an attractor to computing science that is independent of gender,” he said.

Their findings indicated that female students enjoyed creating games as much as their male counterparts; further, they preferred game construction to activities such as story writing. Further, he noted the female students gained and used practical skills that are crucial to understanding computing science.

via Computing science rewriting the program to get girls in the game – ExpressNews – University of Alberta.

Too few women game developers

Really interesting piece about the lack of women game developers, and why the industry wants them.  What I didn’t see discussed here is, “Why aren’t there women in video game development?”  What’s keeping them out?  This is a different question than why there are so few women in CS — after choosing CS, most women are choosing not to develop games.  Why?

While women are playing in greater numbers, working in the industry can feel as lonely as battling aliens on a remote planet in Halo.

According to the Entertainment Software Association, 40 percent of video and online game players in the U.S. in 2010 are female, having inched up from 38 percent in 2006. The number of women working as game developers, however, is much smaller. In a 2005 demographic survey by the International Game Developers Association, only 11.5 percent of the respondents were female.

At Columbia College, Mindy Faber was shocked to discover that the school’s 2009 graduating class of game design majors had one woman out of 26 students. The ratio barely improved in subsequent classes, inspiring Faber to organize a four-day summit about girls, gaming and gender that will take place at Columbia next week.

“Our feeling in our department is that clearly, we can make better games if we diversify the designers,” said Faber, academic manager in the department of interactive arts and media. “If the game designers out there are more inclusive and representative of our general culture, we’re going to make better games that reach more people.”

via Female gamers: Recruiting women as game developers – chicagotribune.com.

Is Media Computation “bait and switch”?

The question that Jennifer Kay raised in her AAAI Spring Symposium paper is about robotics, but her question on the SIGCSE Members list is more general: “Do we have any empirical evidence that cool stuff genuinely does attract more students?”  Bruce Barton changed the question slightly in his message on the list:

Are we doing a disservice to our students by teaching them robotics, animation, game development, etc. when most of the industry is performing fairly mundane computer programming tasks?  I understand that we are trying to increase enrollment and also retention.  But are we perpetrating a bait and switch scam on our students?  Back when I first started out (late 60′s), data processing was where it was at and we enjoyed what we were doing.  Has the video generation had their attention span so decreased that they can only learn if we make the learning experience play-time?  I have heard the reports about video gaming drawing in the students and that video gaming is the new big thing in the industry.  But each year we put out many thousands of graduates who want to become game developers and there are certainly not that many jobs available in that specialty.  Where do the graduates who don’t make it into game development go?  Should we be the voice of reality for them?  Would we really lose that many students if we approached the subject in a less fanciful way?

There is evidence that more engaging approaches in the first semester do lead to improved retention in later classes, even in more traditional classes.  Charlie McDowell found that with pair programming. Beth Simon’s ITICSE 2010 paper shows Media Computation CS1 students succeeding more in a (traditional) CS2, than students in a traditional CS1.

Why does this happen?  Why is it that students stick with computer science, after an engaging start, even if those latter courses are no different than they have ever been?

• One theory is that we simply have to get students engaged, and then they see the value of computing in a broader sense. Once they see computing in the form of a concrete and engaging application area, then maybe they see the value of computer science in its general form.
• Alternatively, maybe the first course sets up the carrot, and students are willing to bear with the rest in order to achieve that carrot.  Students in our Computational Media degree program want to go off to Electronic Arts or Pixar, and they are willing to go through courses that they find less engaging, and even (in their opinion) less valuable, in order to achieve their degree in order to improve their access to the careers they want.  Maybe the first course (in robotics, in media computation, with pair programming) shows them the best that they might find in computer science, and that makes it all worthwhile.

The implication in these statements is that the rest of the curriculum is boring and unengaging, and that most jobs in computing are similar.  Is it true that most computing jobs are boring and unengaging?  That’s counter to what we’ve been telling students the last few years.  Does the curriculum have to be boring and unengaging?  Maybe some students want the pure computing.  In Lana Yarosh’s paper on our Media Computation Data Structures course, we found that about 10% of the students didn’t want the engaging media context — they wanted pure data structures.  In the paper by Allison Tew and others on the use of a Nintendo Gameboy context for a computer organization course, they found that students were much more excited about the “boring” topic of computer organization with the engaging context — and they still learned the computer organization pretty well.

Do we really believe that computer science is inherently boring and unengaging?  Why is that?  Why would we believe that about ourselves and our field?

Women graduate in STEM more than boys: It’s video games?

I found this report interesting, both because of its claim and because of the (what seems to me to be) horrendously flawed logic.  Women are increasingly taking more STEM classes, the author claims, and are nearly catching up to men.  However, more women graduate!  Why?  Well, of course, because men play more video games!  I might use this as an example of correlation-is-not-causation next time I teach the research methods section of my educational technology class.

The number of women taking courses in science, technology, engineering and mathematics, the STEM subjects, has been increasing since 1966 according to a new report. But another study, on boys’ academic responses to new video games, establishes a cause-and-effect relationship that could partly explain the decline in male academic achievement.

Women students in higher education now outnumber men in most countries, except Japan and Turkey. In the US, this has skewed the ratio among the sexes in terms of those who graduate: the proportion of males earning degrees has dropped to 43% while that for women has increased to 57%.

via University World News – US: Women gain in science while video games hold back boys.

Can computing curricula be neutral?

Erik asked a great question in a comment to the “White Boys are Boring” post (a post which was clearly accompanied by a healthy serving of hyperbole, as Kurt pointed out):

Has anyone looked at the comparative efficacies of race/gender neutral programs to increase participation versus ones targeted at specific races or at women?

I do know that curriculum designed to address the needs of women and members of underrepresented minorities work better at attracting those students than the traditional ones — that’s one of the directions that the NSF BPC program has been exploring.  That’s not answering Erik’s question, though. The traditional computing curriculum is not neutral.

Media Computation was not designed explicitly to attract women and minority students.  We designed Media Computation to attract Liberal Arts, Architecture, and Management majors, and we used sources like Margolis and Fisher’s Unlocking the Clubhouse to inform our decisions.  The result is that no published study has found a difference in success rates due to gender or ethnicity, and the published studies show that women are more likely to succeed with Media Computation than with whatever was the traditional curriculum.  That doesn’t mean that Media Computation is neutral — some students dislike it.  The distinction doesn’t seem to be due to gender or ethnicity.

When we design computing curricula, most teachers aim to make assignments and examples motivating and interesting, and in so doing, we speak to some members of our audience, and not others.  When we use video games or robots in examples, for example, we tend to get the boys more engaged than the girls.  I’ve found that it’s hard to be culturally neutral in my own assignments.  One year, I used an example in an object-oriented design course about parts of a car (lots of opportunity for aggregation and part-of relationships there), only to find that my students from the developing world didn’t have much experience with cars and didn’t know anything about parts of an engine.  Our introductory courses used to build assignments around board games like Yahtzee and Risk, which were really engaging for students who knew those games, and a drudgery for those who didn’t know the games.  (Implementing pages of rules for a game you’ve never played is dull.)  There were cultural biases in the choices of games, e.g., favoring the kinds of games that, in the US, middle class kids in Suburbia played.

The question to which I don’t know the answer is whether it’s possible to build “neutral” curriculum.  The academic answer seems to be “no,” but it’s still an issue being explored.  Some of what I’ve found from some digging:

• The prevailing academic answer says curricula are not neutral.  A.V. Kelly’s 2009 book (5th ed) The Curriculum: Theory and Practice says that all approaches to curricular planning have a variety of biases in them.  I found an interesting 2003 journal article that says that that’s not a bad thing (as well as other articles making a similar argument).  Curricular change occurs because of particular strengths/weaknesses of a curriculum and are implemented through leveraging power relations.  The challenge is being aware of the biases.
• That doesn’t mean that there aren’t efforts to create neutral curricula.  In 2004, the UN announced an effort to create a culture-neutral school curriculum.  I found an announcement for an on-going research project that is attempting to build gender neutral curricula.  I found no results on any of these or similar attempts.
• I suspect that some computer scientists would say, “Use Mathematics. Math is neutral.”  I found that the mathematics education community (at least in the articles I found that describing efforts to create neutral curriculum) believes that mathematics is neutral, but mathematics teaching is necessarily value-laden.  I found a nice summary of the claim in Rethinking Mathematics piece at the Rethinking Schools site.

Simply put, teaching math in a neutral manner is not possible. No math teaching — no teaching of any kind, for that matter — is actually “neutral,” although some teachers may be unaware of this. As historian Howard Zinn once wrote: “In a world where justice is maldistributed, there is no such thing as a neutral or representative recapitulation of the facts.”

Bottom line is that I don’t think that anyone can answer Erik’s question.  Maybe the academics are wrong and it’s possible to build neutral curricula — there certainly. are attempts today.  However, if we don’t know if we can build it, then we definitely don’t have any to compare.

Teaching computer games as the next Latin

People still argue that learning Latin improves “critical thinking skills” and “comparative analysis skills.”  Despite these claims, there is little evidence that spontaneous transfer occurs from general learning. Transfer is hard, requires lots of initial knowledge, and works best when students are explicitly taught to transfer. Explicitly, learning Latin does not lead to general thinking skills.  Next up? Creating video games!

Computer games have a broad appeal that transcends gender, culture, age and socio-economic status. Now, computer scientists in the US think that creating computer games, rather than just playing them could boost students’ critical and creative thinking skills as well as broaden their participation in computing. They discuss details in the current issue of the International Journal of Social and Humanistic Computing.

via Teaching computer games.

SIGCSE Preview: Mike Hewner and Game Industry Needs

Next week is SIGCSE 2010, so the sound of scampering feet, practice talks, and impending panic permeates our group here at Georgia Tech.  We have something in seven sessions this year.  Tom Cortina, Program Co-Chair this year and Conference Co-Chair next year, told me how much trouble we’re causing him, to not have us overlap anywhere.  (Barb already discovered that she was double-booked, but got it resolved.)

I thought I’d spend some of my blog posts this week giving previews of talks and sessions that Georgia Tech folk are involved in.  I try to be cautious in talking about student work before it gets published.  This seems like fair pickings, to talk about their cool work (and to drum up more of an audience!).

Mike Hewner is presenting Friday on “What Game Developers look for in a New Graduate: Interviews and Surveys at One Game Company.”  Mike isn’t actually doing his dissertation on game development.  Mike really wants to be a computer science teacher at the post-secondary level.  He realized that many students coming into College today want to be game developers.  So, last summer, he took an internship at a game company, so that he could tell students honestly that he had first-hand experience as a game developer.  While he was there, he did the research for this paper.

There are various efforts going on to define what is the core of CS through efforts like concept inventories, e.g., asking teachers what’s important or hard.  Mike asked a much more focused question, “For what do game developers get hired?” Know what gets you a job as a game developer?  Rather than ask teachers, he asked the people who hire game developers.  He used a variant of a Delphi method, to develop an initial list of needs, then to get his respondents to respond to each other and rank the whole list.

In his dissertation work, Mike is actually interested in a much broader question.  We know that students are showing less interest in computing careers.  Mike is using social psychology to ask the question: How do students become affiliated with computing as a career choice, and how can we influence that affiliation?  He’s got a project going on right now that responds in some sense to Maureen Biggers’ paper about Stayers vs. Leavers.  Maureen found that people who stayed in computing tended to see it as a broad field, while those who left thought it was just about programming.  Mike is trying to see if he can get high school students to broaden their definition of computing, using concept maps to measure that breadth.  That’s probably more than I should say about unpublished (actually, ongoing and unfinished!) work.  If you want to know more, find Mike at SIGCSE next week.

How deep are video games?

I’m excited about the article in this morning’s Atlanta Journal Constitution about games courses in college, with references to two of my colleagues, Blair MacIntyre and Ian Bogost.  I was particularly struck by Ian’s comments.

Georgia Tech professor Ian Bogost teaches students and is co-founder of Persuasive Games, which focuses on social and political issues such as airport security, flu epidemics and tort reform.

Students, he said, are looking for ways to match games with their life passions. One student is trying to meld religious activity with games, he said.

“Games are like folk music of the 1960s,” Bogost said. “They grew up with it. They identify with it. And it isn’t something really co-opted by institutions of power.”

via Gaming courses popular in Georgia colleges  | ajc.com.

Another article in Parade magazine this weekend, Can video games teach kids? includes this quote:

“We’re starting to see agreement that video games are the new liberal arts,” says Kurt Squire, a professor in education communications and technology at the University of Wisconsin. “This school is the first implementation.”

I have Ian’s Persuasive Games but haven’t finished it yet.  Games are “the new liberal arts”?  Games as the “folk music of the 1960s”?  My experience with games don’t go that deep.  I find if I think about them too hard, there’s nothing there but the assumptions and world-view of the game author.  A great example of the bottom not being too deep is the SimCity game player who famously told Sherry Turkle in The Second Self, “If you raise taxes, people riot.” Can games really be as deep as great literature, or great music?  I suppose it’s possible, but I haven’t seen it yet.