Those SIGCSE Reviewers say the darnedest things!

October 26, 2010 at 11:52 am 8 comments

SIGCSE Symposium 2011 acceptances and rejections came out this last weekend.  I’m thrilled with how our group did.  We submitted three papers: Allison’s on her dissertation work, Lijun’s on our community support for CS teachers, and Davide Fossati’s on his interview study of CS instructors. All three were accepted.  Barb and I submitted two workshops — both were accepted. I was on two panels or special sessions — both were accepted.  Cool!

And yet, I’m annoyed.  I’m annoyed because I’m concerned with raising the quality of work at the SIGCSE conferences. We can only do that with reviewers who recognize good quality work, and we still have reviewers in the pool who don’t understand science.

Davide Fossati interviewed 14 post-secondary CS instructors at 3 different institutions, asking them to tell him stories about when they changed something in their classes, and how they decided if it was successful or not.  Each instructor gave at least one story of success and one of failure.  Davide used appropriate qualitative analysis techniques to draw out themes and commonalities from these stories.  In my horribly biased opinion, it’s good work that tells us something that we didn’t know about the decision-making process of CS instructors.  It’s not comprehensive, but you can’t do a comprehensive study without first knowing what you’re looking for.  That’s what Davide did.  We don’t claim that the reasoning we saw represents all instructors.  However, the reasoning we saw really does exist, and the reasoning we saw repeatedly, represents processes that are not unique to a single institution or instructor.  That’s important to know.

One reviewer completely hated the paper. “I don’t believe that interviewing 14 Computer Science instructors from three different institutions sets the appropriate framework for drawing conclusions that are statistically valid and meaningful.” The reviewer and we do agree on the goal — it’s about arriving at “meaning,” drawing conclusions that mean something.  But one can arriving at “meaningful” without being nationwide or worldwide or even statistically significant.  “The paper as it stands is not ready for publication, in my opinion, for the reasons cited above. The paper has the look and feel of a pilot study that will be used to design the actual statistically-based study.”  The “actual” study?!?

The definition of “good science” is not “used statistics!” I do believe in the use of statistics. Statistics are important for testing generalizability and for checking that you’re not fooling yourself.  They are really important in science.  But science is also about seeing what’s there, before you try to measure it or even try to explain it.  Cataloging biological specimens is an important part of science.  Asking people why they do what they do is the first step to understanding why they do it.  If we want teachers to make better decisions, let’s first figure out how they’re making decisions now.

Our papers don’t always get accepted to SIGCSE conferences.  Only half of our ICER 2010 submissions were accepted.  When the reviews are fair and well-informed, you learn something even from the rejections.  ICER reviews have been really, really good.

This review points out the roadblocks to getting good work published at the SIGCSE Symposium, disseminated to a wider audience, and informing (and hopefully, improving) a community.  Reviewers like this allow for the publication of meaningless work that has a good p value, and inhibit meaningful work that tells us something new that we didn’t know before.

To return to my original point: I’m thrilled.  Davide’s paper did get accepted, so the process of multiple reviews and a meta-reviewer (7 reviewers total!) corrected for the statistics-obsessed reviewer.  The system worked.  Nonetheless, it’s important, as a community, to continue to have conversations about what is good quality work and what we should be looking for in a review.  In my opinion, we need reviewers who understand the value of publishing work that uses statistics when it’s important for the claim, and doesn’t use statistics when it’s not important for the claim.

 

Entry filed under: Uncategorized. Tags: , .

Commodification of Academic Research Call for Participation: Computing Commons Collaboration Conference

8 Comments Add your own

  • 1. Alfred Thompson  |  October 26, 2010 at 12:18 pm

    Full disclosure – I am a SIGCSE reviewer but I’m pretty sure (hard to be 100% sure) that I didn’t review any of your papers. If I did and I said something stupid let’s pretend I didn’t. Anonymous reviewing protects both ways.

    In any case, part of the problem with getting reviewers is that when you ask for volunteers you are pretty much stuck with who volunteers. There isn’t, as far as I know, a stringent review process for selecting reviewer. Although I assume there is some serious thought put into review assignments when there is a limited pool to select from chairs are in a tough spot. I feel for them. Having multiple reviewers is the best defense of course and SIGCSE does do that. And there are resources for reviewers to go to for guidance before starting their reviewers but we don’t test them to make sure they study. We assume they know what they are doing. On the other hand having “professional reviewers” (defined as something other than strictly volunteers working with minimal supervision) has its own concerns. The potential for those who control the reviewers to do so to the narrowing of what is accepted is scary.

    Perhaps we need a session at SIGCSE about what makes a good reviewer? A panel perhaps. I’m not sure if we could make attendance a requirement. A lot of people who are very senior and who have been reviewing for ages would no doubt take offence – often rightly so. But a strong suggestion that people who are starting in their careers or who are thinking that reviewing would be a good way to give back to the community (as I believe it is) attend might be well received. It might help improve the level of review for a good many conferences and journals.

    Reply
  • 2. J. Sommers  |  October 26, 2010 at 12:23 pm

    Mark,

    Long time reader, first time commenter 🙂

    I wonder whether this is due to the “anyone can be a reviewer” model for SIGCSE. Do you think you’d get higher quality reviews with a more traditional program-committee-by-invitation model? There’s something appealing about the way SIGCSE does it, but it’s always struck me as a bit haphazard.

    Reply
  • […] This post was mentioned on Twitter by CSCWCourse UVic2009, Jorge Aranda. Jorge Aranda said: Raising the bar for CS peer reviews includes being more knowledgeable of and more open to good qualitative work: http://bit.ly/a6iCO3 […]

    Reply
  • 4. Fred Martin  |  October 26, 2010 at 10:58 pm

    Mark — I think you’re asking a bit too much? You got one person with narrow p-value blinders, and the review system corrected for this.

    I’m not sure how it would be possible for things to perform any better. Having 7 people review a paper is pretty fantastically good.

    This process truly speaks to the quality of the SIGCSE conference, and the people who organize it.

    Reply
    • 5. Mark Guzdial  |  October 27, 2010 at 10:07 am

      Absolutely, Fred — the system worked, and seven reviewers is astounding. Nonetheless, we can always strive for better. Reviewers don’t get much feedback about what they’re doing right or wrong. A blog is one form of feedback to the system.

      Reply
  • 6. Briana Morrison  |  October 27, 2010 at 10:07 am

    I do think that the SIGCSE reviews have improved since the addition of “guidelines for reviewers” and with the meta-review process. At least now I receive comments in all the boxes, even if some of them are still “the system made me write a comment here.” But we are moving in the right direction.

    To further improve the SIGCSE reviewer process I have a couple of suggestions:
    1) I think the meta-reviewers ought to report any reviews that are outliers from the other reviews. If six reviewers say the paper represents a significant contribution with a 4 or higher and one reviewer gives the paper a 1, you can probably be sure that the single reviewer has a skewed since of significant contributions.
    2) Authors should have the ability to refute a single reviewer to have it discounted. Now I don’t think this should affect the program committee’s decisions on what to accept (the time frame is too compressed as is), but after the decisions are announced and authors have the opportunity to read the reviews, they should have the ability to submit a counter argument.

    Then the program committee (possibly a new position of Reviewer Referee?) takes the meta-reviewer notifications and author arguments and compiles a list of the reviewers with the most controversial opinions or reviews. If the same name continues to appear, especially year after year, could we not conclude that this reviewer is not contributing to the quality of SIGCSE submissions? Perhaps this is where the “mandatory attendance” at workshop for reviewers could be enforced before that reviewer is allowed to review again?

    I recognize that improving the skill of the reviewers and quality of the reviews is a continuous improvement process, but until we are able to weed out the non-helpful reviews, we will continue to see them.

    Reply
  • 7. John "Z-Bo" Zabroski  |  October 27, 2010 at 1:57 pm

    Mark,

    There is a lot of potential to use statistics in evaluating computer science education, but most universities I know of do not. It simply opposes the administration’s desires as these universities. They want to increase their reputation as a research institute, and so it does not behoove them to regularly publish results on how effective they are at teaching. MIT made a big hubaloo about switching their core CS curriculum away from Scheme and towards Python, but where are the measures of how effective the curriculum was before and after? Here the motivations seem to be guided by the fact other major universities are the ones with the now-rich entrepreneurs. I once heard the joke that MIT graduates work for Princeton graduates.

    Similarly, there are some physics grad programs right now that are accepting Ph.D. students despite the department having no cash to fund any experimental physics projects. However, the program can maintain its reputation for pumping out Masters students in physics by essentially forcing the Ph.D. students to simply get their Masters and leave.

    But for the paper review process, there are simply too many quality papers these days. The pool of talent has grown tremendously. The world-wide educated population, percentage-wise, has increased exponentially in the past 100 years. Traditional conference and review mechanisms won’t work. But neither will traditional ways of information foraging. Alan Kay and I discussed this on your blog a few months ago…

    Reply
  • […] in April) on teacher identity. Later that morning, Davide Fossati will give his paper about his study of how CS teachers decide to change their practice. Luis von Ahn is the closing speaker at the luncheon. Barb was supposed to have a workshop on Alice […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,021 other followers

Feeds

Recent Posts

Blog Stats

  • 1,977,499 hits
October 2010
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

CS Teaching Tips


%d bloggers like this: