The High Cost of Preparing for Proposals

June 28, 2011 at 2:38 am 8 comments

May I whine?

I just got the rejection and reviews from my NSF Cyberlearning proposal. I was proposing to work on inquiry learning in CS Education, i.e., to conduct studies to explore what questions students had about computing, and with some technological probes, see if students could be prompted to have more questions about computing. I had applied for a smallish, two year grant under the “Exploration” category which is to “explore the proof-of-concept or feasibility of a novel or innovative technology or use of such technology to promote learning.”

As I read it (completely biased as I am in interpreting these), I was rejected for basically two reasons. First, I didn’t make the case strong enough that this proposal was “potentially transformative.” That was my fault. I strongly believe that we do not teach computer science via inquiry today, and the case for inquiry learning is very strong, so it is potentially transformative to move computer science education to an inquiry-based model. But if you don’t know CS education (so know that it isn’t inquiry-based) or don’t know the science education literature (so know the results on inquiry learning), that may not be obvious. That was my job to convey that, no matter what the background of the reviewers was, but I clearly wasn’t successful.

The second reason is aggravating. I applied under the “Exploration” category. Here are quotes from my reviews.

  • From the panel summary: “However the project would be stronger if it first conducted a pilot study of such questions and used those findings to inform the design of the technological innovation.”
  • Reviewer #1: “The project outcomes would be stronger if they included at least some preliminary evidence that some learning is occurring as a result of these activities, and that this learning matches some set of predictions.”
  • Reviewer #2: “It seems that the PI could simply ask students some questions about this as a pilot or preliminary study.”

I thought that the whole idea of having the “Exploration” track was to fund preliminary work. That’s what I was proposing to do. I was rejected because I had not yet done preliminary work. I am willing to believe that everyone acted correctly and in good faith, e.g., the reviewers were well-chosen, well-informed, and evaluated proposals according to the proposal solicitation. But that means is that the bar for “proof-of-concept” is really quite high. I was expected to have done enough preliminary work that the “proof” was pretty obvious given previous studies.

At a higher level, beyond Guzdial whining, this is an example of what Rich DeMillo calls the “Cost of Sale.” This is the cost of developing the proposal. Here I am applying for “Exploratory” funding, and I’m being told that I need to do some exploration first. That’s “cost of sale.” What if the “Exploration” failed? Then that’s research cost that was not supported by an external funder. Whatever an external funder might later provide would not cover those earlier costs for the preliminary or pilot work. This is one of Rich’s top ten reasons why Universities lose money on research.

Okay, back to figuring out the next proposal…

Entry filed under: Uncategorized. Tags: , , .

It’s about getting more people in CS, of all types, and social media helps The Internet is a playground for the affluent: Not a democratic classroom

8 Comments Add your own

  • 1. Alan Kay  |  June 28, 2011 at 7:42 am

    Hi Mark

    You put your finger on several of NSF’s most critical problems and barriers — which include the time and effort it takes to write a proposal “on spec(ulation)” as they call it in Hollywood, and as usual, huge problems with both “peer” review and of number of hurdles in the review chain.

    “Expeditions” sounds good (more like classic ARPA in the 60s) until the fine print is examined, and then it is revealed to have even more hoops to be jumped through than regular proposals.

    NSF rarely finds “peers” for proposals by top researchers, and though the Program Managers have the power to override peer reviews, they rarely have the discernment and courage to do so.

    A trickier part of NSF’s process wrt CS funding is intertwined with its partial charter to provide wide support rather than to filter and provide preferential funding to top researchers. ARPA went rather far in this direction — with great results — and without (as far as I know) abusing the preferential aims of the funds.

    Finally (for this note), one of the most pernicious and damaging parts of most reviews of proposals is the bias of the system (and especially most reviewers) for “engineering type” proposals even for exploratory processes — they want the researcher to tell them how something is going to be accomplished, even though the request for funds is to *discover and formulate* an important problem, and to *figure out* how to solve it. Much of the 60s ARPA-IPTO funding was of this nature, and NSF CS funding has generally failed to find ways to do this.



  • 2. Alan Kay  |  June 28, 2011 at 8:15 am

    P.S. I would be very remiss if I didn’t mention that the NSF staffers, both “floating” and “permanent” are extremely dedicated to their jobs and are quite undermanned as the result of poor support by Congress, especially when there is overspending in other parts of the budget (for wars, recessions, bailouts, etc.). Nonetheless, they just put in very long hours for long periods of time to keep things rolling.

    It is estimated that the US is spending about $150B/year in Iraq and Afghanistan. NSF’s total budget is about $6B/year so we are talking about just one comparison of priorities that reveals the US to be allocating roughly 2 weeks of our yearly direct war budget for our scientific research!

    The CS research funding by NSF is about 85% of the total for the nation — at about $650M/year — and, in my opinion, should be organized and solicited and distributed very differently.

    One of the interesting problems here is that NSF is set up to be driven not just by governmental dicta, but also by “the field”, which in most cases — especially CS — means academia.

    So if academic computing falls below threshold in the sense of what should be going on and how to make progress, then the funding will follow suit in many respects.

    I think academic computing is not as it should be — so during the last 5-6 years that I’ve been on advisory committees for NSF I’ve been urging them to be more proactive about what the field should be doing — both in general and especially in university — and to aim solicitations that will “encourage” more forward looking less incremental and less industry affected processes for making real progress in what computing is all about.



  • 3. beki70  |  June 28, 2011 at 8:23 am

    I once got the following sentence in the panel summary of an NSF grant that was rejected: “The major problem with this proposal hinges on the fact that this work hasn’t been done yet.” I was furious, now I read it as a commentary on how sure the NSF reviewers and PMs need to feel about what they place their bets on.

  • 4. Leigh Ann Sudol-DeLyser  |  June 28, 2011 at 8:33 am

    I think this says it all..

    And part of the reason its so hard for us NSF “newbies” to break into the cycle.

  • 5. gasstationwithoutpumps  |  June 28, 2011 at 11:05 am

    NIH is worse than NSF in this regard. They want only “hypothesis-driven” research that has already been done and is just being confirmed by yet another expensive experiment. They also do not want any basic science, just stuff that can be applied in 2 years.

  • 6. Greg Wilson  |  June 28, 2011 at 12:16 pm

    One hidden cost that’s missing from Rich’s list is how the grant process discourages people moving from industry to academia. Having raised, or helped raise, a considerable amount of money for software development projects, I was at first confused, then angered, by how much pointless boilerplate was required by NSERC (Canada’s equivalent of the NSF). I then had five grant applications out of five refused over three and a half years, in part because I didn’t know how to play the game, but also because my 19 years in industry didn’t count for anything in NSERC’s scoring formula. After I gave up in January 2010 (as other people I know had given up before me), it took me just 11 weeks to raise $200K for my next non-academic project.

  • 7. Briana Morrison  |  June 29, 2011 at 9:59 am

    The other large problem with NSF grants and computing education (and other discipline specific education fields) is the timeframe. NSF wants “transformative” results in a 3 year time frame. No funding for any type of longitudinal study that could really inform the field.

    • 8. Alan Kay  |  June 29, 2011 at 2:42 pm

      Hi Briana

      You make another very important point here.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 8,988 other followers


Recent Posts

Blog Stats

  • 1,869,001 hits
June 2011

CS Teaching Tips

%d bloggers like this: