A2s, the national budget crisis, and irrationality

Apr 06 2011 Published by under Uncategorized

Here’s the thing: I’m surprised and disturbed by the number of experienced scientists saying here that they are “insulted” by the data presented by Drs. Rockey and Tabak. Data cannot be “insulting.” Data are data, and if they tell you something that you didn’t want to hear–well, that sucks but that’s the data. I understand that keeping our labs afloat is our number one emotionally stressful issue–as a pre-tenure assistant prof., I’m just as stressed out (as an aside, these difficult times are perhaps reminding you all firsthand of how it really has been for junior PIs in the last 10-15 years now that paylines are not at the 30-40% range like they were when you got tenure).

But are you all so married to your hypothesis that the A2 would be your golden ticket that any data to the contrary are “insulting?” Taking data personally, and dismissing an experiment as having something “wrong” with it just because it doesn’t tell you what you want to hear and validate your prediction, are examples of “Bad Science 101.” You are all so emotionally invested (because getting/losing funding is a life or death issue for all of us in this business) in needing to see your prediction borne out in your favor that you are trying to demonize the results of an analysis.

If you think the analysis was performed incorrectly, hey: rePORTER is available to all of us, feel free to repeat the experiment yourself. But step back and remove your personal feelings and bias from the analysis, and do not let what you WANT to see in the results cloud your judgment. (and to the person who claimed they saw some fiddling of A2 vs. A0 status in a grant in rePORTER… seriously dude, take off your tinfoil hat.)

The zero-sum aspect of NIH funding makes the results of this analysis be more like a “Well, duh.” If you originally had three categories A, B and C with which to divide up N dollars with X proposals, you remove category C and just have A and B available, so you now have two categories with which to divide up N dollars with X proposals, uhhh, wow, big surprise, now there are proportionately more of X in each of A and B compared to what was in them when C was around. In other words, if your proposal was so meritorious that it would have been on the cusp between B and C, now you’re going to end up in B. You get funded now instead of having to sit around and wait for later. (and yes, I have experienced the “benefit” of the A2 finally getting funded, but would have much preferred for more A1s to have been funded the time I had a borderline score at A1 than have to go through the motions and wait my turn to be A2).

The data are the data. Being insulted by them is just bad science.

The problem is not that proposals scored at the 13% percentile at A1 are not being allowed to resubmit--the real problem is that proposals scored at the 13% percentile are not getting funded in the first place. Here's what would be a much better use of the time spent by everybody who signed the petition and gets involved in the comment thread: lobbying their fellow citizens and Congress to recognize how important NIH funding is to maintaining jobs in our research community and advancing the biomedical science that all citizens need to improve healthcare in the future. It's like we're playing right into the hands of those who want us to keep arguing amongst ourselves and looking for the most offensive piece of lint in someone else's navel, rather than engaging in a process that might actually have potential to make a difference.

 

*update: thanks to ProfLike for his commitment to the plurality of data. lol. 😛

23 responses so far

  • Namnezia says:

    I like the comment someone put up: "DAMN THE DATA PROVIDED BY THE NIH!!"

  • odyssey says:

    Very nicely put. It is amazing how many loonies with PhD's are out there...

  • becca says:

    I do think it is entirely plausible this change will *shift* what grants do and do not get funded, and it's perfectly possible to be curious/concerned about how that is panning out. And I suspect *some* of the gnashing of teeth is a result of that.

    But of course, the real problem is that when a procedure for allocating resources already seems unfair, making the decisions happen faster is unlikely to make them seem fairer.
    I don't think the data are insulting, but I can see the commenter's point. If you are participating in an organization (and extramural investigators, who devote their time and energy to study section *are* participating in the NIH) but you have no real control over how that organization makes decisions, the *pretense* that your objections do not matter *because* of data that are tangential to your point is insulting. The data do *not* address whether this has meaningfully shifted the population of grants that are funded. It's not "empirical observation be cursed!!" it's "these are the wrong empirical observations to make to address the question of interest".

    odyssey- fundamental attribution error. I'd say it's amazing how loony the NIH funding process (and specifically not getting funded at 13%) can drive otherwise intelligent people. It gives me a good deal of pause when considering whether to stay in academia.

  • K says:

    I Ctrl-F'd the comments and I don't think anyone was actually insulted by the data itself (even though saying one is insulted by data makes me think they're kind of dodgy science-wise). They appear to be complaining that there is a problem with the context and completeness of the dataset (specifically, the economic conditions when the data was taken and how they reflect economic conditions right now - I don't know if that's relevant - and whether NIH is collecting all the data it needs to collect).

    I think a little reading comprehension is needed by a lot of people.

  • K says:

    I don't think the data are insulting, but I can see the commenter's point. If you are participating in an organization (and extramural investigators, who devote their time and energy to study section *are* participating in the NIH) but you have no real control over how that organization makes decisions, the *pretense* that your objections do not matter *because* of data that are tangential to your point is insulting. The data do *not* address whether this has meaningfully shifted the population of grants that are funded. It's not "empirical observation be cursed!!" it's "these are the wrong empirical observations to make to address the question of interest".

    I read the comments more as 'hey NIH, actual people's careers depend on this, and a lot of people are feeling vulnerable because of your institutional problems. Your data collection is grossly flawed, and you need to improve.'

  • Arlenna says:

    Ctrl-F'ing the comments won't get you some specific instance where an individual says "I am insulted by data!!11eleventy!!" However, reading through the entire comment set will, which is what I did.

    Yes, these other elements you point out are all there too: there's a problem with the way funding gets allocated, there are problems with people's careers depending on NIH funding. Mine is one of them, so I know this full well, and as I indicated, I am sympathetic to the emotional distress everyone is under. But reinstating the A2 will not change any of these things: it's a logical fallacy, sort of analogous to thinking that if you just buy one more ticket for the lottery you'll be more likely to win.

    The stochasticity of the review/funding decision process means that there is no significant statistical difference in the likelihood of one individual's reasonably good application getting funded as an A2 in the previous system vs. as an A1 in the current system. The data shown by Rockey and Tabak illustrate this by illustrating the shift in the distribution. And yet, all through that comment thread are people responding with their emotions and personal biases that they still want one more A2 lottery ticket JUST IN CASE rather than accepting the distributions as they are now and moving on with it. Yeah, it sucks really really bad that some labs are having to shut down because of this funding situation. But giving some people an A2 opportunity won't magically create any more money out of which to fund their A2.

    • Arlenna says:

      "will (give you a sense of that)" is what I meant above in paragraph 1.

    • K says:

      I don't disagree with you on this. I think the A2 is silly.

      I was referring more to there not apparently being an instance where anyone said they were insulted by data per se - c.f. Namnezia's comment and your response to it; I didn't see anything like that in the comments, which I read all of too.

      • chemicalbilology says:

        It's in a comment by Allen Kennedy from 4/4 at about 8 am--he really does say "Damn the data provided by the NIH!" and he also says "I looked at the data, and as a reply to the petition, I find it insulting."

        I DO sympathize that this is an emotional issue for PIs. And I agree that it sucks really bad that all the terrible things he mentions are happening. But I don't see any other response that NIH could have given that would have satisfied him or the others any more--other than "Oh, okay, here's the A2 back then because you all just WANTED it SO MUCH."

    • becca says:

      Pedantry:
      It's *not* a logical fallacy to think that buying one more lottery ticket increases your odds of winning. Buying another ticket *does* increase your odds (unless for some reason your buying it makes everyone else buy another, which might be theoretically possible for NIH grants but is far less likely for actual lottery tickets). It is not terribly logical to think the extremely tiny increase in odds *matters*. But it does increase the odds.

      "The stochasticity of the review/funding decision process means that there is no significant statistical difference in the likelihood of one individual's reasonably good application getting funded as an A2 in the previous system vs. as an A1 in the current system. "
      Can you demonstrate that with the data provided?
      Even excluding the fact that the intervention may change the data (i.e. people may write grants more carefully without the A2 as an option), isn't it possible that the grants that are getting funded now are more likely to e.g. come from institutions with more fully staffed grants offices? I don't think RePorter has the data available to address that. Is it a priori a ridiculous hypothesis?

      • chemicalbilology says:

        My accuracy of logical fallacy analogy: meh...

        Yes, you can, it is implicit in the relative % of A0, A1 and A2 funded-proposals in each time range. Assuming you accept that any grant scored within the 20th percentile is pretty damn good (and many of the people in that thread make that very argument), and knowing that the majority of funded grants fell within paylines of, what, about between 20-5%ile (we can find those exact numbers elsewhere if we really want them), the effect of stochasticity on who exactly got those A0s, A1s and A2s can at least be roughly observed.

        isn't it possible that the grants that are getting funded now are more likely to e.g. come from institutions with more fully staffed grants offices?

        Of course it is. But knowing that would STILL not kick a whole bunch of A1s that got funded instead of being made to wait until A2 out of the list. My whole point is that the outrage that's being laid on the A2 sunsetting is misplaced because of the math that describes the zero-sum funding situation. That energy would be much better used elsewhere to actually change their very own collective study section behavior, or lobby Congress, or start speaking in their communities about how important NIH funding is to support.

  • Arlenna says:

    They appear to be complaining that there is a problem with the context and completeness of the dataset (specifically, the economic conditions when the data was taken and how they reflect economic conditions right now - I don't know if that's relevant - and whether NIH is collecting all the data it needs to collect).

    Then by all means, K, load yerself up a rePORTER query form and have at it. All of this information is available and this analysis can be performed and modified independently by any of the people who say they are unhappy with the experiment and its results.

    • K says:

      Then by all means, K, load yerself up a rePORTER query form and have at it.

      I admit I have no idea how to do this.

    • K says:

      I should probably add - I've used RePORTER before to inquire about how many grants go to various schools, and I know how to limit grants to certain years, but how the heck does one factor in the data that is being complained is missing?

      • chemicalbilology says:

        K, the whole point is that you can't, because those things are happening as a result of the budget crisis and not because the A2 was sunsetted. You can search by year and by project number and use a wildcard and either A1 or A2 (e.g. *A2 and *A1), which will tell you how many A2s and A1s were activated that year. It only labels grants that way for the first year of their listing in RePORTER, subsequent continuation years won't have the A1/A2 label and will look just like A0s (which don't show up as specific labels).

        I grant that what people seem to want to see is all the proposals that DIDN'T get funded, ever, under both systems (this addresses Becca's comment below, too)--and no, you can't get that info from RePORTER. I wish you could, but I doubt people would really want their unfunded proposals showing up in a public database. And besides that, it changes from the realm of quantifiable metrics to anecdotes when you try to factor in relative "impactfulness" of grants that missed the payline for other (likely stochastic) reasons.

        • K says:

          NIH can't churn out some anonymous data?

          • chemicalbilology says:

            Jeremy Berg has done that before, by showing success rate-type data and scores vs. funding status data. Sure, it would be great if they would do that same analysis here, but looking at the first graph, where %A2s funded rises significantly from 12% to 32% from 2001 to 2007 (to where it's actually higher than the %A0s funded!) and then goes back down again while A0s rise back up to the highest % of the total, what else can we really learn from seeing the scores correlated with this? That in general, a better score correlates with a higher likelihood of being funded at any stage?

            Sure, we should ask Sally to do the analysis that way. But I don't see it giving a different answer.

  • Dr. O says:

    The problem is not that proposals scored at the 13% percentile at A1 are not being allowed to resubmit--the real problem is that proposals scored at the 13% percentile are not getting funded in the first place.

    Amen. Although it doesn't appear that this will change anytime soon, if the Tea Baggers get their way. I'm praying for money from elsewhere...or being lucky enough to find my way into an incredibly well-scored R grant.

  • Namnezia says:

    Well, for the NSF you only get ONE submission. Each grant is considered a new submission, and you can either modify it or not at your own peril. This seems to work fairly well since, at least that's my impression, they will fund the best grants of that cycle, particularly since the panel members shift so much from round to round.

  • Lorax says:

    The problem is not sunseting the A2, but requiring fundamental changes in A1s that were not funded. Basically, you get 2 shots on your proposal and then 'fuck you new investigator'. With current funding ranges the difference between a funded proposal and a triaged proposal is entirely subjective. With this mindset it does seem like a goal, unintended or not, is to reduce the number of biomedical research labs particularly new investigators and the smaller labs that run on 1-2 R01s.

Leave a Reply