John Ioannidis, known for his comments on medical research (see also the Atlantic article), has an interesting opinion piece in Nature on saving researchers' time writing and reviewing grants: fund people not projects. As he concludes, “Requiring [scientists] to spend most of their time writing grants is irrational. It’s time to seriously consider another approach.”
It was thought provoking, but I don’t think any of his ideas will really work. Lots of people complain about peer review, but I think it largely works well and none of the proposed alternatives would actually be better. Here are my thoughts.
We pretty much already do this, in part
Prominent scientists can get sketchy ideas funded, while obscure researchers have to make an extremely strong case. But obscure researchers with a fantastic idea can get funding.
Spread equally or at random?
Ioannidis says, “the imperfections of peer review mean that as many as one-third of current grants are effectively being awarded at random.” So why not use “aleatoric allocation”?
I’d never seen the word “aleatoric” before. I suppose now I’ll see it all the time.
The idea of funding science via a lottery seems too crazy to consider further.
Merit?
We could fund people on merit versus fund particular proposals. Ioannidis points out that the MacArthur foundation does this. He didn’t mention that the NIH also does this to some extent (MERIT awards, for “Method to Extend Research In Time”).
Ioannidis suggests maybe doing this in an automated or semi-automated way: “The system could use indices that exclude self-citations and capture quality rather than quantity (such as average citations per paper instead of number of papers).”
Blech.
“Reward good scientific citizenship practices”?
Roger would be interested to see this comment from Ioannidis: “Researchers might be rewarded for publishing reproducible data, protocols and algorithms.”
But Ioannidis further says, “Some citizenship practices are difficult to capture in automated databases, so would be subject to the disadvantages of peer assessment.”
That’s an understatement.
Simplify application?
“Researchers could be asked, for example, to submit short summaries of their intended research, describing broad goals only.”
The NIH has moved in that direction. But the problem is: we can all generally agree on the important problems. The question is: who is going to solve them and how?
Judging quality
“Many institutions use the size of a scientist’s grant portfolio as a basis for tenure and promotion….Judging scientists by the size of their portfolio is equivalent to judging art by how much money was spent on paint and brushes, rather than the quality of the paintings.”
Who could disagree? Similarly, counting publications (perhaps considering the “impact factor” of the journals) is a poor measure of quality.
The only alternative is to actually read the papers.
How to spread out the money?
“All funding options face a tension over how many scientists should receive awards, and there is no good evidence on whether it is better to give fewer scientists more money or to distribute smaller amounts between more researchers.”
The NIH has indeed struggled over this. But I think they do reasonably well. There are some groups with a ton of funding and some that are struggling. How to decide how much to give each? Well, you could focus on the proposed projects rather than the researchers…
Randomized trials?
“Controlled trials could randomize consenting scientists to different funding schemes, then compare surrogate metrics and long-term successes.”
It’s hard to see anyone really doing this. It would require a long study and the risk of destruction of individual careers.
Scandal?
“It is a scandal that billions of dollars are spent on research without knowing the best way to distribute that money.”
Similarly, Richard Smith wrote an interesting opinion on peer-review (focusing on journal articles), in which he notes the lack of empirical evidence for the efficacy of peer review.
But is it a scandal? Do we know the best way to distribute money for anything?
My conclusions
I would like to spend less time writing and reviewing grants, and I think the current system favors shorter, fashionable, less innovative projects. However, while it is hard to gauge the value of proposed work, I still think it’s easier to evaluate proposed projects than it is to evaluate people. What’s done at the NIH is a bit of a mixture (evaluating the individual investigator and their past work as well as the particular proposed project), and while I don’t always like it, I think it’s hard to improve upon.
The biggest problem is the low level of current funding. When only about 10% of proposals are funded, chance seems to play a larger role in what gets funded, and many more grants are submitted, which gives reviewers much more work. We need to get it back up to 20% for the review process to be healthy again.