Ego balancing
This post is about astro-politics so if you are disinterested in that sort of stuff, please move along and read something more interesting.
I'm pretty low down the ladder when it comes to astronomy and therefore don't have much experience. My hope in writing this post is to elicit some constructive responses from those that are my elders and/or more knowledgeable, to put me right where needed.
Since the great physics funding crisis of 2007/8 I have been trying to get my head around the politics between people, institutions and the main funding council, STFC. Within the funding allocation process is the very admirable idea that projects should be funded based on scientific merit through peer review. I agree with this sentiment. I am also increasingly suspicious that egos play a significant (that doesn't not necessarily mean large) role in it too. This makes me uncomfortable.
In the publication of scientific papers, peer review seems to work fairly well. The person reviewing a paper may reject it but the author usually has chance to request a second (or even third) referee if they feel they were unfairly treated. This doesn't always work but, in general, the checks and balances seem to ensure that papers are published based on their scientific merit. In funding council decisions however, I'm not entirely sure that peer review is implemented in the best way.
As I understand it, the peer review process consists of a panel of astronomers from different institutions deciding on the scientific merit of different proposals. I've heard that when the panel discuss a proposal that involves one of the panel, that person rightly steps outside of the room so that they don't influence the discussion. This is good behaviour as it should, in theory at least, remove the self-interest of that proposal. However, given that overall funding is limited, all proposed projects are competing against one another. That means that there is a vested interest amongst the rest of the panel, not to fund other projects that will compete against any proposals they have submitted. Also, if someone on the panel has a personal dispute with a person/group that submitted a project, it might be expected that they wouldn't be totally objective.
I am being slightly pessimistic here because I'm sure most people behave with professionalism and integrity but I expect the system to be set up in such a way that reduces the chances of these things happening. Of course, there may be mechanisms I am unfamiliar with and that is partly why I write this. My more informed readers can hopefully expand on this below. Are the proposals looked at anonymously (assuming that is even possible)? Is the panel only formed from people who haven't submitted funding proposals of their own? If it is the way I describe, is there any way to make the system better? I must admit that I don't have any suggestions on that front. Suggestions are welcome below. Please be polite.
As a footnote, I'll mention that my job is funded through the process I've described above. That gives me a vested interest. However, and as much as people will disbelieve me, if improving the system meant I lost my job I'd still rather the system was improved. Yes, really.
Comments: Ego balancing
Just a brief answer without comment on most of your post:
Are the proposals looked at anonymously (assuming that is even possible)?
No.
Is the panel only formed from people who haven't submitted funding proposals of their own?
No.
Posted by Kav on Thursday 19th Feb 2009 (00:20 UTC)
I agree with your pessimism. I think in astronomy the community just isn't big enough to have real anonymity in the review process. Also I find that even if reviewers have no "vested interest" in an application, their opinion is often driven by their own preconceptions and prejudices about the work rather than the quality of the proposal.
Posted by Sarah on Thursday 19th Feb 2009 (10:14 UTC)
Anonymity is almost impossible since things like track record are considered in the process.
Posted by Kav on Thursday 19th Feb 2009 (10:59 UTC)
I'd just add that the argument is that members of these panels (PPAN, AGP, etc) are all professional people who leave their biases and prejudices at the door. When it was suggested that this was not the case during the programmatic review the suggestion was roundly denounced.
However, as professional as members of panels might be there are always subconcious biases involved and these do have to be minimised in the process - a broad panel covering many areas would work towards this since members should balance each other out. The argument against is that it becomes difficult for large panels to make decisions.
Some bad practice does creep in though with the best of intentions. For example, members of PPAN used to (still do?) give presentations to each other about their own work - thus everyone becomes familiar with their colleagues. However if the panel is not fully representative this gives advantage (or maybe disadvantage if its a bad presentation!) to fields with members on the panel regardless of measures to minimise the panel makeup bias.
Posted by Kav on Friday 20th Feb 2009 (16:11 UTC)
Posted by Stuart on Friday 20th Feb 2009 (17:17 UTC)
On the face of it, not a bad idea; it would bring problems though - how do you judge their ability to determine what is quality and what is crack-pottery? Sometimes it's a fine line and without a decent grounding in the topic it might be hard to tell.
Of course soon the government will be telling us what to research anyway. ;-)
Posted by Kav on Friday 20th Feb 2009 (18:31 UTC)