Print This PostPre-warning everyone if the title didn’t give it away, this might be dry. Really dry. I was sent the UKRI modernisation strategy and asked to have an opinion on it. It has been duly read, Excel spreadsheets have been opened and looked at, numbers have been (if not crunched) then certainly nibbled at. And that means today we’re going to discuss the new UKRI grant assessment strategy, what it means for early and mid-career researchers and beyond that, what it might mean for the institutional landscape of the UK. And I will say when I first started writing this it ended up as an enormous and extremely deep piece on the future of UK academia, which included many of the numbers chewed from the aforementioned spreadsheets but after two days of a very pleasant science retreat with young people in the Cotswolds, you will be pleased to hear I have decided not to be that dark.
What is changing in UKRI grant assessment?
Let’s start with the positives, since I so very rarely do that. There are a variety of problems with UK research funding that have emerged recently. The most basic being that stuff is more expensive and there is not substantially more money to pay for that stuff. We now have the glories (said sarcastically) of generative AI so grant applications have gone up massively but we do not have the resources to effectively review those grants. Thankfully UKRI have suggested changes that go some way to address these problems.
Its two main strategies are to provide fewer, longer grants and to reduce review times. It’s going to do the latter by taking advantage of everyone who is applying for funding in a ‘distributed peer review’ approach, which shares the load across more researchers rather than concentrating it in a pool of experts. It also plans to trial AI on a retrospective set of grants to see how closely aligned with the actual grant outcomes an AI reviewing system would be. Some of the load will also be distributed to institutions (more on that later) so that applications are triaged locally before even being sent. And finally it plans to weight assessment. At the moment grants are reviewed as a whole. You are asked, as a reviewer to think about the team and the expertise and the appropriateness of the budget, but you still give one overall score. This system would weight and score different aspects of the grant differently to provide a more balanced score. Put more simply, if you judge the science independently of the team and vice versa, do they stand up?
We’ll try and talk about the two main problems separately and see whether there are benefits but also whether there could be some issues with their solutions.
Under the new UKRI grant assessment approach, longer grants could give researchers more stability and time for science, but fewer awards may make the first step towards independence harder.
Fewer, longer grants: a welcome change
Let’s start with the length of grants which, on the whole, I think is a good change. Fewer, longer grants would provide more stability for researchers, more ambitious science, less time doing grant admin and constantly applying for grants and a greater capacity to build a team. All of which have been substantially lacking in the landscape of the 2-3 year grant. If turnaround times are currently a year then you have to start writing the next grant as soon as you have this one and that’s not practical or feasible for a lot of researchers. But in the new proposed system turnaround would be two to three months ( and if you had a five year grant you would be able to work on your science for a solid 3-4 years before having to think about the next application. Yes there would be fewer of the grants but this is absolutely proposing a fix to a problem which is very much in existence and will hopefully benefit a lot of people.
Where does this leave early and mid-career researchers?
However, if this streamlining is happening at the level of the project grant then it’s going to be even more challenging for that early/mid-career collective. The researchers who are neither an ECR capable of applying for early-career ‘research protected’ schemes, nor an established PI someone would trust with multi-million-pound programme grants. These people are much more likely to be filtered out because, from an institutional perspective, they are much riskier. If you sit within a large, important and well-funded research group you are likely a safer bet but at that point the science risks becoming less diverse. A small project grant is often a PI’s first step onto the pathway to independence. They use it to build and scale and work their way up to a programme grant. If their only chance is now to be co-I on a longer grant, because nobody would trust them with it, then how are they to establish their own niche?
A small project grant is often a PI’s first step onto the pathway to independence.
So there are some problems with fewer and longer grants, but if we move on to their new reviewing strategies, this is where we hit some bigger problems.
Institutional filtering: who gets through the door?
First, we have institutional filtering, which introduces a level of local bias that is not going to favour everyone. The ‘Future Leaders Fellowship’ already uses institutional filtering, and shows us how this additional layer of selection bias can work in practice. Researchers are filtered out before their application even reaches UKRI without necessarily receiving a clear explanation of what made their proposal less competitive. Was it the science? The size or reputation of the research group? Institutional or departmental priorities? Existing networks? Or was it just that everyone was actually better than them? If those decisions are not transparent, researchers may have little idea what they need to change before trying again.
This highlights a systemic problem that we have encountered many times on this blog which is ‘define success’. If there is no real institutional or departmental definition of success then there is no way for individuals to live up to those expectations and as such they can do everything they think is putting them on the right track and yet still, very much, be on the wrong track.
If there is no real institutional or departmental definition of success then there is no way for individuals to live up to those expectations.
Every stage of UKRI grant assessment selects for something. Yvonne Couch asks whether funding decisions reward research potential or the ability to navigate the selection process.
Grant interviews: selling science, or doing it?
If they get past the triage stage, the next new strategy is potentially interviewing everyone, at least for higher-risk awards. I am firmly on board with the fact that we should be trying to filter out the speed-AI-generated slop by asking researchers about their science. I like writing and am good enough that I could probably get ChatGPT to write me a grant tomorrow on something I was totally unfamiliar with and it would read well because I would have made enough edits for that to be the case. But I would not be able to defend it.
Claartje Vinkenburg and colleagues have written a very good article on optimising the research funding decision process entitled, at least to start, as ‘Selling Science’. In it they highlight many depressing and obvious things about the interview process. It is very beautifully written and I am going to recommend you go and read the article so I don’t end up just copying and pasting all of it but I will do a few bits to highlight their importance. They start by saying ‘Selling science is an enacted performance that requires a stage and an audience. The enactment in presentations and interviews is an opportunity for applicants to make panelists “buy” their application. The applicants appear and perform on a stage while they are watched by a group of evaluators who respond both nonverbally and verbally to what is enacted. In such a demanding atmosphere, this performance may appear as standing in front of a tribunal, which is not equally appreciated by and fitting for all applicants.’
They go on to highlight how language and non-verbal communication all play a role in how we perceive a candidate. Whether you say I or we, whether you are loud or quiet, whether you gesture or hunch. One particularly important bit I am going to paste is about competence. They say ‘When overselling is part of their enactment, applicants appear confident, and panelists can perceive and easily assess this confidence as competence. Indeed, panelists may have gendered expectations in terms of overselling: While acting (very) confident and assertive is clearly expected and appreciated in the enacted performance of men, this is less the case for women.’ They go on to then highlight how these expectations of candidates can affect how questions are posed and answers interpreted, and even beyond that how the group dynamics of the interview panel itself can impact outcomes. These, they say, can end up accidentally reproducing all of the systemic biases at the level of decision making, where less voice and less ‘power’ is given to minorities within the panel.
They make a series of recommendations as to how to deal with many of these challenges which I hope are adhered to. They conclude by asking the same question we all are which is ‘well then if this isn’t a good way to do it, what is?’. Their answer is that ‘in some cases, algorithms (if unbiased), wild cards and lotteries can take over (some of) the decision-making and reduce (some of) the time and other resources involved in instructing and supporting panels and panel chairs. However, not all funding schemes, evaluation criteria and process steps are identical, and these interventions work best when customized to the particular context.’
Every stage of UKRI grant assessment selects for something
Fundamentally, what we can see is that every additional assessment mechanism selects for something. Written grants select partly for grant-writing ability, institutional triage selects for individual status within an institution and interviews potentially select partly for confidence over competence. The interesting question then becomes how closely each of those things corresponds to the ability to actually do excellent research and what an excellent academic looks like. I think this is something where expectations may differ between UKRI and academic institutions and may make ongoing assessment and selection of the future generation of researchers extremely challenging unless everyone’s definition of excellence is made less opaque.
Written grants select partly for grant-writing ability, institutional triage selects for individual status within an institution and interviews potentially select partly for confidence over competence.
Dr Yvonne Couch
Author
Dr Yvonne Couch is an Associate Professor of Neuroimmunology at the University of Oxford. Yvonne studies the role of extracellular vesicles and their role in changing the function of the vasculature after stroke, aiming to discover why the prevalence of dementia after stroke is three times higher than the average. It is her passion for problem solving and love of science that drives her, in advancing our knowledge of disease. Yvonne shares her opinions, talks about science and explores different careers topics in her monthly blogs – she does a great job of narrating too.
Dementia Researcher
Dr Clíona Farrell
Dr Becky Carlyle
Nature Careers Blog
Beccy Owen