I had a conversation with a colleague once who violently disagreed with me about peer reviewing. He specifically said that he thought peer reviews should continue to be anonymous because it allowed you to say things you otherwise wouldn’t. Which I believe is fundamentally wrong. My attitude towards reviewing, which he believes is fundamentally wrong, is that you shouldn’t write anything down that you’re not willing to say to the person’s face. So what I’m going to do today is give you some basic things to think about when reviewing both papers and grants and help you think about framing them in a way which is both professional and kind. There are some excellent resources online on this by places like Wiley feel free to go and check them out. Here I’ll take you through what I do and you can take it with a pinch of salt.
First Read and First Paragraph: First thing I do when I get a paper is give it a good read through. The same way I’d read any paper I hadn’t read before. I’ll read through the abstract and intro, scootch through to the results (skipping the methods, more on that later) and then read the discussion. Once I’ve done that I’ll have formed a rough opinion on the paper and be able to write my first paragraph. The first paragraph in all my reviews is the Fred did…paragraph. Here I will outline the main findings of the paper. This demonstrates to the authors that you have read and understood their work. Remember, at this point I’ve made no notes about errors or anything I think is wrong or presented poorly, that’s for read-through two.
At this point, at least for me, papers fall broadly into three categories. Those are significantly flawed and will require reworking to make them appropriate, those that are decent and merely require some tweaking to make them up to scratch and those that are really good. At the end of your first paragraph this is where you can highlight this. My go-to sentence is something along the lines of Overall this paper is…Selecting one example at random from my reviews folder…’Overall, the manuscript is well written, however, I have some reservations about the quality of the science and the conclusions drawn. These are outlined below’
Second Read and The List: To be reviewing a paper at all you should have enough experience of reading papers under your belt to spot a good one from a bad one, so you should be able to come to similar conclusions about quality at the end of the first read. If you’re not experienced at reviewing papers but you’re learning, do this process with someone more senior. They will help give context to your criticisms and point out whether they’re warranted or not. Because at this point you must remember that your criticism is supposed to help the authors get published. Even if you reject their paper or grant, for whatever reason, you should provide them with information on why you have rejected but also what they can be doing to improve. The most basic questions you need to ask yourself when reviewing anything are:
- What question are the authors or applicants trying to answer?
- Are the ways they are trying to answer that question the most appropriate?
- Are their claims backed up by their results?
The first one is relatively easy and should be well highlighted by the authors. If it’s not, this is a problem. You should, after the first read through, have a basic idea of what the problem is and if you don’t you need to point this out. And if it’s your field, you should also have an idea of whether the question is novel or not. If they’re showing something which has been shown before that’s fine, reproducibility is important in science, but if they’re showing something that’s been shown before and claiming it’s new, that’s a problem.
Numbers two and three require you to have some experience in the field, which is why it’s useful to carry this process out with someone more senior if you’re new at it. Let’s give you an example which covers two and three so you can see where I’m coming from. An author is interested in microglial activity in the region of a stroke. They have stained for a microglial activation marker in tissue sections and counted the number of active glia in different brain regions. A common criticism at this point is that staining for microglia isn’t as quantitative as flow cytometry. However, my experience says to me that they are interested in the spatial distribution of the cells. Spatial information is much more challenging with flow cytometry therefore whilst their technique is less quantitative it provides them with the information they are interested in. A less experienced reviewer might have leapt in and said ‘the authors should carry out flow cytometry on the brains to give quantifiable information’. If I were thinking about this I would go along the lines of ‘The authors might wish to consider an additional cohort of animals using flow cytometry, to confirm their immunohistochemistry findings’. By using the phrase ‘the authors might want to consider’ I have allowed them to say ‘no, we believe our technique is the most appropriate because of x,y,z.’
And this is the heart of being a good reviewer. If the data has been presented well and answers the questions the authors set out to answer then there is no need to force them to do more experiments. They could always do more experiments that is the nature of research. At this point if you think there are experiments which might make the paper even better than it is, then you should set your statements out in a way that demonstrates this is not required for acceptance. I like to use ‘have the authors considered…’. This gives them the opportunity to say ‘yes, but it cost too much’ or ‘yes but it doesn’t work because of this’. At this point I try to split my comments into ‘major concerns’ and ‘minor concerns’ where major concerns are worries about the conclusions the authors have drawn based on the data and minor concerns are ‘I can’t read that graph axis because it’s pixelated’.
If the data have been presented badly or fails to answer the questions the authors set out to answer this is where tact and diplomacy are required. Try not to use overly emotive language like poor or badly done. Be critical and constructive. I recently reviewed a paper where the images were so unclear as to make concluding anything impossible. In this case my comment was ‘In their current format the images are unclear and the lesions are not discernible. Jones et al, 2020 have an excellent example of this type of image, the authors should reformat their current images so the lesions are clear, as can be seen in this example’. This gives them a template as to what I consider the ‘gold standard’. Do not simply provide them with a problem or a concern unless you are also going to suggest a solution.
Statistics: When reviewing papers it’s important to think about the number of subjects used and the statistical tests and approaches used. When reviewing grants, it’s important to think about the power calculations used to reach those numbers. This will vary depending on your field. For example, a cohort of inbred mice is likely to generate a very neat data set if you challenge them all with something pro-inflammatory like lipopolysaccharide. But a cohort of random people taken from the street is likely to generate a very noisy data set if you do the same thing. Is the number of patients or mice or cells adequate to say that whatever they were trying to prove is true, or not true? Did the authors or applicants consider fully blinding their experiments so the data is unbiased? If you don’t believe it’s true, or you believe their data is underpowered, or you don’t believe they randomised their experiments properly then this is all you need to say. Ask for their original power calculations. Ask for their raw data. Neither of these is unreasonable, nor will they result in them having to do more work. And remember, just because something isn’t different, doesn’t mean it isn’t interesting. Negative data have meaning too so don’t dismiss a paper because the miracle herbal remedy fails to cure dementia.
Language: A significant proportion of science is done in countries where English is not the native language and therefore authors may not be completely fluent. There may be a number of errors in grammar or spelling, or even sentences that don’t make sense. Unless this is absolutely preventing you from reading the paper (this has only happened to me once) a simple statement at the end will suffice. ‘The manuscript has a number of grammatical and typographical errors and would benefit from proof-reading by a native English speaker’. Not offensive (hopefully) and doesn’t labour the point with every single time they wrote where instead of were.
Geoffrey Cohen at Stanford wrote a great paper on criticism where he suggested that one of the best ways to begin feedback is by using these nineteen words: ‘I’m giving you these comments because I have very high expectations and I know that you can reach them’. The majority of researchers do good work, have high standards and are really passionate about what they do. When you’re reviewing their work think about how much effort you put into papers and grants and approach the task with Cohen’s opening. If you assume the authors or applicants are doing their best given their circumstances, then you should be helping them achieve their goals. And if this is by telling them that actually, their paper or their application has some major flaws but here are the solutions to those issues, then so be it. By approaching the process of peer review from an unbiased and open point of view science can be allowed to progress without becoming needlessly competitive and back-stabbing. Which, I think we’ll all agree, would make this difficult career a lot nicer to work in.
Dr Yvonne Couch is an Alzheimer’s Research UK Fellow at the University of Oxford. Yvonne studies the role of extracellular vesicles and their role in changing the function of the vasculature after stroke, aiming to discover why the prevalence of dementia after stroke is three times higher than the average. It is her passion for problem solving and love of science that drives her, in advancing our knowledge of disease. Yvonne has joined the team of staff bloggers at Dementia Researcher, and will be writing about her work and life as she takes a new road into independent research.
Leave a Reply
You must be logged in to post a comment.