Podcast – How to be a GREAT Peer Reviewer

Voice Over:

Welcome to the NIHR Dementia Researcher podcast brought to you by dementiaresearcher.nihr.ac.uk, in association with Alzheimer’s Research UK and Alzheimer’s Society, supporting early career Dementia Researchers across the world.

Adam Smith:

Hello, and thank you for listening to the Dementia Researcher podcast, where we discuss career, science, and research. I’m Adam Smith. And today we’re here to talk about peer review, specifically how to be a good peer reviewer. Joining me we have three brilliant early career researchers who have all undertaken peer review and who I feel you would be happy to have looking at your manuscripts. I’m delighted to welcome regular Dementia Researcher blogger and guru, Dr. Yvonne Couch from the University of Oxford. Hello, Yvonne.

Dr Yvonne Couch:

Hi.

Adam Smith:

We also have the amazing Dr. Martina Bocchetta from the University College London.

Dr Martina Bocchetta:

Hi everyone.

Adam Smith:

And finally, the awesome Dr. Isabel Castanho from Beth Israel Deaconess Medical Center in Boston, all the way from USA. Hello Isabel.

Dr Isabel Castanho:

Hi.

Adam Smith:

Well, as it’s traditional for the show, let’s start with some proper introductions. Yvonne, why don’t you go first, tell us about yourself.

Dr Yvonne Couch:

Thank you, Adam. Hi, I’m associate professor Yvonne Couch and I’m a neuro immunologist. I work at the University of Oxford where I study the potential for stroke to turn into dementia and the role of endothelial cells and extracellular vesicles in that process.

Adam Smith:

Brilliant, thank you. Martina.

Dr Martina Bocchetta:

Hi everyone. I’m Martina Bocchetta. I’m a senior research fellow at the dementia research center at University College London. And when I’m not reviewing others paper, I try to write some on my own. My research area is in trying to measure changes in brain structures by using neuro imaging, and I apply that in various form of frontotemporal dementia.

Adam Smith:

And of course, UCLs at the forefront of all that work. Thank you, Martina. Isabel.

Dr Isabel Castanho:

Hi, as Adam said, my name is Isabel Castanho and I’m currently a postdoctoral research fellow at Beth Israel Deaconess Medical Center, or BIDMC and Harvard Medical School. I did my PhD at University of Exeter, actually in the UK, with Professor Jonathan Mill and Professor Katie Lunnon. But I recently put down the pipettes completely and I’m dedicating myself to computational studies using bioinformatics with Dr. Winston Hide. I joined his lab at BIDMC last year. And in my current research, I’m investigating protective mechanisms to Alzheimer’s disease with the special interest in single cell and spatial transcriptomics.

Adam Smith:

Fantastic. Well, thank you all of you and thank you for making time to talk to me today. And of course, we should say you’ve all been on the podcast before as well. This doesn’t happen very often. Yvonne’s been on both sides of the table as interviewer and interviewee. Martina joined us to talk about FTD before and some of the conferences. Isabel talk was with us. If you want to hear Isabel talk about the stresses of being in the pandemic, you can go back to those. We’re trying to forget about that though now. Back to today, we’re to talk about peer review. Let me set the scene. Reviewers play a pivotal role in scholarly publishing. The peer review system has been criticized for being out of debt, but fundamentally it exists to validate academic work to help us improve the quality of published research and increase networking possibilities within research communities.

Adam Smith:

Despite criticisms, peer review is still a widely accepted method, the only widely accepted method for research validation. And when done well it should improve the outputs. Let’s get into it. Martina, I’m going to come to you first. Thinking generally, what do you look for in a good constructive peer review? Because today what we’re hoping to do is to send people away who haven’t done peer review before with a sense of what they need to do to be good. What do you look for in a good constructive peer review?

Dr Martina Bocchetta:

When I think about giving a very good constructive peer review, I think that we need to be clear and specific. We need to provide structured feedback, and also, we need to be fair, helpful, and friendly. What I mean with the feedback has to be clear and specific, is that we need to specify with example and be as specific as possible. Any comments, any suggestion, any opinions have to be backed up with evidence and it not just a matter of saying, this doesn’t work, or this is not good. We need to specify why and how we might want to address the specific point. And when I think about feedback that has to be structured, I consider helping both the author who has to go back and possibly address the comments, and also the editors who needs to understand whether the piece is good enough to be published or not and by giving a structure.

Dr Martina Bocchetta:

For instance, I when I see to start with a very brief summary of the paper and then perhaps an overall opinion on the manuscript and then followed by major and minor concerns with a bullet point so it’s really clear what’s working and what doesn’t work. And then I mentioned that I believe feedback and a peer review has to be fair and helpful. We just don’t want to criticize or destroy a piece; we want to help to improve the content and being very polite and respectful. I think the number one rule is we want to review others paper as we want ourselves to be reviewed. I think that’s the key element.

Adam Smith:

So, specificity is key in there, no vagueness, well structured. We’re going to get into the details of how you go about doing that. But as a recipient of that feedback, specific feedback is key. Isabel, did you ever get any training? Actually, before I move on to Isabel, does anybody else ever have a comment to make on that? Yvonne, come on. What do you think makes a good peer review?

Dr Yvonne Couch:

I think what Martina said was brilliant. I think just for me, I think you should give feedback in the same way that you would if you were sitting face to face. There is absolutely no need to be rude and there’s absolutely no need to make this egocentric. If they didn’t cite your paper and your paper’s the only one that says something, then sure, tell them that they need to cite your paper, but not everybody needs to cite your paper just because you think your paper is shiny. Don’t be egocentric and give feedback in a way that is approachable and friendly, I think is a great way of describing.

Adam Smith:

However, if you are on the other side of this, that’s a top tip, right? If you are asked to suggest reviewers, you might want to just double check that you haven’t suggest reviewers that you could have cited and haven’t. If you’re going to suggest reviewers, maybe actually would you deliberately go out of your way to find a place to cite them if you’re going to suggest them as reviewers, I can see that being a commonly missed thing, that you’d bang your head against the wall afterwards and go, no, why didn’t I do that?

Dr Yvonne Couch:

Well, for some fields it’s really easy because the field is enormous, and so you can always find somebody to review your work and you might have missed their work because the field is enormous. But some fields are super tiny and there’s actually not that many papers on whatever it is that you are writing on. So actually, hunting for reviewers that you don’t already work with is actually really difficult. And then not citing them is probably an error at that point.

Adam Smith:

Okay. That’s got to get into every top 10 common fail, if you ever did one. Isabel let’s come back to the start then. Did you ever get any training on peer review before you undertook it for the first time, and how would you suggest people prepare to embark on this?

Dr Isabel Castanho:

It’s interesting that you ask that, and I did get some informal training. Well, it started as informal, because I think what helped me the most was my absolutely amazing PhD supervisor, Professor Jonathan Mill at University of Exeter. I think it was in my second year of my PhD I believe; he did this private training for students and postdocs in the group. It was so good. I found it so helpful that about two years later when I was the early career researcher rep for the ARUK Southwest Network, I convinced him making it into a workshop, which was a success. But as part of that training, in addition to teaching us why peer review is important, which you touched on some of the points, Adam, the process, the different types of reviews, giving us advice in general and a structure that was very similar to what Martina said. So that’s usually the structure I use as well.

Dr Isabel Castanho:

In addition to all of that, John shared really good resources that I often go back to. I know that Nature, for example, has a free course, I think it’s a three something hours course that is pretty good. And there are several great articles on the subjects of peer review. If you Google, there are great resources out there to prepare you for your first time. But I would say that practice helps a lot. You get better as you do it. I can really see myself improving the more I do it. Especially by comparing your review with the reviews from other peer reviewers, it gives you confidence when you see that someone said similar things, gave similar feedback to what you did and you weren’t sure what you were saying, but if other people said it, it must be right.

Dr Isabel Castanho:

But also, I’ve noticed sometimes I noticed things that other reviewers haven’t, and that gives me confidence as well. I also noticed that I became faster the more reviews I did for sure. And this is actually something that I’ve been trying to improve, to be more efficient with my own reviews, because initially I used to, and sometimes I still do, I used to take too many hours to read the paper and to write my review. And then it was particularly frustrating when I would realize at some point that the paper was actually not a good paper or was very far from being at a good stage for publication. And then I would realize that I was getting lost into details when I should be giving more general advice. And then I could just go into details on the second round, I guess.

Dr Isabel Castanho:

I still particularly struggle with this, but I remember on the training that John gave us, he advised us reading the paper several times. The first time you look into more general things, and you have a bigger overview of the paper, and you can start by doing the summary that Martina mentioned, and then on the other rounds, then you go into the details. I think something else that I wanted to mention on this question is that I found it relieving to learn that it’s okay not to know about absolutely everything in the paper. When this is the case, it’s important to be honest with the editor. Often you have a section where you can tell the editor that a particular method or particular section of the paper you weren’t too comfortable with, and you advise them finding someone else to make sure that that’s reviewed properly.

Dr Isabel Castanho:

And then if it’s something I used to know, but I don’t remember, I go read papers. I go study to remind myself. And it’s also okay to ask for help from a colleague. But an important note here is to make sure not to break confidentiality so you can make more general questions like, for this particular method, what in your opinion are good practices or what are resources where I could read more about a good example of good practice in this method, et cetera. And then finally, I also find it very helpful to think about what helps me the most when I’m the submitting author. I think both Martina and Yvonne already touched on this, but even before I start writing my review, I try to remind myself when I’m the submitting author, what helps me? What are the main points that would help improve my paper?

Adam Smith:

So, preparation is worth saying that there are courses out there that people can go away and specifically go on. I know I’ve heard before as well how some of the little writing groups that conform in teams and in certain labs if you’re lab based, or if you are clinically based, where even if you’re not an author on the paper, offering to have a look at colleagues as papers in advance and treat it as a dummy run almost. Because fundamentally if you are fifth, sixth, seventh author on a paper, could you also treat that a little bit like peer review? Like you probably didn’t write most of this, your names on it because, I don’t know, you helped with an experiment and they’re throwing you a bone by putting your name on the paper. Right?

Adam Smith:

But you could treat those a little bit like peer review, approach it in the same way which will be a way of maybe doing this before somebody’s career relied upon it. I’ve come across that approach before as well. Thank you, Isabel, that’s really insightful. Yvonne, take us through it. You’ve just been sent, Nature of sent you a paper, you are the best person to review this paper. You don’t know the author, it’s somebody in America. It’s got some clever stuff in there. Take us through step by step, how do you approach this. First of all, from the email that you ignored for three weeks when they chased you and said, will you really, really look at this, and you just ignored it.

Dr Yvonne Couch:

And then they sent you another one at nine o’clock and a Friday night. You’re like, no, I’m not. It’s the weekend, I’ll do it on Monday.

Adam Smith:

you’ve got a week. No, go on.

Dr Yvonne Couch:

You’ve got 20 hours to do this review. Martina and Isabel have already started to touch on some of the things that I do. I review papers in quite a set order. I’ll start by saying, I try not to accept lots and lots of reviews. I only have so much time and everyone’s time is precious. And if it’s going to wipe out half a day or even a bunch of hours, there’s only so many half days, so many hours you have. But it’s important that if I do accept, I make sure that it’s something that I’m qualified to review. I get sent quite because I’ve got the word stroke in some expertise, database somewhere. I get sent quite a lot of clinical stuff, which I always turn down, because it’s not my field. I don’t know enough about clinical/imaging. I don’t know enough about how they present their data to review it well. So, pick something that you are going to be comfortable reviewing.

Dr Yvonne Couch:

Once I get the view, once it’s landed, I start with the abstract, I read through it, and I get a feel for the paper. And then what I do is, this is not the way I normally read papers, but this is the way I read a paper for a review. I’ll then go straight to the introduction to see how they’re pitching it in terms of the field in general. At this point I should have an idea of what the paper is going to be about, but I’ve not made any comment yet. So, all I’m doing at the moment is reading. I do exactly what Isabel said her supervisor did, which is read it a whole bunch of times and you get a bigger picture every time you read it. I’ve just done the introduction. And the only notes I’ve made at this point is maybe spelling and grammar.

Dr Yvonne Couch:

So, science is done everywhere in the world, English isn’t everyone’s first language. And sometimes the introduction is not that clear because the authors aren’t native speakers, but that’s fine and it won’t affect my judgment of the science. I might just scribble, needs checking by a native speaker. I won’t make any notes beyond that. And like I say, it won’t affect my judgment with the science. I then skip over the methods to the results, because if it’s my field, I should be vaguely familiar with the methods. I’ll go back and read them through later with caveats and specific things in mind, but I’ll skip straight to the results. Now, for me the rule with papers is that you should be able to visualize the figure from the text in the manuscript and you should be able to write that text by looking at the figure and reading the figure legend. So those two should work essentially as a double of each other.

Dr Yvonne Couch:

Usually all I do is read the headline from the figure and then look at the figure and the figure legend. And here is where I might start to make judgments. So, do the images show what they say? Are they clear? I’ve seen a lot of papers that have microscopy images where they’ve just, you don’t know which bit of the brain they’re in or the figures they’ve uploaded are really fuzzy and you just can’t see things. So, have they used the right groups in their experiments? Have they approached the experience using techniques which are appropriate for the questions they’re asking? This is where I start skipping between sections and rereading things. I skip between the results, the methods, and the discussions.

Dr Yvonne Couch:

For example, if the controls they used aren’t clear, or they did behavioral experiments I want to check, I’ll go back and look at the methods, see how they did them and make sure they analyze them in a way that I think is appropriate. Did they blind them? Did they do power calculations? Did they do the right statistics? And if they claim something has worked, but they’ve used a technique which I think maybe a bit out of date, I’ll check through the discussion to find out whether they’ve acknowledged it. It’s fine to use things which I might not necessarily agree with. As an example, there are lots and lots of different models of stroke, and some of them are more representative and translatable than others. And if they’ve used one that I might not necessarily use, if they’ve acknowledged that in the discussion, then that’s fine, as long as they’re using something, which is appropriate to answer the questions that they’ve asked.

Dr Yvonne Couch:

So, for all of this bit, I scribble in the margins of the paper as I go along. I’m a scientist, I can’t afford a fancy iPad. If I had a fancy iPad, I’d be scribbling on a fancy iPad, but environmentally friendly and I print it out and scribble in red pen. Once I think I’ve gone through the whole paper and circled graphs and said, I don’t believe this, or this looks great or whatever, that’s when I start writing and outlining my review. I do like Martina says, I always start with a brief overview of the paper, because this shows the authors that I understood what they were trying to do. And it gives them a bit of confidence in my reviewing abilities and the fact that I’ve actually read the paper. Then I split my comments, again, as the other guys said, I split my comments into major and minor comments, where major comments are things that are unclear or things that definitely need fixing because their data don’t back up their claims.

Dr Yvonne Couch:

But sometimes all the comments might be minor. So, they might just need to think about adding some extra stuff to the discussion or adding some different images or some extra images to clarify things or fixing an access label or something. So, unless they claim something really strong, which just isn’t backed up, which happens a surprising amount, I try not to suggest a lot of extra experiments. I try and phrase it by saying, if I don’t want them to do extra experiments, I try saying, have the authors considered? Because that allows them to come back in the rebuttal and say, yes, we have, but here’s why we didn’t do it. And it gives them that flexibility. If I really think they need to do something because they’ve missed a control group, I will say the authors have not done this therefore cannot prove this.

Dr Yvonne Couch:

And I’ll outline exactly what they need to do and why I think they need to do it. So broadly as an overview. What I try and remember is exactly what Isabel said, I try and remember what it’s to be an author when I’m writing a review. I’m trying to be constructive, and I try to be polite. I’ve had the fun experience twice in my time reviewing of being able to say this paper is great, I wouldn’t change anything. And I think that’s important too. I don’t try and change what I’m reviewing unless it’s unclear or wrong. I usually try and think about what I’d say to the authors if we were sitting and just having a coffee. That is broadly how I put together a review.

Adam Smith:

Brilliant. You mentioned at the start there that you give a summary at the start, so you literally would write a short section, repeating back the study at the start just to acknowledge that?

Dr Yvonne Couch:

Exactly. Mine is usually not more than about, I don’t know, five sentences or so. And I say, Fred Etal, in this paper have shown X, Y, and Z. And then I will have a concluding sentence that says, overall, I believe this paper is amazing or overall, this paper is broadly good, however, I have some concerns, these are outlined below and then scooch through.

Adam Smith:

I like the comment you made about being clear as to whether you are expecting them to go away and do more experiments as opposed to the potential that this could have benefited from different experiments. Have you often come across something that you really didn’t understand because whilst the overall paper was within your sphere, they’ve done something that’s not for you? And in which case, what do you do then? Do you comment saying, I’m not an expert in this field or would you go away and look it up?

Dr Yvonne Couch:

It would really depend. Broadly I would say that’s never happened to me for a start. It’s not-

Adam Smith:

Because you know everything.

Dr Yvonne Couch:

Because I know everything. I’m that brainy. No, it’s not. It’s just that I broadly know how most techniques work. I might not be able to nitpick most techniques. And if the whole paper is stuffed with techniques that I’m unfamiliar with, then I wouldn’t offer to review it in the first place. But if let’s say they did a paper and they stuck some MRI on the end of it, I roughly know however I works, but I’m not an expert. So, at that point I might do exactly what Isabel suggested, which is either find a friend who does it and say, if you were doing X, is this the most appropriate technique? Or I’d go away and have a bit of a read. If it’s really, really, if they’ve done something completely left field that’s really out of my remit, then I would do exactly, again Isabel said, make a comment and say, this is not my area of expertise and therefore I cannot comment strongly on it.

Adam Smith:

I assume you’d all be comfortable with, if you say you’ll review something, because of course you don’t get to see the paper until you’ve said you’ll review it. And then you get the paper through and go, oh my God, what is this? This is not what I was expecting. Is it okay at that point to just go back to the publisher and say, I’m sorry, I can’t review this?

Dr Yvonne Couch:

Well, you get the abstract. You usually get the abstract and you can usually get a broad feel from the abstract. Again, I’ve only gone back and said, I can’t review this, once. And that was just because the abstract was fine, but the main body of the text, the English was so poor as to be completely incomprehensible. It was like one of those poorly translated menus that you see where they just-

Adam Smith:

They just put it into Google translate.

Dr Yvonne Couch:

Yes, and it had not worked well.

Adam Smith:

You would’ve loved that though. You love going through and correcting grammar. We’re going to move on to that question later. In fact, actually Martina, I’m going to come to you now. Obviously, Yvonne’s given us a really great overview there as to her approach. What do you find most challenging about reviewing papers?

Dr Martina Bocchetta:

The most challenging thing which has been discussed already, it’s when I don’t understand the methodology or it’s not clear whether the results or the discussion is clearly linked with what they’ve done. And sometimes I’m in a position where it’s either due to two things, either I’m not qualified enough to understand it, or I’m not bright enough to understand what they’ve done, or the author may have to clarify because description and what they describe is not clear, so they might improve that. And sometimes it’s difficult to understand, well, at least for me, one, when it’s one thing or the other.

Dr Martina Bocchetta:

Sometimes it’s easier. If it’s a method that I don’t know as Isabel and Yvonne has already said, it’s either go and look it up or use the editor’s comment where you say, I’ve commented on the section, but I’m not an expert in X, Y, Z. Why don’t you contact a statistician? Or why don’t you contact this other person? Because I’m not confident enough to provide my opinion on this thing. It’s just being honest and being clear.

Adam Smith:

That’s okay at that point as well to actually put somebody in and say, this person will be great for that.

Dr Martina Bocchetta:

I tend to be general for an expert in, but yeah, you could. I’m sure that you can also provide an exact name of an expert in the field and maybe it’s also the other reviewer. That’s helpful. When actually it is probably due to the fact that they haven’t clearly expressed or clarified or described the methodology, but I’m still not 100% sure. What I tend to do is trying to phrase in my comments, some rephrasing what I understood. I understood that you did X, Y, Z. And then I asked question in a way that can please the author or clarify whether they did X and Y. And then it’s up to them, whether they want to clarify in the paper. So, make it clear. Or if they just want to reply in the rebuttal and then it’s up to them, but at least you sort of, this is unclear to me and it’s up to them.

Adam Smith:

That must be some of the most common feedback you provide in peer review. Isn’t it? Is this being unclear because something’s just not been explained particularly well. I’m going to skip around with my little bit of questions. Everybody in advance knew what questions were going to be asked mostly, but I’m going to skip around. I’m going to come to Isabel’s questions after, and I’m going to go into the quick-fire questions, just straight in on these quick-fire fingers on buzzers questions that I’ve written down. Question number one, are you ready? How long should it take to do a good peer review on a standard-length paper?

Adam Smith:

How many hours, minutes do you put in on this? Who wants to take that one? Yvonne’s got her eyebrows raised. But from the way you just described it Yvonne, you must take days.

Dr Martina Bocchetta:

No.

Adam Smith:

This is why you can only do two a year.

Dr Yvonne Couch:

Yeah. This is why I only do two a year. No, I would say that said an entirely loaded, how long is a piece of string question? I reviewed a paper yesterday which was awful, and it took me 20 minutes. But I would say if it’s a decent paper, then it should take a few hours. I try not to make them last more than half a day, because like I say, your time’s precious.

Adam Smith:

That’s an interesting point then. So bad papers are quicker to review than good ones?

Dr Yvonne Couch:

Yes, because you get the impression very quickly. It’s the data. If it’s your field, this was an extracellular vesical paper, so I was very comfortable with the field. The data was not robust. They hadn’t shown what they’re supposed to show. They’d shown a microscopy picture I didn’t believe. It was almost instant reject. It was just at that point nothing beyond what they’ve done there, it’s not going to improve. So, you just flick through the figures and you’re like, nope, nope, you’ve gone in the wrong direction with that graph. No, you’ve only got an N of two here. And at that point, it’s very easy to just go. I don’t even really need to read the discussion because I know that none of this data is going to make me happy.

Dr Yvonne Couch:

And you can bash out comments to say, you need to do this, you need to do this, you need to do this really quite quickly. Whereas a big paper where it’s good and cohesive and all the data is solid, at that point you just making sure that all the data is solid, and for that it’ll take a little bit longer, but again, trying not more than half a day.

Adam Smith:

Interesting. Everybody was nodding there as well. I assume you all agree. Is that partly because you feel a better paper has had more effort put into it, so you should too. No, not really.

Dr Isabel Castanho:

Well, it’s also what I mentioned, Adam, of I’ve been on the side where initially it took me a while to understand this about a bad paper. And then I realized I was spending way too many hours on it. And so, I agree with Yvonne, but at the same time, what you also don’t want to do is give one sentence saying, this paper is bad, because I’ve seen reviews that, and it’s not worth my time. You don’t want to do that. You want to give something for them to improve, but you also at the same time, you want to be mindful with your own time and not go into doing what should have been the author’s job in the first place. But yeah, in terms of time, I agree with Yvonne. And like I said, I’m getting better at it, but I would, as Yvonne said, I think it depends on the paper, some papers take 20 minutes.

Adam Smith:

In those instances, on those really bad ones, do you still try to find something positive and constructive? You’ve at least put something in there, good effort. I can see where you’re going, you just need to spend about six more months on this. No, not really.

Dr Yvonne Couch:

Again, it depends on the paper. Sometimes the broad idea might be good. And so, you can say that as positive, you say, I can see where they’re going, but they’ve executed it poorly. But sometimes, I reviewed one at the end of last year and it was just a massive repetition of another paper that had been done way worse than the other paper and hadn’t cited it. And so, at that point, I’m like, well, you’ve not done your literature search properly. You’ve done a ton of research that really doesn’t need to be done. And then you’ve repeated somebody else’s study badly. I’ve really got nothing positive to say about this beyond reproducibility is nice, but please do it properly. Sometimes it’s very difficult to find something to say.

Dr Martina Bocchetta:

Maybe in that case, you can help the author to say clearly, what is the problem? So, work on this aspect, work on that other thing rather than save the paper because of blah, blah.

Adam Smith:

This is a related question. Second question in the speed round. I think we already know the answer to this, but should you add positive feedback as well?

Dr Isabel Castanho:

I would say that initially I thought about this answer with, I always assume always, you should always provide positive criticism, but as we discussed it’s sometimes challenging if it’s a bad paper. But you should at least provide your feedback in a thoughtful and considerate way. You should be rigorous and fair, but you should deliver your feedback with kindness. There’s a human being on the other side that is going to read it and being rude is definitely not acceptable.

Adam Smith:

That’s the slightly weird thing. Isn’t it? You are being blinded to them. You can leave a nice comment, but they don’t really ever know who left it. Making a free donation anonymously, it’s not quite the same thing.

Dr Yvonne Couch:

It’s becoming more open now, isn’t it? Lots of journals say, would you be happy for your name to be published with this review? And I do think that changes the way people review, because at that point, your name is then going to be associated with that paper even if it wasn’t your work.

Adam Smith:

Well, that came up in our, ISTAART early career researcher survey, there was a whole section on peer review and what was wrong with it. And we are working on a separate paper to share and look at those results. But some of the biggest pieces of feedback we had about peer review was that people felt also that the authors of the manuscripts should be blinded to the reviewers. I don’t know. Would you all agree with that? The view was that if you didn’t like that person, or you saw them as a competitor, or you didn’t know them, that knowing the person who wrote it influences your peer review and it would be better if you didn’t know, so that you treated everything the same. Can you be truly independent if you know who wrote it?

Dr Yvonne Couch:

I think at that point it becomes, I’ve had this conversation with a whole bunch of colleagues, and I love it as a concept, but part of peer reviewing is almost knowing that somebody has expertise in their field. If I’m reviewing a paper from a colleague and I know they’ve been doing stroke work for 20 years, then I know already that that work has a fairly solid foundation. But if somebody’s just leapt into the field with zero experience, at that point, do they know the right models to use and are they approaching the question-

Adam Smith:

But should that really make a difference? I might challenge that. Should that really make a difference? That would suggest that you might let somebody who you know very experienced off a little bit if you know that they’re good compared to somebody who you’d never heard of.

Dr Yvonne Couch:

It is difficult. But I think part of the problem is that some fields are enormous, and some fields are tiny. And often if you’re working in a very tiny field, you’ve got a limited pool of people you can cite. So, you often end up doing that self-citation thing. And so, you’ve said, we’ve shown this before. And at that point, everybody knows who we are because you’ve referenced yourself. If you go in for the double-blind review process, you then have to go through your own manuscript and take out your own, not your own references, but you have to take out all of those we reference, which I think for a lot of people in tiny fields might be really challenging.

Adam Smith:

Okay. Next question. Are you ready? Are you doing the review for the authors or the journal editor, go?

Dr Isabel Castanho:

Both.

Adam Smith:

You all agree both. Technically don’t we argue that it’s actually for the editor to then decide whether they’re going to give your feedback. Do editors always include all the reviews from everybody? I don’t know this, because I don’t know an editor. I’m assuming that every time I’ve done a review, everything I’ve shared has gone back.

Dr Isabel Castanho:

I actually think they should filter some, Adam, to start a note here, when is a bad review like I mentioned. Once I reviewed a paper where we were 13 or 15 peer reviewers. That was insane. That was just insane. I think they actually include everyone, and they shouldn’t if it’s a bad review, filtering bad reviews is still an issue in my opinion.

Adam Smith:

I agree. And then feedback to the reviewer saying, would you like to rethink this? Okay, next question. How do you avoid the temptation to fix the paper? We were talking about this off air before we started recording today. I get the impression Yvonne particularly likes fixing papers.

Dr Yvonne Couch:

Have are read. So, fine. I’ll answer it. I think you should avoid the temptation to fix the paper by remembering that it’s not your job to fix the paper. If it’s generally being conducted poorly or it’s not clear, then you give feedback. But if it’s peppered with typos and poor grammar, that’s definitely not your job to go through and fix them all and say, by the way, you mislabeled B as C, that’s not your job. Your job is to judge the quality of the science and whether they’ve used the right techniques to answer the questions that they were aiming to answer

Adam Smith:

Martina. Do you fix papers? Are you guilty of that problem?

Dr Martina Bocchetta:

I’m guilty, but I’m trying to get better. I’m trying to repeat to myself, this is the manuscript I have in front of me, and this is what I have to consider and I’m considering the science, and this is my role. I’m trying to help and provide my comment on an opinion what the author has done. So, this is mantra of keeping repeating that.

Adam Smith:

It’s a good habit to form as well, I think. Last of the quickfire questions. You’ve given your feedback, you’re on let’s say round two. I don’t know. How many rounds is too many? I’ve certainly had one paper recently where we had four, over two years as well. But anyway, so you are on the second round, the author didn’t change the paper to your satisfaction. How do you approach it the second time around? Do you just put the same comment again maybe with an exclamation point at the end or in capitals like you’re shouting? Or do you rephrase it saying, I see this still hasn’t been addressed? I would have particular concerns. How do you do it? Let’s go with Martina again. You go first.

Dr Martina Bocchetta:

I tend to rephrase the issue and then check what the other reviewers commented on. Because if that’s something that I only spotted, maybe it’s just something that it’s on me and it doesn’t bother the others. But if it’s something that the other reviewers have picked up and I really believe that this is something important to solve and the authors are ignoring it, I try to rephrase it. And if it’s still not answer maybe in round three, then I might write something like, I think this is still an issue, but I’ll leave the decision up to the editor to decide what to do with this specific item or whatever.

Adam Smith:

There’s a particular Disney song coming into my head, right? From Frozen right now. I think you know the one I’m talking about. Go on. Isabel, you had got an amazing singing voice. I feel sure you could sing that.

Dr Isabel Castanho:

No, go ahead.

Adam Smith:

Do we let it go, Isabel? Do you let it go?

Dr Isabel Castanho:

Damn it, I was going to say, well, it depends on what do you mean by changes to your satisfaction? Because it depends if it’s a major or minor change, how important it is basically for the quality of the paper. I actually recently had an interesting, related experience to this. It was not in the same journal. Recently, I thought it was a year or two ago. I ended up reviewing the same paper, but in two different journals. That was interesting. The approach I took was being honest, especially to the editor I said that I had reviewed this paper before and so some of my comments were-

Adam Smith:

Wait, wait, I need to stop you there. I’m trying to understand that. Did they send the same paper to two journals?

Dr Isabel Castanho:

No, they got rejected.

Adam Smith:

I see. Right. Was it like, well send it to both and see which one we can get?

Dr Isabel Castanho:

No.

Adam Smith:

That’s quite cunning, right?

Dr Isabel Castanho:

Well, in most journals you have a section whereas an author you sign that you only submitted, that it’s not in another journal, but yeah, I don’t know.

Adam Smith:

Do they know really?

Dr Isabel Castanho:

I don’t know. No, in this case it was rejected. It had massive problems, but it was rejected in the first journal.

Adam Smith:

It came back to you the second time.

Dr Isabel Castanho:

And then back to me the second time in a different journal. But what I noticed was that there were some minor improvements compared to the first submission where, like I said, it had been rejected. But many of my major concerns had not been addressed and persisted for the second submission. I was honest with my review. I actually started saying that I had reviewed this paper before. And basically, at some point I copy pasted some of my comments because it’s like, these are the concerns that I still have. Many of them were with the methods, with the results they were claiming. I was definitely honest with the editor. It ended up being rejected in the second journal as well.

Dr Isabel Castanho:

But overall, I think it’s crucial to reflect on how important those changes are for the quality of the manuscript like I said. And in this particular situation, they were massive. I guess the important thing to remember as well, is that the editor is the one that decides on the final outcome, but they’re looking at us to provide them with honest and unbiased feedback so they can make the best decision.

Adam Smith:

You’re the experts, right?

Dr Isabel Castanho:

Yeah.

Adam Smith:

Exactly. The other thing I think you can always tell as well is just how quickly it comes back to you, right? If you get that paper back within a day, fine. I know we all academics, work 24/7 and are keen to get these papers published and turned around. But if that’s come back within a day, all they’ve done is retype it, right? They haven’t gone away and seriously thought about anything. They’ve probably haven’t done any more data analysis. They’ve just reworded or added, which is fine sometimes if all you’ve asked them to do is to better describe something or to say why they didn’t use a particular method, then that can be dealt with. Martina, go on.

Dr Martina Bocchetta:

Based on what Isabel just said, it just reminded me that, what happened to me was I comment on the paper and then got rejected. And the author sent exactly the same paper to the same journal, and surprise, surprise, it came back to me. And they didn’t change anything. Absolutely nothing. You also need to remember that this is a small community, so it’s very likely that you get the same reviewer for the same journal. In that specific occasion, I just wrote back to the editor saying, this was submitted two weeks ago, and this is the comment. And also, it was not just me being a nasty reviewer or reviewer number two, but all the other I think three reviewers commented exactly on the same thing.

Adam Smith:

Looking at the other reviewer’s comments as well can make the difference. I feel we’ve dealt with that question now. Did you have anything to add, Yvonne?

Dr Yvonne Couch:

No, I would say just exactly what the other guys said. It depends on your feedback. If they need to clarify something experimentally, because it was unclear or the data didn’t back up what they were claiming and they didn’t do it, I’ve got a limit. I do not review stuff more than twice. If they don’t do what I’ve asked them to do on a major concern by the second review, I write back to the editor and say, I’m sorry, they’ve not done this. I’ve asked for it twice, I’m not reviewing this paper again. And at that point I would reject it. But if it’s minor concerns rather than just one of my, have the authors considered questions and they’ve turned it down but for a good reason, then I’d be fine with it. But it’s at your discretion.

Adam Smith:

That sounds all good and well, but I think it does take a certain level of confidence to do that. I think if you are a PhD student or something, maybe doing your first review or your second, I think it takes a certain level of confidence to go back to the editor and to start to say I won’t review this again. I want to reject it. A great way, we’re going to talk about this in a little while, which although time is ticking on, about being a reviewer for abstracts, for a conference and things as well is a nice way to easy yourself into that. But you do feel this pressure that you don’t want to be too mean, you don’t want to dishearten everybody. And ECRs are very conscious of all this now. Confidence comes into that, I would say. Wouldn’t you, Yvonne?

Dr Yvonne Couch:

Yeah. But I would also say that this is very opinionated, but you were expecting that.

Adam Smith:

Well, no way.

Dr Yvonne Couch:

I would say, I don’t think PhD students should be reviewing papers on their own. At that point-

Adam Smith:

Controversial.

Dr Yvonne Couch:

No, I have no issue with them reviewing papers, but I just don’t think they have the long-term experience that you need sometimes to have perspective in the field. They can review a paper, sure, but do it with someone more senior. Because it helps you get that experience. But at that point it’s going to waste your time if it comes back four times, and the authors are still not changing what you asked them to change, and the experiment is wrong or is inappropriate. That’s a waste of your time. But as a PhD student you might not know that. And so, you need to have someone senior on board who’s maybe got the cojones to go back-

Adam Smith:

And you would expect the journal would select reviewers appropriately.

Dr Yvonne Couch:

Exactly.

Adam Smith:

So, they wouldn’t for example send an entire paper out just to PhD students. Actually, that comes up quite a lot on social media. I’m a PhD student, I’m in my third year, I’ve been asked to do a review. Can I? Should I? What would you recommend? Yvonne, would you say no, don’t?

Dr Yvonne Couch:

No, I’d say, as PhD students, I reviewed when I was a PhD student, but I reviewed with people. My supervisor would go, I’ve had this paper to review. Would you look at it? And I would go in and I would read it and then I’d take him my opinions back and we’d sit and have a chat about it. And then he’d do the review. He might add stuff to mine, but I would’ve given it. Because as a PhD student, like Isabel said at the beginning, when you’re first starting to do this, you’re so nit-picky, you’re going in and you’re going, the axis label is not quite right. You’re picking up every single tiny, small detail, which is not necessarily what you need to be doing because it takes ages.

Dr Yvonne Couch:

And it’s not really what the authors need and it’s not really your job. So actually, PhD students are quite good because they do pick up those small details. But then you need someone with more experience to get a bigger picture on the whole thing.

Adam Smith:

As part of a team. And so, per review team. Okay. Time is marching on, so we’re going to come to this final section. And in this final section, I’m going to give common examples of peer reviewer comments that are flagged as bad and ask how you would do it differently. This has come up, this is a genuine comment that’s been added to a manuscript, which is you, well, I don’t think it really was added in this exact phrasing but needs more experiments. How do you put the comment? When do you think this needs another experiment and more experiments? How can you do that without literally just saying, needs more experiments? I’m assuming, we’ve already covered this. Right? Martina said, be specific.

Adam Smith:

You can specifically say, I think another experiment is required, it would be this one and be mindful. You know yourself how long experiments take. If you are asking somebody to go away and do another month’s worth of work at the end of a project that’s possibly already finished and had no money left because they published at the end, not at the midway through, be realistic. I’ve answered my own question there. What would you say?

Dr Martina Bocchetta:

Well, I think first of all, I think we need to learn to distinguish between things that must be fixed before publication and things that there’s no need to be fixed and might be suggestion for future research or for things to consider. It’s a matter of phrasing the comment into, could de author comment on or have the author consider, and then it’s up to them whether they feel they have the time, the data in order to specifically address the point or they can turn it into, this could be interesting, but that’s outside the scope of the manuscript.

Adam Smith:

Is that being mindful about something that’s genuinely, because I guess if another experiment would genuinely add value, you could say this other experiment would add value, but it could be dealt with in another paper, it could be done at another time. Does that extra experiment really stop what’s there being published? Is it good enough?

Dr Martina Bocchetta:

Yes. And then you also give the opportunity to the author understand whether they can do that in a reasonable time or if it’s future research.

Adam Smith:

I think this is probably, one of the reasons why peer review is so criticized at the moment is because the people that have been asked to do it are quite often journals, build up these relationships. Don’t they? With the same people, repeatedly. They’re quite often senior people who they like and who don’t necessarily anymore take the time and the effort that’s needed to really consider the way that feedbacks provided. I think you’d encourage this to journals, this is another reason why we do need more people, listeners at home to go away and volunteer to get involved in peer review, because it broadens the pool, and it takes pressure off Yvonne, Martina and Isabel having to review slightly fewer papers each year.

Adam Smith:

But that means you can put more effort into the ones that you do review if we broaden the pool of people and you can give proper feedback and take time to explain what you mean about experiments, not just say, this needs more experiments. I’m going to move on. Not good, fix it. Okay. Bye. Have you ever had that? That sounds like feedback Yvonne might have given on a paper she reviewed this week. No, I’m sure you wouldn’t.

Dr Yvonne Couch:

No, I wasn’t that brutal. I did say, this is not good. No. I think if you review properly, you should never give problems without solutions. You need to give a reason it’s not good. You need to give a suggestion on how to fix it. I think that’s-

Adam Smith:

I think you’ve perfectly gone on to answer the next issue, which is needs more, needs more. It’s the same thing. Isn’t it? If you explain, you just need to better describe.

Dr Yvonne Couch:

I think you just need to be a considerate reviewer and realize that actually suggesting a 50 to £100,000 experiment that involves a new colony of mice that will take a year and a half is insane. And like you said, and how much will that really contribute to the science that is coming out of that paper?

Adam Smith:

And if you disagree, if you’re on the other side and you disagree with the reviewer comment, you can reply to reviewer comment saying, I’ve done this. Explain. If it’s just missed, just explain it. Reviewers are generally nice people, right? You don’t do this. Nobody’s getting paid here, this is all for the benefit of science and nobody volunteers to do it to benefit themselves, or do they? Isabel, back to your question, yours is going to be the last question today. Early career researchers out there are listening to this all over the world, no pressure, but why should they get involved in peer review?

Dr Isabel Castanho:

Well, you touched on some of the points already on your introduction, but I would say that benefits for ECRs include gaining insights into other developments in your field, in a timely manner. Before they’re even published, which it’s important. Playing a greater role in the research community in general, because you’re helping improve the quality of the papers that are published. You are in a way also, I don’t want to say controlling where the research is going, but at least making sure that where it’s going is a good place. It’s a place of good quality research. It also helps to develop research and writing skills. And honestly it helps with your own work, because often you see things in papers, they’re like, wow, okay. I never thought of putting a figure like this before or explaining it this way. I think it really helps.

Adam Smith:

New ideas, right?

Dr Isabel Castanho:

Absolutely.

Adam Smith:

How many times have you just, you can nod, but then I’ll say how many? Do you take away ideas from papers and go, oh yeah, that’s a great. Martina does. Yvonne never gets ideas from papers. No, she does really. Okay. Thank you, Isabel. And of course, along with the show notes for this podcast, we will give some links to where we are aware of some peer review training that’s involved. I think Isabel, if that ARUK Southwest Network got that somewhere videoed, it’d be great to put up to our YouTube channel. Also, as well we do have a WhatsApp community group where you can join. And if you’ve written a paper or you’d like to be involved in review, or just want to offer out your services, I’m sure you could mention that in the WhatsApp group and people would get you involved.

Adam Smith:

I like the idea as a few of you mentioned about the idea of sitting down with somebody and doing this together. So, when you are finally in a position as you supervise others and you’re reviewing a paper, maybe invite that PhD student to come and sit with you while you go through and say, hey, I’ll do it in tandem. I think the problem is of course the confidentiality bit stops you doing that. Doesn’t it? Which is a shame, because you’d quite like to say you review this, I’m going to review your comments. That must be the one thing supervisors can’t delegate just to their research assistants and PhD students, is the peer review bit. Or do they anyway?

Dr Yvonne Couch:

Yes, I think they do. That’s exactly the problem. You can tell a PhD student has reviewed your work because it will be way more nit-picky than it needs to be. All big wig professors, you’re sending it to big wig professors you wish they were reviewing it, they’re not, they’re farming it out to anyone who is in their lab at that point. It should not be like that, but sadly I think it is.

Dr Isabel Castanho:

And it’s important that the students have the recognition. But on a note, I would say that some editors allow, if the professor discuss with the editor, sometimes they allow you to, if you explained you would like to share with the PhD student and review with them, some editors allow that. But on another note, it’s really important if that’s the case that the student gets credit for it.

Adam Smith:

I think you made a really good point there, Yvonne, is as an early career researcher, try to avoid being a bit nit-picky. Literally you don’t have to comment. Not everything needs feedback. Sometimes you can just let it go through. Right?

Dr Yvonne Couch:

Exactly. Not everything. It’s a paper, it does not necessarily need fixing. It might be great. So, you don’t have to go looking for mistakes or looking for errors.

Adam Smith:

Unless you’re looking for a way to make it better or to fix it.

Dr Yvonne Couch:

Exactly.

Adam Smith:

You can just leave it; you don’t have to comment. Okay. Very last. We’re way over time. Last point. We are not talking about what’s broken about peer review here today. We’re assuming peer review is awesome, which we know it isn’t. But I’m going to give you all one thing you can highlight, one thing about peer review that if you could fix it, what’s the one thing you’d to fix about peer review? And I’ve given you no time to prepare for this. I’m going to go to Martina first. What would you like to fix about peer review, Martina?

Dr Martina Bocchetta:

Well, two things come to mind. One is-

Adam Smith:

I gave you one thing.

Dr Martina Bocchetta:

Sorry. Sorry. Sorry.

Adam Smith:

All right. If they’re connected, we might let you. Go on.

Dr Martina Bocchetta:

One is incentive to people who actually review papers, and the other one is probably make reviewers accountable for what they are doing, so they stop trashing papers. That’s it for me.

Adam Smith:

Good points. Isabel.

Dr Isabel Castanho:

Okay. Proper recognition and at the same time accountability for bad reviews.

Adam Smith:

Okay. We’ve got a matched pair there. Yvonne.

Dr Yvonne Couch:

Thanks. You came to me last and now I’m all out of ideas, I’ve already sucked my juice today.

Adam Smith:

I know you’d have some new ones.

Dr Yvonne Couch:

No, I don’t. Because mine is basically reflective of Martina and Isabel’s. And it’s just, this will not work because everyone’s way too busy, but I think peer review should just be a meeting. I think it should be face-to-face chat.

Adam Smith:

Three people. Round-

Dr Isabel Castanho:

I think some journals are trying that. I’ve done a review recently where it asked me if the authors were interested in having a meeting with me, if I was interested in that actually.

Adam Smith:

That sounds more like a counseling, like arbitration. This is marriage counseling where [inaudible 00:59:58] you’d have the meeting if you’ve got to round three. Eventually say, look, we’re all just going to have to get around the table. Because it is quicker sometimes just to get around the table. Actually, maybe, would that be good? Would instead have written feedback, audio, notes, you can just talk about a paper and going, hey look, I really like this, but this section here confused me. This experiment will be good, but don’t feel obliged to do it. You could quickly talk. I bet you could cram all your peer review notes into a 10 minute. If I was listening to that from somebody, I’d like that.

Dr Yvonne Couch:

Or listen to your peer reviews at the gym.

Adam Smith:

There you go. Publishers out there, audio feedback in the format of peer review is something we’re endorsing here. Thank you very much, everybody. We really have come to the end of our time. We’re at an hour now. As a final set of takeaways, our advice would be to be specific, courteous, constructive. And then it should be understood, but you also need to maintain confidentiality. We haven’t talked about this a little much today but be timely. One of the biggest complaints in our ECR survey about the peer review process was how bloody long it takes and how sometimes papers can just take months and months and months. And it’s not always due to the peer review, sometimes the journals are just a bit slow, and a bit bogged down, but if you agree to do a peer review, make sure you do it on time and be timely. And if you don’t have time, just say you don’t have time.

Adam Smith:

And the best and final takeaway, do as you would be done unto. Avoid being rude, vague, smug, committed to particular issues, theories, or methodologies, and do a great job. And really do go out there and offer to get involved in peer review because it just makes everybody’s life a little bit easier if there are more people to spread the work around. Thank you very much. Again, the amazing Dr. Martina Bocchetta, the fabulous Dr. Isabel Castanho, still got it wrong, and the unstoppable Dr. Yvonne Couch. Thank you all so much everybody. Along with our show notes they’ll be all the resources we talked about today and everything you’ll need to help you prepare to go off and do it and be a peer review. We’ll see you again two weeks in. Thanks everybody. [inaudible 01:02:29]. Bye. Bye.

Voice Over:

Brought you by dementiaresearcher.nihr.ac.uk, in association with Alzheimer’s Research UK and Alzheimer’s Society, supporting early career Dementia Researchers across the world.

END