Guidance for reviewers
Step 1: Read the abstract
Ask yourself the following:
Can I provide an informed review?
You will often be faced with papers that fall outside your own specialised area, or papers within your area that describe techniques or analyses with which you are unfamiliar. This does not matter, provided you feel you can make a reasonably informed judgement about the paper – you will also learn something by doing so! If you are genuinely out of your depth, however, you should not agree to referee. Never try to comment on things you know little about, e.g. statistical analysis, but tell the Editor what your limits are.
Can I provide a fair review?
Conflict of interest is hard to judge. The simplest rule is to declare anything that might make you uncomfortable if it became public, e.g. that you are a close personal friend (or enemy) of the author, or a paid consultant for the company sponsoring the study. If you know that a fair review is impossible, decline the invitation. If there is a potential conflict but you think that you can nonetheless write a fair review, ask the Editor what to do. You may already have reviewed this paper for another journal. If so, you should inform the Editor. There is no automatic reason why you should not comment again, but it is important to use your judgement and try to be fair. It is very hard to be fair if you suggested a lot of changes last time round and the authors ignored everything you said. If so, you should not agree to review again, but the Editor would be interested to learn of your experience.
Step 2: Read the paper. Ask if it justifies full review
Common reasons for rapid rejection
- The paper is derivative. It simply repeats previous studies, or adds relatively little to existing knowledge.
- The paper is flawed.
- The paper is boring.
- The paper is incomprehensible.
- The authors have published the same material in 15 different journals.
Offer constructive advice whenever possible
If the paper falls into one of these categories, it is your job to explain this to the authors, and to offer constructive advice whenever possible. Your report need not be long. Some referees provide the Editor with a long confidential critique explaining the failings of the paper, and then write something nice to the authors. Please avoid this – the authors will only write back to challenge the rejection.
Step 3: Re-read the paper
A negative answer to an interesting question is more important than a positive answer to a boring question
Ask the following:
Is it important?
- Why was the study done?
- Is the question relevant and interesting?
- "So what?"
- Would you like to see this published?
- And remember - a negative answer to an interesting question is more important than a positive answer to a boring question.
Is it original?
- Remember that there are very few truly original papers (and these usually get rejected!). The question to ask is whether the study makes a useful contribution to knowledge.
- Confirmatory studies are useful and necessary, but only up to a point. This is where an Editor relies on your knowledge of the field.
- Check PubMed to see that the authors have not previously published similar data. Depressingly many have.
Is it valid?
- Is it primary research (experiment, RCT, cohort, case–control, cross-sectional, longitudinal, case report/series)?
- Is the design appropriate?
- Are the sample selection and size appropriate?
- Are the methods adequately described?
- Have standard guidelines been followed, e.g. CONSORT for clinical trials?
- Is the statistical analysis appropriate and comprehensible? Do we need a stats review?
- Are the conclusions compatible with what the authors actually describe?
- Is there evidence of systematic bias, e.g. in favour of the trial sponsor?
- Was the study ethical?
- Is it clear? Well-written? Well-argued? Well illustrated? Well referenced?
- Do the authors need linguistic help from an English speaker?
- Is it too long? Shorter papers are generally better.
- Have the authors structured their discussion? We recommend the BMJ format as follows:
- Statement of principal findings
- Strengths and weaknesses
- Relation to previous studies
- Meaning, explanations, implications
- Unanswered questions, future research
Step 4: Write your review: Author's section
Making papers better is the most important job of the referee
- There is no standard format for this. It is helpful for the Editor if you state what the study has shown and list its strengths and weaknesses.
- Number your points for ease of reference.
- Try to comment on importance, originality, validity and presentation.
- Always try to be specific. If you advise revision, state exactly what changes you would like to see, and what questions you would like answered. Support your argument with references where possible, ideally without mentioning your own work! Do not put unnecessary obstacles in the way of the author: “do as you would be done by”. Making papers better is the most important job of the referee.
- Be constructive. Even if the paper is likely to be rejected, your job is to help the authors as much as you can.
- Avoid comments about whether the paper should be accepted.
Step 5: Write your comments to the Editor
Remember, you make the recommendation, the Editor makes the decision! Your job is to guide her/him in the right direction.
This section should be brief; there is no need to repeat or extend your comments to the author.
You should always try to avoid making comments to the Editor that you would not make to the author. Exceptions arise where you are concerned about bias, plagiarism etc.