Open peer review could result in better quality of peer review

The study, published in the open access journal BMJ Open, also found that reviewers suggested by authors were more likely to recommend acceptance than those who were chosen through other means.

A judgment on the scientific validity of a research paper is usually based on the recommendations of two or more experts in the field who independently assess the scientific claims being made. To investigate the quality of peer review reports under different models, two journals from BioMed Central's BMC series were compared -- BMC Infectious Diseases, which operates under the open peer review model and BMC Microbiology, which operates under the single-blind model.

Under open peer review, all parties know the identities of those involved, reviewer reports are signed and, if the manuscript is accepted, reviewer reports accompany publication. Under single-blind closed peer review the reviewers are anonymous and the reports are not made public.

The two BMC series journals differ in their peer review models but are otherwise similar in that all editorial processes are the same, they have a similar acceptance rate and threshold, and cover similar topics. A third journal was also investigated, Journal of Inflammation, because it transitioned from open peer review to single-blind peer review, allowing further investigation into the effects of changing the peer review process on the quality of the reports in a single journal.

For the BMC series journals, 200 reviewer reports from each journal were compared to each other. Each reviewer report was independently assessed for quality by two members of BioMed Central's editorial team using the Review Quality Instrument, an established tool for this type of assessment. This involved rating each peer review report on elements such as discussion of the research question, identifying strengths and weaknesses in methods and constructiveness of reviewer comments.

Following a statistical analysis of all the ratings for BMC Infectious Diseases and BMC Microbiology, it was found that there was a 5% improvement in the quality of the peer review for reports provided under the open peer review model, a statistically significant result.

A similar rating study was carried out for Journal of Inflammation comparing 200 reports provided during three years while the journal operated under open peer review and 200 reports from the three years after the change to single-blind peer review. However, this did not reveal a significant difference in reviewer report quality in this case, which the authors speculate may be because the result of other variables, such as a change of editorship.

Data from surveys of the authors of the BMC series journals that formed the basis of this study show that the authors found comments from open peer review more helpful than those from single-blind peer review. The authors in the open peer review journal BMC Infectious Diseases rated most other aspects of the editorial process more favorably too. For the Journal of Inflammation, the research did not find statistically significant differences between the surveys returned by the authors of manuscripts that underwent open or single-blind peer review, though that may be due to the low number of available survey responses.

The research also found that, in all three journals, author-suggested reviewers tended to recommend acceptance of the manuscript more often than non-author suggested reviewers. Recommendations by non-author suggested reviewers were found to be a better predictor of whether or not a manuscript was accepted than author-suggested reviewers. The authors suggest that this would seem to indicate that editors place more weight on these reviews, be it unconsciously or consciously.

Maria Kowalczuk, lead author, BioMed Central's Biology Editor for the Research Integrity Group and co-Editor-in-Chief of Research Integrity and Peer Review, said: "As advocates of openness, we are excited to find that upon analysis reviewer reports under open peer review are of comparable, or even higher, quality than those of the more established model of single-blind peer review. However, we appreciate our results do not undermine the single-blind model of peer review. While we also found that the quality of reports written by author-suggested reviewers is similar to other reviewers, they tend to recommend acceptance more frequently. "