ABSTRACT
This paper provides reference for the implementation of open peer review (OPR) in Chinese scientific journals, so as to promote the openness of Chinese scientific journals in peer review and further conform to the spirit of scientific openness. The OPR process of PLoS One, Atmospheric Chemistry and Physics, F1000Research, the Semantic Web Journal, and Acta Psychologica Sinica was investigated. The five journals show many of the same OPR characteristics, but none of them implement OPR in the same way, which reflects the diversity of OPR practice process. Based on the actual situation, Chinese scientific journals should combine the advantages of open access and OPR with the advantages of traditional peer review flexibly and effectively, and optimize the implementation process of OPR.
Key words: scientific journal, open peer review, case, characteristic, diversity
Peer review is an important way of self-control in the global academic system, the "gold standard" of academic communication, and has a profound impact on the global academic community. Peer review usually refers to the evaluation by professionals (peers) of the quality, originality, theoretical and empirical reliability, and potential impact of a paper, conference, or manuscript. Traditional peer review, mostly in the form of blind reviews, has been used as a review process to assist in the allocation of limited resources. With the rapid development of the Internet and technology, the traditional peer review model has been questioned in academic publishing circles.[1–17] Traditional peer review cannot meet the needs of efficient communication and quality assurance. Besides lack of fairness, objectivity, transparency, and ethics,[1–6] a hidden review process often hinders or delays the dissemination of innovative ideas and concepts.[3,7,11] To solve the disadvantages of traditional peer review, open peer review (OPR) has emerged as a new form.
At present, there is no unified concept of OPR. McCormack[4] views OPR as a process that does not disguise the identity of author and reviewer. Ware[5] suggests OPR may mean that, as opposed to double-blind reviews, reviewers and authors know each other's identities and are open to the public. In practice, however, the definition of OPR becomes complicated because it can refer to openness at various stages of the review process. For example, some journal review experts are anonymous, but the review opinion is public. To illustrate the complexity of the OPR concept, OpenAIRE conducted a survey and identified 122 different definitions of OPR.[6] This study argues that OPR is a term proposed in the spirit of scientific openness compared with the traditional closed peer review, including the disclosure of the identity of the author and reviewer, the review opinion, the author's original manuscript and final revised manuscript, the review process, the disclosure of the interaction, the publishing platform, and so on. The signed review, disclosed review, editor-mediated review, transparent review, and public review in OPR reflect certain openness characteristics.
Some studies argue that OPR can make reviewers and authors more responsible for their own behaviors in the process of academic communication.[7–9] It is easier to identify academic misconduct[10] and the quality of submissions improves over time[11,12] and helps win academic credibility and respect for reviewers.[8,10,12–14] Furthermore, academic exchanges and dialogues can be enriched,[8,15–18] and new discoveries can be published and disseminated more quickly.[3,7,11]
In recent years, domestic scholars have also begun to study OPR. Zhang et al.[2] investigated the essence and review model of OPR. Wang[18] explored the feasibility of OPR. Zhang et al.[19] classified the OPR model as completely, limited, and partially open. Liu et al.[20] proposed four OPR modes. Liu[21] et al. analyzed the advantages and disadvantages of OPR. However, few domestic researchers have studied and introduced OPR cases. The current study examines the website of PLoS One, Atmospheric Chemistry and Physics (ACP), F1000Research, the Semantic Web Journal (SWJ), and Acta Psychologica Sinica in China,[22–26] combines related materials,[14,27–31] and compares the OPR process of these journals, providing reference for the practice, innovation, and reform of peer review of domestic science and technology (hereinafter referred to as sci-tech) journals.
OPEN PEER REVIEW CASE OF SCI-TECH JOURNAL
PLoS One
Published by the Public Library of Science, PLoS One is the world's first multidisciplinary open access (OA) journal. PLoS One articles are reviewed using a rigorous combination of pre- and post-publication public review. Every paper submitted to PLoS One must go through a rigorous quality control and peer review process.[22,27]
PLoS One currently uses an online peer review system provided by eJournal Press, to which all author submissions are submitted for initial quality checks to ensure that they meet the requirements specified in PLoS One. Initial internal quality control checks address issues such as competitive interests, study participants, or ethical requirements for animal research. The editorial board or peer-reviewed experts will not be able to see a submitted paper until it passes a quality control check. Once a paper passes such check, it is assigned to an academic editor, who invites peer reviewers to review the paper. The review opinions of the experts are detailed and comprehensive, providing constructive suggestions for the improvement of the academic quality of the paper. The academic editor will decide whether to accept or reject the manuscript according to the review opinions of the reviewers, or propose suggestions for modification. The author will modify the manuscript according to the requirements to ensure it meets the publication requirements of PLoS One. If necessary, the academic editor will also re-evaluate the revised manuscript. As a result, PLoS One's pre-publication peer review process is much the same as that of other journals.
The most prominent feature of PLoS One is the ability to provide post-publication reviews and article level indicators. PLoS One allows users to comment on each article without being anonymous, reviews must comply with the norms of scientific discourse, and any conflicts of interest must be declared. Article level indicators help to understand the impact of a study before actual academic citation. All the Public Library of Science (PLoS) journal articles present the PLoS article rating indicator, a quantifiable measure that includes both academic and social indicators that document how scientists and the public participate in the review of published articles in a variety of ways. The indicator is displayed on the rating label of each published article. In addition, the top right sign provides key metrics for citations, views, shares, and bookmarks.[28] As a result, PLoS One is not just a platform for digital papers, but also a potential place for discussion based on the topic context of the relevant papers. One of the benefits of PLoS One's process of documenting public discussion is that linking the comments section of the article to the media and recording full media coverage of the published article helps increase the journal's impact beyond the PLoS One platform and the academic community.[27]
The Semantic Web Journal
Published by IOS Press and Aka-Verlag, SWJ is a new professional journal dedicated to semantic Web research. SWJ combines prepress OA with an open and transparent review process. All submissions are published online and readers do not need to register to access them. In addition to being reviewed by experts invited by the SWJ Editorial Board, comments from researchers and the public can be uploaded to the journal website. The standard workflow for reviews is as follows:[23]
(1) Authors use the journal web site to submit papers. After preliminary screening, papers that clearly do not meet the journal's quality standards will be rejected.
(2) Publish the article on the journal website. The paper is published on the journal's blog and then assigned to the editor in the editorial department.
(3) The responsible editor invites three experts to review the paper, and the review is usually completed within six weeks. In addition, the responsible editor invites other researchers in the field for open comment.
(4) The comments of invited reviewers are usually published on the journal website together with their names and affiliated institutions (especially when there are large differences of opinions, the responsible editor will arrange a reviewer to write a short comment). Invited reviewers can also choose to publish their comments anonymously. Any other researcher may post non-anonymous reviews on the journal's website, along with comments from invited review experts and the public.
(5) The responsible editor shall discuss with other editors in the editorial department to decide whether to accept the paper.
(6) If the article is accepted, the identity of all reviewers and editors involved will be disclosed in the final printed version and shown in the title of the article.
(7) Authors are strongly encouraged to upload the final draft of their accepted paper on their local website and the printed version will be published through IOS Press.
Submissions that are not accepted may only exist for a limited time on their website, along with their comments. Although SWJ can provide preprint copies of manuscripts online, it is not the same as a traditional OA journal. The final printed edition can only be subscribed as a journal or PDF file through IOS Press.[29]
Atmospheric Chemistry and Physics
ACP, one of the top international journals, founded based on the concept of OPR and Public Peer Review & Interactive Public Discussion, is an OA journal dedicated to publishing and publicly discussing the investigation of the Earth's atmosphere and its potential chemical and physical processes. The ACP has achieved great success in the use of OPR and is rated by Thomson Reuters as the top journal in the field of Meteorology & Atmospheric Sciences.[30] A total of 11 journals under European Geosciences Union (EGU) Publishing Group, including ACP, adopt the publishing mode of open forum peer review and the two-stage peer review of the paper publication. ACP adopts the parallel system of OPR and traditional peer review, and its specific process is divided into two stages.[14,24]
(1) The journal editor reviews the paper quickly and briefly before publishing it in the journal's Atmospheric Chemistry and Physics Discussions (ACPD) forums for peer review and interactive public discussion, with interested members of the scientific community participating in interactive discussions and authors responding to comments. Comments from invited reviewers and a large number of public reviews are published. Invited reviewers can choose to sign or remain anonymous. Comments from other scientists (registered readers) are automatically signed. At the end of the 8-week discussion period, the review function of the paper is closed, and the journal editors make the final decision of acceptance or rejection based on the comments of the reviewers and the public comments.
(2) After the peer review process is completed, the final revised paper will be published on the ACP if it meets publication requirements. Both ACPD and ACP are registered on International Standard Serial Number (ISSN). To ensure priority of publication for authors and to provide a lasting record of scientific discussions, each discussion document and interactive commentary will be permanently archived and individually referenced.
F1000Research
F1000Research is an OA journal founded by F1000 (Faculty of 1000), which is an open research and publishing platform based on life science and medicine. Its publishing scope includes life science and medical case reports, reviews, letters, system evaluation, thought experiment and network tools, dissertations on clinical practice, data, research, and methods, and research plan.[25] For published articles, the journal uses a Creative Commons Attribution License, which anyone can download and use freely.
Peer-reviewed articles are indexed in PubMed, Scopus, and other databases. During the review process, some articles may receive a unanimous negative review, in which case the article remains "published" and is only removed from the default search site. The icon notes on the site interface clearly indicate the status of pre-approved, approved, or unapproved review. F1000Research does not consider unapproved articles as accepted revisions, and the rejected status is visible in the review process.
Acta Psychologica Sinica
Sponsored by the Chinese Psychological Society and the Institute of Psychology of Chinese Academy of Sciences and published by Science Press, Acta Psychologica Sinica mainly publishes the latest and highest level of psychological scientific papers in China.[26] Acta Psychologica Sinica is a typical representative of the exploration and practice of OPR in Chinese sci-tech journals, but the process is very tortuous.
In the face of the heated discussion about OPR in the scientific publishing circle, Acta Psychologica Sinica also considered trying OPR on a voluntary basis of the author, and planned to set up an open comment column. When the plan of open review was submitted to the editorial board for discussion, it was opposed by most of the editorial board, and finally it had to adopt the flexible way of open review. Since 2014, Acta Psychologica Sinica has adopted the anonymous public review method. All previous review opinions and the author responses are organized into a separate PDF document, and there is a "review" link behind the online early edition of the journal article. The "review opinions and the author response" can easily be viewed by clicking on the links. Acta Psychologica Sinica takes the step of publicizing the review after traditional peer review, which is not strictly OPR. Objectively, this is a middle ground between traditional peer review and OPR, but Acta Psychologica Sinica has taken a step toward OPR.[31]
DISCUSSION
A comparison of OPR characteristics of five journals and the relative openness of OPR process is presented in Table 1. It can be seen that these five journals show many of the same OPR characteristics, but they do not implement OPR in the same way, which reflects the diversity of OPR practice process. PLoS One uses a combination of rigorous pre-publication peer review and post-publication public review. SWJ combines prepress OA with an open and transparent review process. ACP adopts the parallel system of OPR and traditional peer review. F1000Research adopts peer reviewed after publication. In Acta Psychologica Sinica, the review method is anonymous and public. The OPR of the five journals has the following characteristics:
| Characteristics | PLoS One | SWJ | ACP | F1000Research | Acta Psychologica Sinica |
| Signed review | Optional, but positive encouragement | Optional, but positive encouragement | Optional, but positive encouragement | Yes | No, anonymous comments |
| Disclosed review | Only if reviews are signed | Only if reviews are signed | Only if reviews are signed | Yes | No, only comments will be published |
| Editor-mediated review | Yes | Yes | Yes | Yes | Yes |
| Transparent review | No, proponents' comments and authors' responses are not published undefined | Only if reviews are signed | Only if reviews are signed | Yes | No |
| Public review | Yes, but after publication | Yes | Yes | Yes | No |
| Pre-publication review | Yes | Yes | Yes | Yes | Yes |
| Post-publication review | Yes | No | No | Yes | No |
| Synchronous review | No | The article is published once it is posted on the website | No | No | No |
| Whether OA | Yes | Not typical OA, only published before OA | Yes | Yes | No |
(1) All the five journals show the characteristics of preliminary review and mediation. Before the article is reviewed, the editor carries out a preliminary check on the quality, scope, and compliance with the requirements of the journal. In the process of article review, the editor's adjustment promotes the smooth progress of OPR.
(2) The five journals all show certain openness in signed review, disclosed review and transparent review. F1000Research performs best in this step. The identities of authors and reviewers are all open, which is a real open review. The Acta Psychologica Sinica publishes the opinions, but the reviewers are anonymous. PLoS One, ACP, and SWJ strongly encourage the reviewers to sign, but signatures are optional. Is it the OPR when experts can choose whether to disclose the identity of the review? How to define OPR? There is no consensus. According to the definition given by Ware,[5] a review without disclosing the identity of the reviewer is not a real OPR, but the peer review of these journals also contains certain characteristics of openness. For example, the review experts of Acta Psychologica Sinica are anonymous, but the review opinions are open. The implementation process of OPR is complicated. The author believes that OPR is a concept proposed based on the openness of science. As long as it has an open spirit and includes an open evaluation process, it can be considered as OPR.
The anonymity of reviewers makes it easier for them to raise critical opinions and questions, and they may be reluctant to risk involvement in disputes while volunteering to serve the academic community.[1] It may also be that publishers have noticed that the acceptance of OPR publishing practices varies from discipline to discipline, so signatures are optional.
(3) Apart from Acta Psychologica Sinica in China, PLoS One, ACP, F1000Research, and SWJ in foreign countries all have different forms of public comment, but the time of public comment is different. Public comment in ACP and SWJ take place in the discussion stage before publication, while public comment in PLoS One and F1000Research occur after publication, but with slight differences. Public review for PLoS One takes place after peer review, while that for F1000Research occurs simultaneously with public review. Although public review is a feature of OPR, it may be carried out differently in peer review processes in different journals.[27]
Does the public comment take place before or after publication? The author thinks it is better to review before publication. This is because, if the article has been accepted for publication, the public may not be motivated to comment. Some constructive comments put forward before publication are beneficial to the further modification and perfection of the article.
(4) F1000Research will put the article in the published state within one week after receiving the manuscript. If a unanimous negative opinion is obtained, the published article will be deleted from its default search, and the "unapproved" status will be displayed in the review procedure. Do "unapproved" articles in F1000Research count as published? How to define the published status? Periodical circles should give a clear definition. The author does not consider "unapproved" articles as published articles because they have not been peer reviewed.
(5) PLoS One, ACP, and F1000Research are all OA journals, while SWJ has the characteristics of OA only in the review stage of preprint. The OA publishing model itself contains the spirit of scientific openness, so it is easier to implement OPR in OA journals than in non-OA journals, and authors and reviewers of the latter may have certain resistance to OPR.
(6) In the process of OPR implementation, there is still a certain gap in the openness of Acta Psychologica Sinica compared with the four foreign journals, which to some extent reflects that there is still a certain gap in the openness of Chinese journals compared with foreign journals in the process of article publishing. In particular, further efforts are needed in the public comment section of the article. Foreign journals combine public review with designated expert review to carry out interactive discussion among authors, review experts, the public, and editors, which attracts more reviews, improves efficiency and transparency, and effectively solves the problem between rapid scientific communication and total quality assurance.
SUMMARY
As a new form of peer review, OPR is still immature, and even the definition and mode of OPR have not been unified in academic circles. However, its free, open, transparent, interactive, efficient, and sharing characteristics promote its rapid development in foreign countries, and it has been implemented in journals of many disciplines. Although OPR started late in China, the research on OPR is increasing day by day. Through the study of the OPR process in the PLoS One, ACP, F1000Research, SWJ, and Acta Psychologica Sinica in China, the author found that these five journals show many similar OPR characteristics, but do not implement OPR in the same way, which reflects the diversity of OPR practice process and inspires us to flexibly and effectively combine the advantages of OA and OPR with that of traditional peer review, and implement OPR flexibly according to the actual situation of each journal. The anonymous public evaluation method of Acta Psychologica Sinica is carefully chosen according to the actual situation. It is an important topic to further explore a more effective and local way of OPR, combining China's national conditions and social and cultural atmosphere.
DECLARATIONS
Secondary publication declaration
This article was translated with permission from the Chinese language version first published by Chinese Journal of Scientific and Technical Periodicals. The original publication is detailed as: Wang FC. [Case study of open peer review in scientific journals]. Chin J Sci Tech Period. 2018;9(3):242–247.
Source of funding
This article is supported by the National Research Project Cultivation Fund (5102109171301).
Author contributions
Fengchan Wang: Conceptualization, Writing—Original draft preparation, Writing Reviewing and Editing.
Conflict of interest
The author has no conflicts of interest to declare.
REFERENCES
- Pöschl U. Multi-stage open peer review: scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation. Front Comput Neurosci. 2012;6:33. DOI: 10.3389/fncom.2012.00033 PMID: 22783183
- Zhang XY, Luo P. [The essence of open peer review of academic journals in the Web 3.0 era and the construction of review model]. Acta Editol. 2016;28(3):220–223. DOI: 10.16811/j.cnki.1001-4314.2016.03.005
- Pöschl U. Interactive journal concept for improved scientific publishing and quality assurance. Learn Publ. 2004;17(2):105–113. DOI: 10.1087/095315104322958481
- McCormack N. Peer review and legal publishing: what law librarians need to know about open, single-blind, and double-blind reviewing. Law Libr J. 2009;101(1):59–70.
- Ware M. Peer review: recent experience and future directions. New Rev Inf Netw. 2011;16(1):23–53. DOI: 10.1080/13614576.2011.566812
- Ross-Hellauer T. What is open peer review? F1000Res. 2017;6:588. DOI: 10.12688/f1000research.11369.2 PMID: 28580134
- Kriegeskorte N. Open evaluation: a vision for entirely transparent post-publication peer review and rating for science. Front Comput Neurosci. 2012;6:79. DOI: 10.3389/fncom.2012.00079 PMID: 23087639
- Fitzpatrick K. Peer-to-peer review and the future of scholarly authority. Soc Epistemol. 2010;24(3):161–179. DOI: 10.1080/02691728.2010.498929
- Mulligan A, Akerman R, Granier B, Tamber PS, Pöschl U. Quality, certification and peer review. Inf Serv Use. 2008;28(3–4):197–214. DOI: 10.3233/isu-2008-0582
- Boldt A. Extending ArXiv.org to achieve open peer review and publishing. J Sch Publ. 2011;42(2):238–242. DOI: 10.48550/arXiv.1011.6590
- Hu C, Zhang Y, Chen G. Exploring a new model for preprint server: a case study of CSPO. J Acad Librariansh. 2010;36(3):257–262. DOI: 10.1016/j.acalib.2010.03.010
- Prug T. Open-process academic publishing. Ephemera Theory Polit Organ. 2010;10(1):40–63.
- Bornmann L, Daniel HD. Reliability of reviewers' ratings when using public peer review: a case study. Learn Publ. 2010;23(2):124–131. DOI: 10.1087/20100207
- Poschl U. Interactive open access peer review: the atmospheric chemistry and physics model. Against Grain. 2009;21(3):26–32. DOI: 10.7771/2380-176x.2311
- Friedman R, Whitworth B, Brownstein M. Realizing the power of extelligence: a new business model for academic publishing. Int J Technol Knowl Soc. 2010;6(2):105–118.
- Lipworth W, Kerridge IH, Carter SM, Little M. Should biomedical publishing be "opened up"? Bioethical Inquiry. 2011;8(3):267–280. DOI: 10.1007/s11673-011-9312-4
- Maharg P, Duncan N. Black box, Pandora's box or virtual toolbox? Int Rev Law Comput Technol. 2007;21(2):109–128. DOI: 10.1080/13600860701492104
- Wang FC. [Research on the feasibility of open peer review of sci-tech journals]. Chin J Sci Tech Period. 2018;9(1):14–19. DOI: 10.11946/cjstp.201707230615
- Zhang CL, Shang LN, Ni SX. [Exploration of the open peer review model of sci-tech journals]. Chin J Sci Tech Period. 2015;26(11):1151–1155. DOI: 10.11946/cjstp.201507090620
- Liu CL, He QC. [The production, development, effectiveness and feasibility of open peer review]. Chin J Sci Tech Period. 2013;24(1):40–44.
- Liu LP, Liu CL. [Analysis and suggestions on the advantages and disadvantages of open peer review]. Chin J Sci Tech Period. 2017;28(5):389–395. DOI: 10.11946/cjstp.201704270323
- Journal Information. PLoS One. Accessed September 1, 2017. https://journals.plos.org/plosone/s/journal-information
- About the semantic web journal. Semantic Web—Interoperability, Usability, Applicability an IOS Press Journal. Accessed September 1, 2017. http://www.semantic-web-journal.net/
- Aims and scope. Atmospheric Chemistry and Physics. Accessed September 1, 2017. https://www.atmospheric-chemistry-and-physics.net/about/aims_and_scope.html
- How it works. F1000Research. Accessed September 1, 2017. https://F1000Research.com/about
- [Introduction to the journal]. Acta Psychologica Sinica. Accessed September 1, 2017. https://journal.psych.ac.cn/xlxb/CN/column/column1.shtml
- Ford E. Open peer review at four STEM journals: an observational overview. F1000Res. 2015;4:6. DOI: 10.12688/f1000research.6005.2 PMID: 25767695
- Binfield P. : new approaches and initiatives in the evolution of the academic journal. Against the Grain. 2009;21(3):9. DOI: 10.7771/2380-176X.2309
- Janowicz K, Hitzler P. Open and transparent: the review process of the Semantic Web journal. Learn Publ. 2012;25(1):48–55. DOI: 10.1087/20120107
- Xuexin. [Using public review manuscripts to promote the rapid development of Chinese sci-tech journals]. Psychol Dev Educ. 2013;29(1):109–111. DOI: 10.11946/cjstp.201409250929
- Li JZ, Zhuang JC, Qiu BW. [Trail open peer review conducted by. Acta Psychologic Sinica. ]. Chin J Sci Tech Period. 2015;26(2):139–142. DOI: 10.11946/cjstp.201409250929
