ABSTRACT
The academic quality of scientific journal manuscripts is integral to the peer review system of journals. To qualitatively enhance the review process of submitted medical manuscripts, this pilot study proposes a semi-structured review model that integrates semi-structured interviews based on the manuscript content with the traditional free-form review process. At the initial review, the associate editor should ask targeted questions to the review experts who are expected to respond openly. The findings of this pilot exploration of the semi-structured review model are summarized according to the introduction, methods, results, and discussion sections of the manuscripts. Strengthening the communication between the associate editor's initial review and the expert's peer review enhances the quality of academic review for medical papers. This approach may also serve as a pivotal safeguard against paper mills.
Key words: initial review, semi-structured peer review, mill paper
INTRODUCTION
Peer review is the lifeblood of a journal's sustainable development. It is an institutionalized quality control mechanism that acts as a "reputation brake" within the knowledge-value chain; without it, adverse selection and erosion of progressive credibility would precipitate a market-for-lemons collapse of the academic publishing. In 1665, Philosophical Transactions pioneered external refereeing, yet the phrase "peer review" itself did not emerge until the early 1970s.[1,2] The journal's seventeenth-century process undoubtedly bore little resemblance to contemporary peer review practices. Hence, in the ensuing decades, the academic community continues to explore more equitable peer review models—single-blind, double-blind, and even triple-blind—to foster a fairer and more scholarly publishing industry.[3] In the open-access (OA) era, a few of the leading OA journals have begun piloting open peer review models with full transparency.[4] The type of peer review models (e.g., single-blind, double-blind, or open) relies on the journal's specific aims and scope as well as impact factor.[5]
Recent studies have explored structured peer review models. A 2024 pilot study initiated by Malicki et al.[6] adopted a nine-question template to standardize reviewer assessments that achieved a consensus rate of 31% to 41% and prompted reviewers to address more quality-related dimensions. While reviewer participation and transparency improved markedly, agreement on methodological and interpretive issues remained limited. As Mario Malicki wrote in Nature,[7] "quality control of scientific literature should be as openly standardized as aircraft safety checks; otherwise, unreliable research will slip through like hidden hazards". Theoretically, since both the abstracts and the main texts of academic papers are now well structured, they align with a structured external peer review.[8] However, in practice, this model fails to sustain adherence among peer reviewers. The format is initially acceptable, but after repeated use, reviewers tend to skip answering the questions and revert to unconstrained traditional reviewing. At the same time, this model only requires a peer reviewer and does not make any demands on the editor.
The structured peer review model offers many theoretical advantages. However, since every manuscript, regardless of article type, follows the same template, complacency has become inevitable. A further innovation in the model is needed: invite medical editors into the peer review process to tailor a bespoke review for every single manuscript. Semi-structured interviews are considered a primary method for qualitative research,[9] in that they elicit key information from interviewees. This format blends the strengths of structured and unstructured interviews: it can soften pointed questions, provides respondents room to elaborate, and still allows the interviewer to keep control of the conversation. In this study, conventional peer review was integrated with a semi-structured interview model: the questions raised by the associate editor during the initial screening were forwarded to the peer reviewers in a question-and-answer format that mimicked a semi-structured interview. The experts responded to the associate editor's queries and reported any other flaws that were not flagged. This approach is referred to as "semi-structured peer review" with the following three objectives: serves as an initial editorial screen to flag potential mill papers, avoids collusion between editors and authors resulting in the publication of a mill paper, and prompts peer reviewers toward more substantive and comprehensive review comments.
DESIGN AND IMPLEMENTATION OF THE SEMI-STRUCTURED PEER REVIEW METHOD
The current editorial review process is initiated by the handling editor's initial review of new submissions. In this study, the inclusion criteria were original clinical research articles following the introduction, methods, results, discussion (IMRAD) structure with references. The exclusion criteria were poorly written manuscripts, incomplete formatting, and a high similarity index (> 20%). For manuscripts scheduled for external peer review, the journal's review guidelines provided a set of tailored, semi-structured review questions based on the manuscript's content.
The semi-structured review form contained three parts: (1) Instructions were provided for completing the semi-structured review. (2) Potential weaknesses identified by the associate editor were organized under IMRAD sections (these points awaited open-ended responses from the reviewers). Reviewers could list any other issues they wished the authors to address in an additional blank section. During the initial review, associate editors could consult the EQUATOR checklists. Based on the items in those checklists, they should query external peer reviewers about any potential weaknesses identified in the manuscript. (3) Two scoring items were perceived publication value (0 = lowest, 5 = highest) and acceptability of the semi-structured review format (0 = lowest, 5 = highest). The form was inserted in a prominent position within the invitation email so that external peer reviewers could focus their assessment accordingly.
SEMI-STRUCTURED REVIEW ACCEPTANCE DEGREE
Manuscripts were submitted for peer review during the period from September 2024 to January 2025. By May 31, 2025, 34 manuscripts had completed the full review cycle. A total of 57 invitations were issued, and 38 reviews that met the inclusion criteria were returned (response rate, 55.3%); 19 reviewers did not respond. Of the 38 returned reviews, 21 adopted the semi-structured template, whereas 17 followed the traditional free-form format. After author revisions guided by the external comments, 18 manuscripts were accepted and 16 were rejected, yielding an acceptance rate of 47.4%. The reviewers' acceptability score for the semi-structured format (mean ± standard deviation [SD]: 3.48 ± 1.50) and their rating of publication value (2.62 ± 1.50) were strongly and positively correlated (r = 0.794, P < 0.001; Figure 1).
Figure 1. Scatterplot of publication value vs. acceptance (n = 21).
THE INTRODUCTION SECTION OF THE PAPER
The statement in the introduction helps journal associate editors and peer reviewers judge the significance of the submission.[10] To be considered a good paper, it must, in its introduction, pose the problem to be solved, describe the research background of the topic, review relevant studies, accurately explain what issue the present work will address and how it will do so, and clarify the study's importance and innovation.[11]
In addition, certain study types require further questions. For investigations of novel drugs (e.g., "Tocilizumab for massive pericardial effusion after haematopoietic stem-cell transplantation: A two-case series"), reviewers should be asked whether the use of tocilizumab for pericardial effusion constitutes off-label prescribing; for rare-disease reports (e.g., "Langerhans-cell histiocytosis initially presenting as sclerosing cholangitis: A six-case clinical analysis"), the disease's incidence and whether a series of six patients is sufficiently informative; and for studies evaluating new technologies (e.g., "a prospective observational study of mNGS-guided therapy for neonatal infection by rare pathogens"), whether mNGS is already standard for neonates with infection of unknown origin or whether its use risks overtreatment.
THE METHOD SECTION OF THE PAPER
In a clinical research paper, the reliability and validity of the results depend on the methodological design of the study and on the collection, management, and analysis of the data throughout the research process.[12]
Design of research methodology
Clinical research is broadly categorized as observational studies (e.g., cohort, case-control, cross-sectional) or experimental studies (e.g., randomized controlled trials, non-randomized controlled trials, diagnostic accuracy studies). For each design, reviewers determine the appropriateness of the design, the feasibility of its execution, or the reproducibility of the methods.
In the paper "exploring the value of metformin in adolescents with T1DM using a continuous glucose-monitoring system", reviewers should be asked whether a retrospective study that nevertheless involved an intervention ought to be classified as a clinical trial. In "a prospective observational study of mNGS-guided therapy for neonatal infection by rare pathogens", reviewers should assess whether the prospective design is sound and whether the study was in fact conducted prospectively. In "application of the subjective global nutritional assessment (SGNA) in hospitalized children with neurological impairment", reviewers should evaluate whether it was feasible to perform two assessments in 1466 children.
Collection, organization, and analysis of research data
This section focuses on issues related to sample size, quality-control procedures during the study, and the statistical analysis. In "Clinical efficacy of the AVDC/ICE regimen in ten children with extracranial malignant rhabdoid tumors", reviewers should determine whether a single-arm series of only ten cases provides sufficient evidence. In "Outpatient status and related factors of language-delay among preschool children in Haidian District, 2020-2022", convenience sampling was used; reviewers should be asked whether this design introduces selection bias. In "Clinical characteristics of extremely preterm infants conceived by assisted reproductive technology", reviewers should be asked whether propensity-score matching was necessary, whether the authors clearly described the matching procedure, and whether the results remain reliable after the exclusion of 31 assisted reproductive technology (ART) infants whose data were missing.
Other potential weaknesses in the methods section
The methods section should also address whether the selection criteria for enrolling patients are adequate, including the eligibility criteria at recruitment, explicit exclusion criteria, and any potential withdrawal/dropout criteria applied during the study. Additional points to examine are the possibility of overtreatment, underlying bioethical concerns, declared or undeclared conflicts of interest, healthcare costs, and the objectivity of the outcome measures.
THE RESULT SECTION OF THE PAPER
In a clinical research paper, the results section presents the measured values and the results of the statistical analyses, as demonstrated by figures and/or tables.[13] External peer reviewers most closely scrutinize the clinical implications of the results, yet associate editors should still prompt them with two routine checks: (1) Were all reported results already fully specified in the methods? (2) Are the figures truly representative of the results (when applicable)? For any manuscript suspected of ghostwriting, reviewers must also be explicitly asked to verify whether the data are authentic.
Certain special situations warrant further queries. For example, in the paper "A prospective observational study of mNGS-guided therapy for neonatal infection by rare pathogens", reviewers should be asked whether the reported positive rate is consistent with real-world clinical experience and whether the infants finally included truly met the inclusion/exclusion criteria stated in the methods. In "Clinical features and influencing factors of diabetic ketoacidosis in children", reviewers should assess whether variables already used for grouping in the methods (e.g., blood-glucose levels) need to be compared again in the results. If associate editors suspect statistical errors, they may also ask reviewers whether an additional statistical expert should be invited.
THE CONCLUSION SECTION OF THE PAPER
Several researchers have advocated that the discussion section should also be written in a structured format.[14] An example of a five-point structured discussion template comprises the following points: (1) Restate the study's principal finding in one sentence. (2) Summarize the study's key strengths and limitations. (3) Compare the present results with previous work, explicitly addressing any discrepancies and clarifying where this study holds advantages or disadvantages. (4) Highlight the broader significance by outlining the plausible mechanisms and discussing the potential impact for clinicians or policy-makers. (5) Identify the questions that remain unanswered and propose directions for future research.
In editorial practice, it is common to receive manuscripts whose discussion drifts far from the actual findings while indulging in a lengthy, unfocused narrative. In our semi-structured review form, the single most frequently posed question was "does the discussion remain tightly anchored to the present study's results, and are the key findings explicitly analysed within it?"
DISCUSSION
This study proposes a semi-structured peer review paradigm using a semi-structured interview model integrated with the traditional external peer review process. Results demonstrate that external peer reviewers generally did not reject this model: On a 0-5 scale, their acceptance averages 4-5. The acceptance scores were also positively correlated with the publication value of the manuscript. In other words, peer reviewers are markedly more willing to invest the extra effort required by a semi-structured review when the manuscript is judged to be of high publication value; for manuscripts deemed to have low value, they prefer a rapid, straightforward review leading to an immediate recommendation to reject.
Of the 34 manuscripts submitted for external peer review in this study, all undergo an initial review to screen low-quality papers. However, the acceptance rate of manuscripts that proceed to external peer review remains below 50%. The semi-structured peer review model places even heavier demands on associate editors: Not every manuscript is suited to this semi-structured review model. Before opting for a semi-structured review, an associate editor must read the paper in its entirety—carefully, critically, and with full conceptual engagement—so that every argument, dataset, and nuance is thoroughly understood before deciding whether the manuscript proceeds to a semi-structured external review.
In 2015, the largest OA publisher, BioMed Central (BMC), announced the retraction of the publication of 43 papers.[15] BMC's fake peer review scandal was not the first of its kind, but its industrial scale made it the hallmark case. Leading publishers abolished author-suggested reviewers and mandated the use of open researcher and contributor identifier (ORCID) plus institutional e-mail verification—yet the fraud was not eradicated; it simply went underground and grew more sophisticated. Some agents, paper mills, have moved into manipulating publication, lured by large, low-risk profits. They bribe journal editors and pose as early-career scholars to secure slots as guest editors or editorial board members, in order to funnel their manuscripts through in bulk.[16] Even editors at well-known journals such as PLoS One have been implicated, thereby rendering their published articles questionable.[17] Paper mills have upgraded their tactics and become more covert in order to get papers published.[18] Russian academic researcher Anna Abalkina admitted in an interview, "Simply ensuring that peer review is robust would weed out most paper-mill products".[19] However, peer review alone is not a firewall, as reviewers see only a handful of manuscripts and rarely acquire the breadth of experience needed to spot mill products. Merging an editor's daily exposure and the topical expertise of reviewers might be effective in detecting most paper mill manuscripts. Too often, the hallmarks of a mill paper are obvious to any editor who takes the time to read the manuscript carefully.[20] Compared with Malicki's structured peer review[6] format that uses a fixed nine-question template, a semi-structured peer review model allows reviewers to formulate queries individually for each manuscript, resulting in a personalized set of questions that reflect the paper's specific content and context. Implementing a semi-structured review process would, through policy, oblige editors to scrutinize a manuscript thoroughly before it is sent out for peer review. This erects a first barrier against paper mill manuscripts and prevents editors from shifting responsibility onto external reviewers. Meanwhile, a semi-structured questionnaire likewise compels reviewers to examine the manuscript carefully in order to answer each item accurately, thereby deterring hastily composed or perfunctory review reports. Moreover, how personalized initial review questions are formulated can reveal how well the editor carefully examined the paper. To some extent, this can help prevent covert collusion between the editor and the agent of paper mills.
Enhancing the openness, transparency, and reproducibility of academic papers primarily involves four aspects:[21] Transparency in research design, standardization of research materials, data sharing, and standardization of analysis methods, all of which are focused on the methods section of the paper. The semi-structured review model places certain demands on associate editors who need to master the writing conventions for various types of clinical research, especially the methods section. Consequently, some challenges in the implementation process remain. Before conducting a semi-structured peer review, the quality of the articles should be fully assessed, and high-quality manuscripts with publication potential are selected for further review. The academic level of the associate editor is inevitably tested; the handling editor needs to master various academic writing standards and bear medical knowledge before carrying out semi-structured peer review, read the submissions carefully, and ask targeted questions to avoid raising templated questions for each article.
Limitation
This study had limitations. This was a pilot exploration of the proposed semi-structured peer review model on submitted manuscripts for medical journal publication. However, there were various forms of research papers such as pre-clinical research, systematic reviews, health economic evaluations, and case reports. As this is a pilot study, the experience with this semi-structured review model was limited, and therefore, the model could only be implemented on a small scale. Future investigations should consider increasing the number of manuscripts subjected to peer review and diversifying the types of manuscripts for review, so as to refine the semi-structured review model step by step. Moreover, the number of editors should also increase, so as to further explore and obtain additional quantitative data. Lastly, feasibility assessments should be added in order to broaden the spectrum of article types, enlarge sample sizes, and incorporate other journals. Handling submissions can be extremely time-consuming for early-career editors.[22] Rapidly accommodating the diverse publication types in the field and spotting their potential weaknesses may still pose challenges for early-career editors. The follow-up study will therefore develop a set of adaptive templates tailored to different publication types, enabling early-career editors to initiate semi-structured peer reviews more promptly.
CONCLUSION
Semi-structured peer review can address some of the shortcomings in the initial review by associate editors and the peer review by experts, thereby raising the quality of the entire evaluation process of submitted manuscripts. It may also serve as a pivotal safeguard against paper mills. In today's academic climate where paper mills are rampant, every medical editor must actively shift roles to become a review editor[23] who oversees peer review and screens out paper mills. We are in the era of the paper glut, where quantity is rewarded over quality.[24] The involution of quantity has brought no benefit to the research ecosystem. Editors-in-chief must take the lead in slowing down the pace, guiding associate editors to scrutinize and cull paper mills rigorously during the initial review stage and to restore a healthier scientific environment.
DECLARATIONS
Acknowledgement
The author thanks the 21 anonymous peer reviewers who conducted the semi-structured review for this study.
Author contributions
Zou Q contributed solely to the article.
Source of funding
This study was supported by Research Fund (No. SHGX2024C17) of the Science and Technology Journals Committee of the Shanghai Higher Education Society.
Ethical approval
Not applicable.
Informed consent
Not applicable.
Conflict of interest
The author has no conflicts of interest to declare.
Use of large language models, AI and machine learning tools
During the preparation of this work the author used Kimi (www.kimi.com) to polish the language of the manuscript. After using this service the author reviewed and edited the content as needed and take full responsibility for the content of the published article.
Data availability statement
Data used to support the findings of this study are available from the corresponding author upon request.
REFERENCES
- Kronick DA. Peer review in 18th-century scientific journalism. JAMA. 1990;263(10):1321-1322. PMID: 2406469
- Moxham N, Fyfe A. The royal society and the prehistory of peer review, 1665-1965. Hist J. 2018;61(4):863-889.
- Bazi T. Peer review: Single-blind, double-blind, or all the way-blind? Int Urogynecol J. 2020;31(3):481-483. DOI: 10.1007/s00192-019-04187-2 PMID: 31820012
- Ford E. Open peer review at four STEM journals: An observational overview. F1000Res. 2015;4:6. DOI: 10.12688/f1000research.6005.2 PMID: 25767695
- Faggion CM Jr. Peer review blinding practices of highly ranked dental journals: Analysis and discussion. Br Dent J. 2021;231(4):219-223. DOI: 10.1038/s41415-021-3319-y PMID: 34446840
- Malički M, Mehmani B. Structured peer review: Pilot results from 23 Elsevier journals. PeerJ. 2024;12:e17514.
- Malički M. Structure peer review to make it more robust. Nature. 2024;631(8008):483. DOI: 10.1038/d41586-024-01101-9 PMID: 38627512
- Zhu D. [On structured peer reviewing]. Acta Editologica. 2013;25(6):135-139.
- Adams W. Conducting semi-structured interviews. In: Newcomer KE, Hatry HP, Wholey JS, eds. Handbook of practical program evaluation. Wiley; 2015.
- Van Damme H. Steps to writing an effective introduction. Acta Chir Belg. 2015;115:1. PMID: 26466390
- Fried T, Foltz C, Lendner M, Vaccaro AR. How to write an effective introduction. Clin Spine Surg. 2019;32(3):111-112. DOI: 10.1097/BSD.0000000000000714 PMID: 30234565
- Kiani AK, Naureen Z, Pheby D, et al. Methodology for clinical research. J Prev Med Hyg. 2022;63(2 Suppl 3):E267-E278. DOI: 10.15167/2421-4248/jpmh2022.63.2S3.2769 PMID: 36479476
- Bahadoran Z, Mirmiran P, Zadeh-Vakili A, Hosseinpanah F, Ghasemi A. The principles of biomedical scientific writing: Results. Int J Endocrinol Metab. 2019;17(2):e92113.
- Docherty M, Smith R. The case for structuring the discussion of scientific papers. BMJ. 1999;318(7193):1224-1225. DOI: 10.1136/bmj.318.7193.1224 PMID: 10231230
- Dyer O. Major publisher retracts 43 papers, alleging fake peer review. BMJ. 2015;350:h1783. DOI: 10.1136/bmj.h1783 PMID: 25832507
- Van NR. How big is science's fake-paper problem? Nature. 2023;623(7987):466-467. DOI: 10.1038/d41586-023-03464-x PMID: 37949983
- Richardson RAK, Hong SS, Byrne JA, Stoeger T, Amaral LAN. The entities enabling scientific fraud at scale are large, resilient, and growing rapidly. Proc Natl Acad Sci U S A. 2025;122(32):e2420092122. DOI: 10.1073/pnas.2420092122 PMID: 40758886
- Abalkina A. Publication and collaboration anomalies in academic papers originating from a paper mill: Evidence from a Russia-based paper mill. Learn Publ. 2023;36(4):689-702.
- Ro C, Leeming J. Authorship for sale: Nature investigates how paper mills work. Nature. 2025;642(8068):823-826. DOI: 10.1038/d41586-025-01824-3 PMID: 40494985
- Zou Q. [Characteristics and recognitions of medical ghostwriting papers]. Chin J Sci Tech Period. 2020;31(12):1431-1435.
- Nosek BA, Alter G, Banks GC, et al. Scientific standards. Promoting an open research culture. Science. 2015;26;348(6242):1422-1425.
- Menon V. Serving as a handling editor? Thirteen simple messages for early-career editors. Indian J Psychol Med. 2023;45(5):456-458. DOI: 10.1177/02537176231185934 PMID: 37772140
- Budrikis Z. What it's like to be a Reviews editor. Springer Nature. Accessed November 12, 2025. https://communities.springernature.com/posts/what-it-s-like-to-be-a-reviews-editor
- Crous CJ. Could disruptive technologies also reform academia? Web Ecol. 2017;17(2):47-50.
