The jump in journal impact factors during this pandemic

INTRODUCTION

Using a single journal impact factor (JIF) as a surrogate measure of the quality of individual research articles, the contribution of an individual scientist, or for recruitment, promotion, and funding decisions is not recommended.^[1] Nonetheless, this practice has become a standard measure for evaluating scientific research output.^[2] Despite widespread criticism,^[3,4] the pursuit of high JIFs has compelled editors to prioritize them as key targets.

Fischer et al. analyzed data from journal citation reports (JCRs) from 1997 to 2016 and demonstrated that JIFs have been growing steadily,^[5] although they were expected to plateau in the future. Contrary to expectations, JIFs have not stabilized, and some journals have experienced exponential growth. Manley suggested that loopholes,^[6] such as the inclusion of citable items (CIs) like letters, have affected JIF calculations. Some editors have even manipulated JIFs by increasing the numerator in the equation or reducing the denominator.^[7] The numerator of the JIF represents the total number of citations for papers published in the journal in the previous two years, while the denominator consists of CIs published during the same period.

The inflation of JIFs has long been a concern.^[5,8,9] Several factors, including longer publication delays,^[10,11] an increased volume of publication larger reference lists,^[5,11,12] and the inclusion of non-CIs,^[13] contribute to rising JIFs. The coronavirus disease 2019 (COVID-19) pandemic likely exacerbated these issues. The protracted nature of the pandemic influenced editorial decisions, leading to an unprecedented surge in scientific research, fundamentally altering the landscape of academic publishing.

The 2022 JCR revealed that many biomedical journals experienced dramatic increases in their JIFs—some doubling or tripling—during the pandemic. Seven journals had JIFs of more than 100, all of which published high quantities of COVID-19-related research.^[14] Guo et al. observed a mismatch between the reputations of top journals and their JIFs due to publication delays.^[15] We found similar mismatches in the 2021 and 2022 JCRs.

While scholars and the publisher Clarivate have acknowledged the pandemic's role in boosting JIFs, quantitative analyses of COVID-19's impact of JIFs remain limited. The JIFs of the traditional multidisciplinary journals (Science and Nature) were significantly outpaced by those of the four major medical weekly journals (The Journal of the American Medical Association [JAMA], The Lancet [LANCET], The New England Journal of Medicine [NEJM], and British Medical Journal [BMJ]), likely due to COVID-19-related research. To investigate this hypothesis, this study examined COVID-19-related papers published in 2021 and 2022, analyzing references cited in these papers by year.

DATA COLLECTION

COVID-19-related papers published in 2021 and indexed by the Web of Science (WoS) Core Collection were collected. Due to the WoS limit of 10,000 per query, two separate retrieval periods were used to compile the data. The retrieval formula included the terms "covid-19", "covid-2019", "2019-nCoV", or "severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)" (all fields), with the timespans set to January 1, 2021, to June 30, 2021, and July 1, 2021, to December 31, 2021. Papers matching this formula were defined as COVID-19-related papers. A total of 89,294 were retrieved on July 21, 2022, and 71,079 records were retrieved on July 22, 2022. For papers published in 2022, the same formula was used, resulting in the retrieval of 84,810 and 64,693 records on July 5, 2023.

The "Full Record and Cited References" option in WoS's export feature was selected, generating TXT files for every 500 records. The total number of items published in a given year was determined by combining the retrieval formula with the publication year. The retrieval formula was also used to collect all COVID-19-related papers published in six well-known journals. All items were defined as the sum of CIs and other document types.

A Python program was developed to scan the exported TXT files. The program required the user to input a folder path and keywords for exact matches. The program logic read each line of text, identified matches for keywords, and checked for publication years (2019 or 2020). If the 2019 or 2020 counter was positive, a + 1 score was given, and if it was negative, the program continued to the next line, returning a 2019 counter and a 2020 counter. The process was repeated for the 2022 folder, returning counts for 2020 and 2021 citations.

JCR abbreviations were used as search keywords: Lancet (LANCET), New England Journal of Medicine (NEJM-NEW ENGL J MED), Journal of The American Medical Association (JAMA-J AM MED ASSOC), British Medical Journal (BMJ-BRIT MED J), Nature (NATURE), and Science (SCIENCE). To avoid mistakenly including subjournals, a comma was appended to the keywords following bibliographic rules.

STATISTICAL ANALYSIS

Data collection was performed using the Python program, while data sorting and descriptive analysis were conducted in Microsoft Excel 2021. Statistical analysis was performed using IBM SPSS 25.0. The Mantel-Haenszel chi-square test was used to examine the proportion of citations of COVID-19-related papers relative to total citations across years using a linear-by-linear association model.

RESULTS

A total of 160,373 COVID-19-related records were published in 2021, nearly double the number published in 2020 (87,268). In 2022, 149,503 COVID-19-related records were published, representing a decrease of 6.78% compared to 2021. The JIFs of the six prominent journals examined—excluding BMJ—also decreased to varying degrees in 2022.

The number of CIs (denominator of the JIF) in these six journals remained relatively stable in 2021 and 2022. However, the total number of items potentially citable by other journals declined across the four medical journals. The proportion of COVID-19-related papers varied across journals. Significant reductions in COVID-19-related papers were observed in Science (χ² = 37.660, P < 0.001), JAMA (χ² = 7.557, P = 0.006), and BMJ (χ² = 16.249, P < 0.001). In contrast, changes in the other three journals were not statistically significant (Table 1). Among the six journals, the publication patterns of COVID-19-related papers published were comparable, except for BMJ.

The citation counts identified by the Python program are presented in Table 2. For six journals, a significant decrease in citations to COVID-19-related papers was observed: citations in 2021 to papers published in 2020, citations in 2022 to COVID-19-related papers published in 2020, and citations in 2022 to COVID-19-related papers published in 2021 all showed a significant decline (linear-by-linear association test, all P < 0.001).

During the same periods, the average number of citations per COVID-19-related papers was greater than that of non-COVID-19-related papers (Table 3).

DISCUSSION

The COVID-19 pandemic has been a major research focus, leading to a surge in publications. In 2021, nearly twice as many COVID-19-related papers were published as in 2020. Despite a 6.78% decline in 2022, publication numbers remained high. Publishers facilitated rapid dissemination by reducing publication timelines and providing free access. Holbach reported that the time between submission and publication of COVID-19-related papers decreased by 49% on average.^[16] Peer review and production stage delays were shortened as much as possible in journals.^[17,18] Multiple factors contributed to significant increases in JIFs, with specializing in respiratory and infectious diseases directly related to COVID-19, such as Lancet Respiratory Medicine, even tripling their JIFs. A dermatology journal observed a 50% increase in its JIF due to COVID-19.^[19]

This study collected as many studies indexed in the WoS Core Collection as possible and used Python to conduct bibliometric research. The impact of COVID-19-related papers on JIFs in 2021 and 2022 was analyzed based on publications from 2020 and 2021. In 2021, more than half of the citations to items published in 2020 were related to COVID-19, except those in Nature. By 2022, this trend persisted, with over half of the citations to 2020 items being related to COVID-19, except in Nature and Science. For items published in 2021, more than half of the 2022 citations were COVID-19-related in only two journals (NEJM and JAMA). The rapid publications of COVID-19-related articles in prominent journals shortly after the outbreak of the pandemic attracted significant citations. However, by 2021, the number and proportion of citations had declined.

In 2020, as COVID-19 emerged, Science and Nature also published COVID-19-related papers, comparable in volume to other medical journals (Table 1). However, the JIFs of these multidisciplinary journals were surpassed in the 2021 and 2022 JCRs, likely due to author citation preferences. Medical researchers focused more on articles in the four leading medical journals, while Science and Nature had larger denominators for their JIF calculations—approximately four times larger.

The ongoing COVID-19 pandemic did not reduce the number of COVID-19-related papers published in 2022. Consequently, JIFs remained inflated, pushing the impact factor of top medical journals beyond those of traditional multidisciplinary journals. This inflation may distort the perception of journal quality, particularly in contexts were JIFs are heavily emphasized.^[20] The COVID-19 pandemic produced an abundance of JIFs—this is not a positive result. Over time, the gap between medical journals and Science and Nature is expected to widen, even without the COVID-19 factor, due to publishing patterns in science, technology and medicine (STM). Further investigation is needed to understand this shift.

Alternative metrics to JIFs have been proposed in some fields. For example, Fassin suggested using financial times 50 (FT50) for economics journals,^[21] and the SCImago Journal Rank (SJR) indicator, based on the Scopus database, offers another option.^[22] The Declaration on Research Assessment (DORA) was released 10 years ago to promote improved methods for evaluating scholarly outputs.^[23] These developments signal a shift in how journal quality may be assessed in the future.

Riccaboni et al. noted that COVID-19-related papers have dominated publication space,^[24] displacing other clinical studies. This trend is concerning. However, as the virulence of COVID-19 decreases and the World Health Organization (WHO) has declared a to the global health emergency, we expect JIF inflation to subside. Coverage of COVID-19-related papers in the four major medical journals has already declined, as shown in Table 1. Delardas et al. demonstrated that the JIFs of six well-known medical journals (Annals of Internal Medicine, BMJ, JAMA, The Lancet, Nature Medicine, and NEJM) were significantly increased by COVID-19-related manuscripts.^[25] Similarly, our findings indicate that COVID-19-related papers increased JIFs, although their average citation frequency decreased over time (Table 3). With the exclusion of 2020 citations from the 2023 JCR, JIFs are expected to stabilize.

During the pandemic, many publishers offered open access (OA) to COVID-19-related papers to enhance their dissemination.^[26] In the digital age, papers have become less tied to specific journals, and many authors prioritize readership over citation counts.^[27] The rise of the OA publishing model could reduce the emphasis on JIFs as long as accessibility is maintained.^[28] Further, the development of OA offers an opportunity to find alternative impact factor indicators. However, quality concerns in OA publishing, such as the increase in retractions—peaking in 2023, primarily from one OA publisher—highlight potential drawbacks.^[29]

CONCLUSION

A journal's reputation determines its academic standing, with JIFs often serving as a key reference for library acquisitions. However, the focus on JIFs can have long-term negative effects on publications and academia. The COVID-19 pandemic temporarily inflated JIFs due to a surge in citations of COVID-19-related papers. While JIFs remain a key bibliometric index, they do not reflect journal quality. The pandemic also accelerated review and publication processes,^[30] which sometimes compromised the reproducibility of results.^[31] Indeed, at an international conference, some academics raised concerns about the quality of COVID-19-related papers published in well-known journals.^[32] Using JIFs to compare multidisciplinary journals like Science and Nature with medical journals is problematic, as hot topics like COVID-19 can distort these metrics. It is crucial to consider the jump in JIFs rationally. Scientific progress should prioritize knowledge dissemination and collaboration over JIF-driven evaluation.