ABSTRACT
As a unique type of scientific production, the assessment of scholarly book has been recently revalued under the trending Chinese technology policy. This article performs a survey of different attempts applied to various assessment problems for scholarly book, such as comprehensive assessment framework, the localization of international impact measurements the improvement measures, and the theorical recognition from different angles. The paper analyzes the questions of what, who, and how to reach an eventual integration with scholarly book assessment platforms. The conclusions indicate that the exquisite categorization and the negotiated criteria framework are firstly needed to be studied in order to initiate an assessment project of scholarly book in China.
Key words: academic assessment, monograph evaluation, Chinese academic publishing
INTRODUCTION
The critical role of scholarly books has been consistently highlighted by international researchers and evaluators, especially in the research evaluation for some Art, Humanities and Social Science disciplines.[1–4] At the same time, it is well recognized that more distinct challenges are faced with the assessment of scholarly books, as it compares with the development of scholarly journal publishing. Some researchers and evaluators are intrigued by these disputed challenges, and try to build an objective and scientific evaluation model for scholarly book assessment. Those leading practical attempts also shown in the European countries and Australia, such as the Australian Research Quality Framework and the national Spanish Evaluation and Foresight Agency.[5] However, in China, some researchers also start to focus on this topic, and makes some tentative steps to establish a Chinese scholarly books assessment system. Despite of multiple complexities of the publishing and academic environment of China, it is still valuable and beneficial to analyze on Chinese scholars’ previous works on the opinions, methods and models of assessment system of Chinese scholarly books. On the one hand, as a lens, the academic values and profits of Chinese scholarly book publishing industry can be observed and redefined from the assumed perspective of evaluator. On the other hand, some ideas and models raised and experimented by the Chinese scholars might be helpful for the relative international research of the scholarly books assessment system.
Under the China’s centralized government system, a highly-regulated publishing market formed a policy-oriented environment requiring every Chinese academic publisher to learn how live with. Though some implementation measures have promulgated ever since 2012 to strengthen its marketing role, while the censorship and review regulations also been highlighted at the same time.[6] This intervention is further exemplified through the Regulations on Publication Administration[7] (the fourth revision in 2016) issued by General Administration of Press and Publication (GAPP), where the goal of publishers is been noted as: “Social benefit shall be given first priority, and an optimal combination of social benefits and economic returns shall be achieved”. However, it ends up with an uncompleted transformation[8] where the social benefits and economic benefits are trapped in a tricky tension left for publishers to prove and argue their social contributions in the reporting routine to their “upper-level”. For the assessment of scholarly publishers in China, a unique question remains as to how to coherent the business logic of international publishers with the required social benefits to fit the Chinese regulation model. This provides a special background and incentive for researchers to quest an assessment model with Chinese characteristics. Through this review, it aims to present the works have been done by Chinese researchers and evaluators.
METHODOLOGY
A systematic review is organized in this article as the focuses of the assessment system of scholarly books is related with multiple stakeholders, including publishers, scholars, and government. To conduct it, two main steps have been taken for this review: Search for the related works on the mainstream platforms until now, including China National Knowledge Infrastructure (CNKI), Wanfang Data, VIP (Weipu) Chinese Journal Service Platform, Web of Science, Science direct, EBSCO etc.; and analysis of the collected work. The search is conducted with the keywords based on the combination of key terms, which can be refined as “monograph” or “scholarly/academic books/writing” after initially reviewed the search results. The search was open to any domestic research on the assessment of scholarly books in China. Thus, the screening criteria for the returned results is to exclude the works on the similar topics, such as the assessment of journals and the administrative assessment for publishers. Through a cautious screening of work vaguely focused on the academic publishing, 32 articles and 9 dissertations were selected as the analyzing materials. The analysis of the work is aiming to answer the following questions: (1) The object of assessment, namely, they think what can be categorized as scholarly book; (2) The subject of assessment, namely, they think who should take responsibilities for assessment of scholarly books; (3) The goal and the methods of assessment, they think what’s the ultimate goal to achieve through conducting some ways. The time range of the collected work is from 1988 to 2024 (Figure 1), showing a stable researching frequency in the area the academic publishing in recent ten years (from 2013 to 2023). However, even only analyze the total count of all the works dating back from 1988, the current research progress can barely be seen as abundant as it should be in China, given that the academic book takes a consequential status in Chinese academic assessment system. Among all the collected works, it should be noted that some studies solely pointed out the systematic problems and its countermeasures, which can be seen as a special writing type for Chinese authors who has highly academic impact in the publishing industries. Number of articles/dissertations Published year
Figure 1. The time range of collected work.
A REVIEW OF THE ASSESSMENT OF SCHOLARLY BOOKS IN CHINA
There are some efforts have been made to land the so-called “social benefit” as some practical and objective criteria, in order to help the competent institutions conduct the assessment of publishers. Such work clearly focuses on the governing the publishers in the general sense, but the academic publishing is not only regarded with balancing the relationship between governors and publishers, but also has to solve the following questions from other stakeholders who may be deeply influenced by the assessment system, including individual scholars, libraries, academic journals, educational management institutions, and academic funding institutions. As it entangled with multiple critical decision-making process, the assessment system for scholarly system is still under a pilot research period, where open to all kinds of discussions and arguments. This fact leads this establishment of assessment system in China will be a continued and distracted process holding on to further contributions. To keep the accordance of this review, the following results will highlight the major disputes argued by authors, without further discussions on the testified detail of quantitative studies.
The oldest article referred in this review was published in 1988 (1 paper), which elaborated the trend of the Chinese book review method is transferring from single-factor to muti-factors. After that, the constant production of works has been kept from 2013, and rose to a high point and peaked in 2020 (8 papers). For the distribution of authors’ institution, the biggest group (18.42%) of authors is from Nanjing University, of which China social science research evaluation center of developed a dominated database called Chinese Social Sciences Citation Index. This practical researching needs promote more authors to focus on the evaluation of scholarly books. The most cited paper is published by Jiyuan Ye (one of the influencing academic leaders in the area of academic assessment) in 2016, which tries to explain the meaning and referend of some blurred concepts, including academic book, academic writing, and academic monograph.[9]
The question of assessment of scholarly book has been studied from several angles, being that 4 angles were summarized in a total of 38 works in this review. The most popular angles are the comprehensive assessment framework (16 papers), the localization of international impact measurements (15 papers), the improvement measures (12 papers), the theorical recognition (9 papers). Since some of papers might discussed this area from 2 or 3 angles, especially for some enriched dissertations contained with at least 2 angles. Therefore, this summarize is classified solely based on the relativity of its content. The following sub-sections present a brief introduction to the four angles found in this literature review.
The comprehensive assessment framework
The assessment for scholarly book is facing a dilemma, compared with journal assessment. On the one hand, the traditional measurement evaluation is poor in timeliness, and have to count on too much on the peer review. On the other hand, Altmetrics evaluation indicators is not well organized for evaluating books.[10] A coherent assessment framework is been widely suggested from the perspectives of factor, dimension, measurement, data resource, and method. The leading idea is to integrate more consideration into accounts for assessment of scholarly book.[11–13] Start from this standpoint, the sub-level of factor, dimension, and data varies in regard of authors’ trustiness for different platforms and sources. Despite the disagreements on the sub-level, the procedure of weight coefficient is processed in a multi-objective model.[14–16] For the method, the combination of qualitative and quantitative is seen as a more scientific way to evaluate scholarly books.
The localization of international impact measurements
The impact measurement is considered as the trustable foundation for the academic evaluation. Among of all impact measurements, the Altmetrics is been widely introduced and studied to adopt Chinese academic environment. The different branches of applying Altmetrics shown in the phase of tool development.[17–19] The representative researches are working on developing a stratified framework of Altmetrics evaluation indicators,[17] an assessment index through analysis discipline characteristics,[18] and a detailed introduction and comparison of the mainstream Altmetrics tools.[14] The following steps can be summarized during the localizing process of Altmetrics in China. (1) The introduction as a new tool: The Altmetrics is imported as a complementary method for the traditional citation impact in China, as it has only been considered as the limited index for evaluating the influence (not the actual quality) of academic publications.[20] (2) The detailed elaboration for its boarder application: It is been constructed as a more comprehensive assessment discourse in the digital communication era in the contrast with traditional citation impact, which likely to reshape the cognition of scientific efficiency in China.[21] (3) Some experiments conducted in domestic communication environment: The experiments are mainly conducted on Chinese platforms (such as Sina Weibo, Douban, and Amazon), to explore a multi-index assessment metrics featured on its Chinese data sources, including awards, translations and citations from state-controlled medias.[22]
The implement measures
Academic assessment is a systematic procedure implemented by different stakeholders. The whole procedure consists of some main critical parts, including the selection of the evaluator or reviewer, the goal and function of evaluating institution, the collecting data, and the feedback and publishment of the result.[23–25] When conducting this assessment loop, some researchers argued to be more critical about the interventions from social relationship (“Guan Xi” in Chinese), political supervision, and academic capitalism.[25,26] In addition, there are more challenges need to be fixed in the disciplines of social science.[10,27]
The theorical recognition
It seems like the assessment of scholarly book is supposed to be a practical project, however, the unsettling theorical issues still exist in the Chinese academic publishing industry. The social interest and economic interest is still struggling to find a balance when it comes to the assessment of academic publishers.[28,29] it is unavoidable for the assessment of scholarly book to gain enough discussion on the object,[9] subject[24,30] and goal[31] of conducting it. To resonate the goal of assessment with the national policies, some critical logical issues are needed to be clarified, for example, what does the assessment of scholarly books mean for constructing the Chinese academic discourse system internationally.[26,28]
DISCUSSION
In this section, the further analysis and observations are presented based on the arguments of the selected works. To concentrate on the critical issues highlighted by authors, a systematic structure is deployed to sort out the relationship and logic of ideas analyzed from the literatures. This imported structure is based on a 5W1H principle to approach the basic elements with the assessment problem, which are what, who, when, where, why and how. Since when and where is no need to further discuss, what, who, why and how are the key issues to answer and discuss, which are elaborated as the object (what), the subject (who) and the goal (why) and the methods (how) in the assessment system of scholarly book.
The object of assessment
The object of assessment refers to the understanding of scholarly/academic book, academic writing, or (academic) monograph. More specifically, it should be clearly distinguished from other non-academic books, such as textbooks and public literature. The locating for scholarly books is rather challenging in the fields of social science and humanities, which can be proved by a number of research efforts has been made within the fields, according to Nederhof’s study in 2006.[32] Under the Chinese linguistic context, the category of scholarly books is an initial and decisive problem deserving to be studied, according to the result of the topic of the most-cited paper. Ye’s arguments on the category of this definition have been widely cited by 141 papers. Through the analysis of the practical usage of the similar terms in different areas, he argues that “scholarly/academic book” is the broader term to be unified when conducting the statistics and assessment project. As for the other terms, such as academic writing/monograph, are classified as the lower class in the controlled vocabulary system. In general, the way of defining the scholarly book is to highlight the academic value, compared with the non-scholarly books. This binary thinking makes the definition tangled with another unsettling term, which is “academic”. Therefore, the question of what is going to be evaluated is not just the range of scholarly book, but also the paradigm of science has been understood by the whole society.[31] In China, new policy initiatives are shifting from quantity-oriented to quality-oriented assessment criteria, leading to a trend that high-quality publications are prioritized.[6] This shift may reshape Chinese assessment criteria for the academic value and classifications for the scholarly books in the near future. Meanwhile, the definition for “academic” remain an unsettled theoretical question for Chinese scholars, which requirs the further and deeper quests on depicting the boundaries of scholarly books across different disciplines in China.
The subject of assessment
Regardless of the blurred definition of academic book, another practical problem is who shall be accountable for the assessment results. Like the international researches has also discussed about, the peer review system is an essential way to balance the relationship between subjective and objective evaluating data. Also, the professional knowledge is the fundamental part for a specialized publications to be scientifically evaluated.[33] There are two issues which are quite unusual for Chinese assessment system: (1) The rewarding or prize systems initiated by governments, which has largely affected the status and prestige of academic publishers, and further be considered as a critical factor to evaluate book.[29] However, some authors also concerned of if the assessment been initiated by governments, may bring the utilitarianism into academic publishing field.[25] (2) The publisher’s role in the assessment process, in other words, the roles of editors in the academic publishing industry. There are some opposite opinions on the question of whether should let editors also be accountable for the assessment. On the one hand, some authors insist on a more diverse evaluators group jointed by editors, readers and academic authorities.[34,35] On the other hands, the concerns of bringing the editors into assessment system are raised by some authors, such as their academic abilities.[25,36,37]
The goal and the methods of assessment
The methods and the goal are bounded with each other, and jointly present the Chinese characteristics of academic assessment system. The discussion on the goal is actually landed on an ultimate issue, that is who will be the users of this assessment system. Generally, academic assessment is designed to serve the academic ability and effectiveness of each scholar as a researchers’ personnel management tool. In this case, assessment system is eventually been used by the scholarly performance evaluators. However, if the goal of this assessment system is exactly located on the academic quality of books, then will lead to another type of users, who are prone to be supported by the administrative role or the supervising role.[38] When the assessment is been used for the former goal, peer review will be used as the leading method to make the rules and results more compatible with the academia norms. Starting from the latter goal, publishers will be the easiest tunnels of conducting this assessment. it is noted that Chinese publishers are been classified as three administrative levels, including national level (A class), provincial level (B class), and other level (C class). This hierarchy system has deeply hindered to rebuild an academic-oriented evaluation system through the assessment of publishers,[30,39,40] which can partly explain the assessment of publisher is not as feasible as it can be in the western countries. Still, there are some studies believed the prestige of publisher is a positive factor to make the assessment system more practical.[41,42] If so, the following challenge is how to translated and integrate the wights of the social benefits, academic benefits and economic benefits required by the GAPP in the constructed assessment system.
CONCLUSION
This paper presented a review of research efforts has been devoted to the assessment system of scholarly books in China. It briefly introduced the assessment environment of Chinese academic publishing and described the problems focused by interested authors, described the data source and reported the overall trend of previous works, listed representative arguments such as the term classifications in the domain of scholarly book, and further elaborated an undergoing assessment system combined by the targeted methods and possible application scenarios based on the principle of 5W1H.
The findings hardly reached a systematic competence of scholarly book assessment in the majority of the reviewed works, but still made some progress on the evaluating models scattered in the limited disciplinaries. Even the criteria of scholarly book are not as clear as it should be, more authors still called for a diversified assessment system as a comprise to strike a balance among the social interests, academic integrity and economic interests. However, the practical solution of how to put into practice are still with low research work, such as the publisher’s evaluating mechanism. The finding also shown that the intervention from Chinese publishing administrative system strongly affected the construction of academic publishing field, which bring out a tension between markets and policies in the leading status of evaluator. For a proper integration of the assessment system of scholarly book, the next phase of research should initially focus on unified the categories of scholarly books and specified the leading role from the existing players in the academic publishing field.
DECLARATION
Author contributions
Wang LL: Investigation, Writing—Original draft. Chen F: Writing—Review and Editing.
Source of funding
The authors are financially supported by the National Social Science Foundation in China (No. 23@ZH003).
Conflict of interest
The author has no conflicts of interest to declare.
Data availability statement
No additional data.
REFERENCES
- Glänzel W, Schoepflin U. A bibliometric study of reference literature in the sciences and social sciences. Inf Process Manag. 1999;35(1):31–44. DOI: 10.1016/S0306-4573(98)00028-4
- Summerfield M, Mandel C, Kantor P. Perspectives on scholarly online books. J Libr Adm. 2001;35(1/2):61–82. DOI: 10.1300/J111v35n01_05
- Engels TCE, Istenič Starčič A, Kulczycki E, Pölönen J, Sivertsen G. Are book publications disappearing from scholarly communication in the social sciences and humanities? Aslib J Inf Manag. 2018;70(6):592–607. DOI: 10.1108/AJIM-05-2018-0127
- Toledo EG. Why books are important in the scholarly communication system in social sciences and humanities. Sch Assess Rep. 2020;2(1):6. DOI: 10.29024/sar.14
- Giménez-Toledo E, Román-Román A. Assessment of humanities and social sciences monographs through their publishers: a review and a study towards a model of evaluation. Res Eval. 2009;18(3):201–213. DOI: 10.3152/095820209X471986
- Wang J, Halffman W, Zwart H. The Chinese scientific publication system: specific features, specific challenges. Learn Publ. 2021;34(2):105–115. DOI: 10.1002/leap.1326
- Regulations on Publication Administration. Updated August 23, 2014. Accessed October 15, 2023. https://english.www.gov.cn/archive/laws_regulations/2014/08/23/content_281474983043721.htm
- Ren SL, Cheng WH, Liu XM, Yan J, Liu PY, Liu J. [Thoughts on speeding up the development of China's high-quality journals]. Bulletin Natl Nat Sci Found China. 2018;32(6):645–651.
- Ye JY. [A study on the concepts of academic book, academic writing and academic monograph]. Journal of Library Science in China. 2016;42(1):21–29.
- Zhu SQ, Qiu Y, Chen H Y. [Research on the adaptability of Chinese book comprehensive evaluation system incorporated with review indicators]. Libr Inf Serv. 2021;65(9):23–31. DOI: 10.13266/j.issn.0252-3116.2021.09.003
- Qiu Y, Zhu SQ, Fan DD. [Research on the construction of quality evaluation system of Chinese books based on multi-dimensional index fusion]. Libr J. 2021;40(3):109–115.
- Lin XH. [Study on academic impact evaluation of Chinese books under web2.0 multiple dimensions: case study of computer books]. New Century Libr. 2018;(1):22–25+34.
- Zhang Y, Pan YT, Yuan JP, et al. [The construction of academic impact evaluation system of Sci-Tech books in China under multiple dimensions]. Libr Inf Serv. 2015;59(7):69–76. DOI: 10.13266/j.issn.0252-3116.2015.07.009
- Lei SY, Lv XJ. [Study on the impact evaluation of academic books from an Altmetrics perspective]. J Southwest Minzu Univ. 2017;38(6):225–231.
- Tang JR, Ye Y. [A comparative study of academic trace and impact torque on single publication]. J Libr Sci China. 2015;41(2):4–16.
- Yang CH, Chen R. [An empirical study on the quantitative evaluation of the academic impact of biomedical publications]. Sci-Tech Publ. 2013;(11):125–128.
- Li M, Chen M. [Stratified framework of academic books Altmetrics evaluation indicators]. J Mod Inf. 2018;38(5):106–109.
- Jiang CL, Guo QQ, Zhang GY. [Bookmetrix-based Discipline Characteristics analysis of the academic book assessment indexes]. J Intel. 2020;39(8):193–202. DOI: 10.3969/j.issn.1002-1965.2020.08.029
- Xing X. [A comparative study of the academic book impact from a disciplinary perspective]. Wuhan: Wuhan University. 2020. (Thesis)
- Liu CL. [Scientometrics in the Web 2.0 Environment: Altmetrics]. Libr Inf Serv. 2012;56(14):52–56+92.
- Qiu JP, Yu HQ. [Some Basic Problems in Advancing the Development of Altmetrics]. J Libr Sci China. 2015;41(1):4–15.
- Yao L, Chen ZQ. [Study on the influence evaluation system of National Library of philosophy and Social Sciences]. J Southwest Minzu Univ. 2016;37(6): 232–236.
- Ren J. [The evaluation of academic book based on web data]. Nanjing: Nanjing University. 2015. (Thesis)
- Hu HL. [Research on academic works' credibility evaluation]. Wuhan: Wuhan University. 2013. (Thesis)
- Zhang ZQ, Yin ZK. [Evaluation of Chinese academic books: major problems, strategic scrutiny, and issue consensus]. Mod Publ. 2020;(4):16–22.
- Wang WJ. [Chinese academic book evaluation system in the new era: methodology and practice]. Mod Publ. 2020;(2):39–44+38.
- Wei QX. [Research on the implementation mechanism and construction of the influence evaluation index system of the academic monograph of humanities and social science]. Dalian: Dalian University of Technology. 2018. (Thesis)
- Xu LP, Niu MW. [Current situation and thoughts on Chinese academic books going global under the perspective of scientific evaluation]. China Publ J. 2021;(5):60–64.
- Zhang YL, Cai JH. [Empirical research on academic books evaluation—Evaluating the content of leather books with applied research results as an example]. View Publ. 2016;(14):36–38.
- Hao YL, Feng B. [Research on the construction of evaluation mechanism of domestic science and technology books]. Sci-Tech Publ. 2015;(2):81–83.
- Wang KQ. [A brief discussion on book evaluation standards with Chinese characteristics—Analyzing the development trend of book evaluation standards from single-level to multi-level from the perspective of convergence of natural sciences and social sciences]. Documentation, Inf Knowledge. 1988;(1):63–67+50. DOI: 10.13366/j.dik.1988.01.028
- Nederhof AJ. Bibliometric monitoring of research performance in the Social Sciences and the Humanities: a Review. Scientometrics. 2006;66(1):81–100. DOI: 10.1007/s11192-006-0007-2
- Yuan XL, Shen C. [Comparative study on peer review and bibliometric analysis used for monographs evaluation in the area of humanities and social sciences]. Libr Inf. 2020;(05):24–30.
- Ge J, Xue MY, Yang SC. [A preliminary study of evaluation methods for scientific and technical publications]. Ke-ji Chengguo Zongheng. 1999;(1):9–11.
- Jin YC. [The academic evaluation function and enhancement path of academic editors]. Publ Res. 2017;(11):56–59.
- Tian WP. [Professional editor positioning in academic evaluation]. Hebei Acad J. 2023;43(4):9–14.
- Zeng Q, Liu J, Liu M. [Research on academic evaluation of books from a user perspective]. Shipbuild Vocat Educ. 2023;11(6): 73–76.
- Dai GX, Zhou XY. [The Comparative Research on book evaluation standards in library and in press]. J Acad Libr. 2013;31(5):36–40.
- Sun SY. [An overview of the evaluation of scholarly publishing organizations in China]. Publ Res. 2016;(7):31–34.
- Zhang J. [Book evaluation should not follow the rut of "review articles by periodicals"]. Update March 25, 2015. Accessed October 15, 2023. https://news.sufe.edu.cn/43/b5/c186a17333/page.htm
- Wang WJ. [The double variation of academic and market: academic book publishing in the process of building socialist market economy]. Southeast Acad Res. 2022;(04): 237–245.
- Li XX. [Overview of book evaluation index system and methods]. Comput Knowl Technol. 2020;16(23):23–26.
