ABSTRACT

This article introduces the practical development of Competence Measurement (COMET) professional competence assessment, aiming to evaluate work-based learning outcomes in vocational education and training (VET). It is found that, compared to traditional exams, advantages of COMET lie in reflecting competence levels and cognitive characteristics, identifying relationships between teacher competence and training quality, guiding curriculum reforms, and reducing the cost of assessment. The primary challenge in promoting COMET in China at present is the high demands on organization and management, which have dampened VET institutions' enthusiasm for participation. It is suggested that efforts should be made to leverage digital technologies to enhance the quality and reduce the costs of assessment.

Key words: professional competence assessment, Competence Measurement, work-based learning

INTRODUCTION

In recent years, the most significant keywords in the vocational education and training (VET) in China has been the "integration between industry and education". This concept is closely related to apprenticeship, and they both aim to address the disconnection between VET and industry needs, albeit with slightly different emphases. Integration between industry and education as a macro framework, emphasizes the establishment of long-term cooperation mechanisms through policy guidance and the consolidation of resources between VET institutions and enterprises (e.g., co-building training bases, developing curricula, etc.). In comparison, modern apprenticeship systems in China are often viewed as micro-level initiatives, focusing on innovative training models that integrate work and learning. The ultimate goal of both approaches is to enhance the professional competence of VET learners; therefore, it is imperative to develop effective competence evaluation methods.

To evaluate work-based learning outcomes in VET, Chinese VET institutions have explored various evaluation methods, such as skill examinations and competency assessments. While these methods have played a positive role in ensuring the quality of VET, they often suffer from issues such as incomplete evaluation content and low reliability and validity. High-quality professional competence assessment should not only evaluate individual students' professional development but also diagnose institutional performance, whose results can serve as a reference for enterprise staff recruitment. This article introduces the new development of the practice of Competence Measurement (COMET) professional competence assessment in China.

THE COMET PROFESSIONAL COMPET-ENCE ASSESSMENT MODEL

COMET is a professional competence assessment program collaboratively developed by universities in Germany, China, and other countries. Based on vocational pedagogy theories, it establishes professional competence and evaluation models by incorporating guiding principles "shaping orientation", the developmental logic "from novice to expert" (Dreyfus et al., 1987), developmental tasks, and work process knowledge (Boreham et al., 2002; Rauner, 2024). The COMET professional competence model comprises three dimensions: "requirement", "content", and "action". The requirement dimension ranges from nominal competence, functional competence, processual competence, to holistic shaping competence. The content dimension involves four types of work tasks: career guided tasks, systematic tasks, problem-based special tasks, and unpredictable work-based tasks. The action dimension divides the work process into six stages, from information acquisition to evaluation (Rauner et al., 2021). COMET adopts an open book written test format, with questions developed through expert worker workshops (EXWOWO) and scoring consistency improved through rater training (Finnjust coefficient).

THE PRACTICE OF COMET COMPET-ENCE ASSESSMENT

Beijing Normal University participated in the development of COMET concept and applied it to assess the quality of talent cultivation in several "integration between industry and education" projects.

Assessment subjects

In 2021, we conducted a competence test of students from 32 VET institutions specializing in manufacturing-related fields. The study selected five majors (automation, mechanical engineering, automotive, information and communication technology [ICT], and electronics) across 14 provinces, involving 1541 students. Industry leaders, including several Fortune 500 companies, contributed to test question design and rating.

Assessment results and data analysis

In the automotive, electronics, automation, and ICT majors, 4.3%, 5.5%, 9.9%, and 38.0% of students respectively demonstrated nominal competence, while all students from mechanical engineering achieved at least functional competence. Automation students were predominantly at the functional competence level (60.1%), whereas automotive, mechanical engineering, and electronics students were mainly at the processual competence level (45.9%, 67.8%, and 74.7%, respectively). Students with nominal competence cannot independently complete work tasks and fail to meet basic enterprise requirements, placing them in a "risk group". Functional competence means the students have basic knowledge and skills, enabling the completion of simple tasks. Processual competence signifies the ability to handle complex tasks using work process knowledge. Over 75% of ICT and automation students scored below 50, concentrating in the lower score range.

Overall, students could meet the functional requirements but often failed to provide comprehensive solutions. They showed limited consideration of processual and design competence indicators, particularly in areas such as "economic efficiency", "environmental sustainability", and "creativity" (Figure 1). Processual and design competences require workers not only to complete tasks but also to understand task content in specific contexts, act responsibly, and engage in self-reflection (Zhao & Gao, 2022).

Figure 1

Figure 1. Student occupational competence profiles. Mechanical Eng. (V = 0.254, N = 255) Computer (V = 0.698, N = 287) Electronics (V = 0.226, N = 91). K1 = clearness/presentation, K2 = functionality/operability, K3 = sustainability/utility, K 4= efficiency/effectiveness, K5 = business and work process orientation, K6 = social acceptance, K7 = environmental compatibility, K8 = creativity. KF = functional competence, KP = processual competence, KG = holistic shaping competence.

RESEARCH FINDINGS

Incomplete development of students' professional competence

VET Students' competence levels followed a near-normal distribution, with most students concentrated at lower or intermediate levels. They performed well in lower-level functional competences but struggled with higher-level processual and design competences. The lower scores of ICT students may be attributed to their internships primarily being in small and micro enterprises, while the test questions reflected the standards of large manufacturing enterprises. The abstract and symbolic nature of ICT learning content poses challenges for VET students, suggesting that teachers should consider the cognitive characteristics of VET students.

Occupational motivation as a key factor in competence development

"Occupational identity and commitment" significantly influenced students' motivation and competence development, whereas work ethics had little impact. VET students' motivation stems from diverse sources, including extrinsic factors such as performance-oriented incentives and intrinsic factors such as interest in the work itself. Occupational identity, commitment, and performance orientation had the most substantial impact on competence development, reflecting students' internal acceptance of specific professional roles. Enterprise identity, commitment, and work interest also positively correlated with competence but to a lesser extent.

Significant impact of internships (apprenticeships) on competence levels

Beyond the type of internship enterprise, students' competence levels varied significantly based on whether they had internship experience, the duration of internships, and the methods of securing internships. Generally, students with internships lasting over six months and those who obtained internships through formal channels exhibited better competence development, highlighting the importance of standardized internship. However, the effectiveness of internships varied widely, with some students showing limited professional competence despite multiple internships. Factors positively correlated with competence included internship support, task characteristics, work orientation, and the internship environment.

PROSPECTS OF COMET COMPETENCE ASSESSMENT

Functions of COMET professional compet-ence assessment

Compared to traditional skill test/examinations, COMET professional competence assessment differs in the following aspects.

Reflecting competence levels and cognitive characteristics

COMET can assess the competence levels, grades, and cognitive profiles of individuals, majors, and VET institutions, serving as a valuable supplement to traditional skill tests, while traditional tests mainly evaluate task-specific skill proficiency.

Identifying relationships between teacher competence and training quality

COMET can monitor training quality by identifying the relationship between teachers' competence and students' competence development. Teachers' and students' competence profiles exhibit similar structures, indicating a "transfer phenomenon" (Zhang et al., 2022).

Guiding work-integrated curriculum reforms

By combining background surveys with quasi-experimental studies, COMET can rigorously examine factors influencing competence development. For instance, comparing experimental and control classes within the same institution can evaluate the effectiveness of curriculum reform project.

Cost-effective competence assessment

As the connotation of skills expands, so do the requirements and costs of skill assessment. For example, the WorldSkills competition evaluates not only operational skills but also design and innovation abilities. COMET ensures evaluation validity while reducing costs, offering insights for improving skill competitions and college entrance exams.

Challenges and prospects

The connotation of professional competence varies across occupations. Research has found that COMET's predictive validity for action competence differs slightly between interactive and design-oriented occupation. In China, the primary challenge in promoting COMET is the high demands on organization and management, which have dampened VET institutions' enthusiasm for participation.

The advancement of ICT offers new opportunities for COMET's development. Efforts are underway to leverage digital technologies to enhance the quality and reduce the costs of competence assessment, such as developing an intelligent competence diagnostic system based on COMET. This system integrates multi-agent, chain of thought (CoT), and retrieval-augmented generation technologies, enabling large language models to simulate human evaluation processes to improve diagnostic accuracy (Wu et al., 2023).

DECLARATIONS

Acknowledgement

None.

Author contributions

Zhao ZQ: Conceptualization, Writing—Original draft, Writing—Review and Editing, Supervision. The author has read and approved the final version of the manuscript.

Source of funding

This research received no external funding.

Ethical approval

Not applicable.

Informed consent

Not applicable.

Conflict of interest

Zhiqun Zhao is the Associate Editors-in-Chief of the journal. The article was subject to the journal's standard procedures, with peer review handled independently of the editor and the affiliated research groups.

Data availability statement

No additional data.

REFERENCES

  1. Boreham, N. C., Samurcay, R., & Fischer, M. (2002). Work Process Knowledge. Routledge.
  2. Dreyfus, H. L., Drey-fus, S. E., & Zadeh, L. A. (1987). Mind over machine: the power of human intuition and expertise in the era of the computer. IEEE Expert, 2(2), 110-111. https://doi.org/10.1109/MEX.1987.4307079
  3. Rauner, F. (2024). Handbook of Fundamentals of Modern Vocational Education. Springer Nature Singapore.
  4. Rauner, F., Heinemann, L., Maurer, A., & Haasler, B. (2021). Competence development and assessment in TVET (COMET). Springer.
  5. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., & Wang, C. (2023). AutoGen: enabling next-gen LLM applications via multi-agent conversation. arXiv. https://arxiv.org/abs/2308.08155
  6. Zhang, Z., Tian, J., Zhao, Z., Zhou, W., Sun, F., Que, Y., & He, X. (2022). Factors influencing vocational education and training teachers’ professional competence based on a large-scale diagnostic method: a decade of data from China. Sustainability, 14(23), 15871. https://doi.org/10.3390/su142315871
  7. Zhao, Z. Q., & Gao, F. (2022). Professional competence of students in Chinese vocational colleges and its influencing factors—an analysis on COMET data of five major areas of manufacturing. China Educational Technology, (6), 47-55.