A scoping review of methods used to analyze engineering curricula quantitatively with curricular analytics

Course level metrics - defining "important" courses in the curriculum

Using the concepts of in-degree and out-degree, the bottleneck course metric can be used to identify gateway courses students must complete to access substantial portions of their remaining program (Molontay et al., 2020). The formula to determine if a course v_ij is a bottleneck uses the logical operator or (V) and three inequalities.

The values a, b, and c represent the thresholds for a course to be considered a bottleneck. Specifically, a represents the minimum number of prerequisite courses (in-degree); b refers to the minimum number of courses for which a course acts as a prerequisite (out-degree); and c indicates the total degree (sum of in-degree and out-degree) required for a course to qualify as a bottleneck. If any of the inequalities hold, the course is considered a bottleneck. Unlike the degree centrality metric, the bottleneck course metric enables us to distinguish between different types of prerequisite arrangements using threshold values. For example, consider B and D in Figure 2. Both courses have a degree centrality of 3. However, if we require that a and b be 3 and c be 5, (so the course must have at least three prerequisites, be the prerequisite for at least three courses, or be connected to five or more courses), B would be considered a bottleneck course, whereas D would not. There is scant advice on what the values of a, b, or c should be, with the exception of Wigdahl et al. (2014) using (3, 3, 5) as their threshold values.

The bottleneck classification is inherently parameter-dependent. The set of courses labeled as bottlenecks can change substantially depending on the chosen thresholds: (a, b, c). This creates a comparability issue when bottleneck results are contrasted across programs, institutions, or time, because observed differences may reflect threshold choice as much as underlying curricular structure. In the same curriculum, a threshold such as (3, 3, 5) may flag only a small number of highly connected "hub" courses, whereas (2, 2, 4) could classify a much larger set of gateway courses as bottlenecks. Therefore, for comparative analyses, researchers should (1) report (a, b, c) explicitly, (2) justify the choice using a clear rationale (e.g., prior work, curriculum size/density), and (3) consider a sensitivity check showing whether conclusions hold across a range of thresholds.

Continuing with the idea of prerequisite relationships, we have the blocking factor (b_c) next, which is the number of courses a student cannot take if a prerequisite is failed (Heileman et al., 2018; Slim, 2016).

where I is the indicator function signifying if there is a prerequisite chain linking two arbitrary courses: v_*a and v_*b.

Unlike the bottleneck course metric, which only considers the requisite relationships the course is in proximity to, the blocking factor considers dependencies throughout the network. In the example shown in Figure 2, if Course B is failed, it directly blocks Courses C, D, and E. Additionally, Courses F, G, H, and I are indirectly blocked, bringing the total to seven blocked courses for Course B. The blocking factor directly measures the extent to which a particular course would hinder student progression in the degree program (Heileman et al., 2018).

The reachability factor (r_c) is the complement of the blocking factor. While the blocking factor measures how many courses become inaccessible if a specific course is failed, reachability instead quantifies the number of prerequisite courses that must be completed before a given course can be taken. To calculate the reachability of a course, we count the number of courses that must be completed to enroll in the course we're considering (Heileman et al., 2018):

For example, the reachability factor for Course I is 3 because one must take Courses B, D, and G to ultimately enroll in Course I.

In addition to the relationships between courses, timing within the curriculum network is crucial in determining how course failures affect student progress. The deferment factor quantifies the impact of repeated failures by measuring the threshold before additional semesters are required (Molontay et al., 2020).

where k represents the maximum number of allowable failures before exceeding the expected program completion time. A higher deferment factor indicates low flexibility, whereas a lower value suggests that failing a course is less likely to lead to an extended time-to-degree. For example, assuming a four-term completion time, B, D, G, and I have deferment factors of 1. This means that students cannot fail those courses without also extending their time-to-degree. On the other hand, A has no requisite relationships. Therefore, a student could retake Course A three times, corresponding to D_c(A) = 0.25 before that specific requirement begins impacting their graduation timing. By incorporating graduation timing alongside the structural properties of the curriculum, the model quantifies how prerequisite relationships influence a student's expected completion time. Specifically, the deferment factor identifies courses where failures or delays create cascading effects, highlighting critical courses that disproportionately impact student progression and overall time-to-degree (Molontay et al., 2020).

At its core, the deferment factor concerns the overall structure of prerequisite sequences. The most closely related metric is the delay factor (d_c), which represents the longest prerequisite chain that includes the course. Heileman et al. (2018) provide the following definition.

For example, as shown in Figure 2, the longest course sequence chain is 4 for courses B, D, G, and I, since the sequence {B→D→G→I} exists. Therefore, each of these courses has a delay factor of 4. A course with a high delay factor represents a high-stakes course for timely graduation (Heileman et al., 2018). While the blocking factor measures how many courses become inaccessible if a course is failed, the delay factor captures how prerequisite chains can impact a student's time to degree completion.

Another aspect of the original conceptualization of curricular analytics, as noted with the deferment factor, is the influence of timing on curricular complexity. A metric that incorporates timing directly into its calculation is the inflexibility factor I_f(v_ij). This metric aims to quantify the impact of courses with limited offerings, such as fall-only courses, by examining the effect on students' time to degree when course completion is delayed (Reeping & Grote, 2022). The formula for the inflexibility factor builds on the delay factor by applying two penalty terms: one for the number of terms that would ultimately extend the students' time to degree, and one for the number of terms the course was shifted.

Here, j denotes the term the course was originally scheduled in, p(v_ij) is the number of terms that the course extends beyond the expected time to degree once moved (if applicable), and t_w(v_ij) is the number of semesters the student must wait to attempt the course again. For example, consider course G in Figure 2, which has a delay factor of 4 (d(G) = 4) that is positioned in term 3 of the curriculum (j = 3). Assume the course is offered only in the Fall and the expected graduation term is Term 4. When we move it to the next possible semester, the student's time to degree is extended by 2 semesters beyond the expected graduation time (p(v_ij) = 2). Because the course was Fall only and requires waiting two terms for the next offering, then t_w(v_ij) = 2. Taken together, the inflexibility factor would be I_f(G) = 4 (3 + 2 + 2) = 28. This score indicates that the course imposes substantial timing constraints, which can impact a student's ability to complete the program on time. The higher the inflexibility factor, the more critical the course becomes in delaying student progression.

Course level metrics - combining other factors to form "importance" scores

Having examined how blocking, reachability, deferment, and delay factors reveal the structural constraints within a curriculum, we now turn to a metric that identifies the most critical courses within it - cruciality. A course's cruciality c(v_ij) is the sum of the course's blocking and delay factors as follows (Heileman et al., 2018; Slim, 2016).

For example, course B's cruciality is 11, because (B) = b_c(B) + d_c(B) = 7 + 4 = 11. Course cruciality identifies courses that are critical to timely graduation (Slim et al., 2014), but authors have extended this concept to incorporate timing in the calculation. To accomplish this, weights are assigned to terms to penalize crucial courses scheduled later in a curriculum. Term weighted course cruciality is defined as the product of course cruciality c(v_ij) and the term j in which the course is taken.

Term-weighted cruciality can be used to evaluate the effectiveness of course placement within the curriculum (DeRocchis et al., 2021). Higher values suggested that crucial courses are being scheduled too late, potentially delaying student progression. In contrast, lower values indicate better alignment, with key courses appearing earlier in the sequence, supporting smoother advancement toward graduation. These term-weighted values can be compared with the unweighted crucialities to uncover unexpected bottlenecks based on their placement in the program.

Course level metrics - other perspectives on "importance"

Another perspective quantifying the importance of a course is centrality. Centrality is the number of paths that include the course, then the centrality of v_ij, denoted q(v_ij), is defined as follows (Heileman et al., 2018).

Here, p_vi = {p₁, p₂, …, p_n} is the set of all paths in the curriculum graph that include v_ij, start at a source node (i.e., a course with no prerequisites) and end at a sink node (i.e., a course with no successors). Unlike the delay factor, which focuses on the longest prerequisite chain that includes the course, centrality captures all prerequisite chains that do. Under this operationalization of importance, a course is more important when it is an element of multiple prerequisite chains. Note that courses at the beginning or end of a pathway have zero centrality by definition. For example, in Figure 2, Course D has a high centrality score because it connects foundational courses, such as B, to advanced courses, including F, G, and I.

Although centrality captures how frequently a course appears on any long path in the curriculum, betweenness centrality focuses specifically on a course's role in connecting other courses via shortest paths. It quantifies how often a course serves as a bridge between two different courses (s and t) in the shortest possible sequence. Formally, it is expressed as follows.

where σ_st is the total number of shortest paths between courses s and t (Molontay et al., 2020). The other term, σ_st(v_ij), is the number of those shortest paths that pass through the course v_ij between courses s and t. For example, in Figure 2, Course D has a high betweenness centrality because it lies on the shortest paths connecting B to G, F, and I. Both centrality and betweenness centrality focus on the importance of specific courses; however, there is another type of betweenness centrality (Davis et al., 2020) that extends the analysis from courses to requisite relationships in the network. The formula for betweenness edge centrality has nearly the same form as σ_st(e). The only difference is now σ_st(e), represents the number of shortest paths between vertices s and t that include the edge e.

This is one of the few metrics that focuses on assigning values to the prerequisite relationships themselves, instead of using those connections to assign values to the courses. In our example curriculum in Figure 2, (D, G) has a high value for betweenness edge centrality.

Curriculum-level metrics - summarizing complexity with single scores

Next, we will focus on curriculum-level metrics, which provide insight into the complexity of an entire program or specific subsets of a curriculum. First, we can readily describe the structural complexity - also referred to as program complexity - of the curriculum as the sum of all cruciality values in the network. This metric offers a summary score of the interdependencies within a curriculum (Heileman et al., 2018; Slim, 2016), The structural complexity, given by α_c, can be used to compare different programs. In Figure 2, the overall structural complexity is 52.

Like the term-weighting that has been explored with cruciality, we can sum these weighted crucialities to form the term-weighted curricular complexity (α_wc; DeRocchis et al., 2021). Note that, unlike the structural complexity, which has a unique value for each curriculum, introducing term weights leads to a result that is dependent on how courses are arranged. For example, when adding term weights, our term-weighted structural complexity becomes 109. Moving J's prerequisite sequence up by one term increases the term-weighted structural complexity to 121 - an 11% increase.

Structural complexity is not the only metric for summarizing the curriculum's complexity. For example, curriculum rigidity (C_r) measures the density of prerequisite and corequisite relationships within a curriculum, calculated as the ratio of the total number of prerequisite and corequisite relationships to the total number of courses.

Curriculum rigidity is comparatively easier to calculate than structural complexity, with a higher score (i.e., > 1) indicating a more constrained curriculum in which courses depend more heavily on one another. For example, if our example curriculum in Figure 2 has 12 courses (|V|= 12) and 9 prerequisite/corequisite relationships between them (|E|= 9), its rigidity would be 0.75. So, it is generally considered somewhat flexible.

When formulating what it means for a curriculum to be complex in terms of rigidity, the focus is on inflexibility, frequently measured by the density of prerequisites. In other words, the metrics have consistently conveyed the sentiment that a higher number is a negative indicator. On the other hand, the degrees of freedom (z_c) metric approaches curricular complexity from a different perspective: Flexibility. The degrees of freedom metric is a count of the number of valid ways students can arrange their course schedules (Reeping & Grote, 2022). Heileman et al. (2018) do not provide a formula for calculating the degrees of freedom, and it is perhaps not possible to express this metric in a closed form, considering some arrangements are not feasible (e.g., too many or too few credits). The degree-of-freedom metric is closely related to the concept of weakly connected components in the curriculum graph, which is the maximal subgraph in which any two vertices can be reached by an edge (ignoring direction). These weakly connected components form subgroupings of requirements, which can help isolate specific design patterns in the curriculum. A curriculum with more weakly connected components in more scattered clusters tends to have greater scheduling flexibility. In contrast, fewer, larger components indicate a more rigid structure with limited course arrangement options (Heileman et al., 2018). Calculating degrees of freedom using weakly connected components is possible, but it likely requires more computational resources to simulate all possible arrangements.

Alternatively, we can center the degrees of freedom on the number of terms a course can be moved in the curriculum and sum those values to obtain a score on the curriculum's flexibility. Thus, the degree of freedom for a curriculum can be defined as (Reeping & Grote, 2021):

In this case, n_t is the total number of terms available for scheduling that do not exceed the students' expected time to degree t_e, d'(v_ij) is the delay factor ignoring corequisite relationships, and u(v_ij) represents the number of ineligible terms where a course is not offered. If a course is only offered in specific terms (e.g., Fall only), u(v_ij) captures how many semesters the course cannot be shifted within a student's schedule. For our example curriculum, our z_c is 11, after accounting for the number of moves all courses can make. In the usual operationalization of curricular analytics, u(v_ij) is zero for all vertices because courses are assumed to be available at all times. By identifying such constraints, the degree of freedom helps assess structural bottlenecks that limit flexibility in curriculum planning, enabling curricula to be restructured with multiple valid scheduling arrangements.

To further develop the conceptualization of curricular complexity, efforts have been made to introduce metrics that are sensitive to specific student populations. The most pronounced addition is the integration of transfer-specific metrics to provide deeper insights into the obstacles these students encounter (Reeping & Grote, 2022). Although curricular complexity metrics have been effectively used to analyze graduation rates for First-Time-In-College (FTIC) students (Heileman et al., 2018; Nash et al., 2021), they do not necessarily capture the unique challenges faced by transfer students (Grote et al., 2021). Transfer students encounter additional complexities, including credit loss, course timing discrepancies, and structural misalignments between sending and receiving institutions. Researchers have sought to address these gaps by enhancing existing structural complexity metrics (Reeping et al., 2021). One transfer-specific metric is the transfer delay factor (T_d), which captures rigid prerequisite structures that affect a transfer student's time to graduation by extending it due to how courses are arranged. Unlike other measures that do not incorporate graduation timing, the transfer delay factor penalizes sequencing that extends a student's expected time to graduation. The transfer delay factor is defined as the sum of delay factors for all courses that a student must complete beyond their expected time to degree (t_e):

For example, consider a transfer student who enters a program structured as shown in Figure 2. Suppose the curriculum was intended to be completed in three terms (t_e = 3), but the course sequencing makes it impossible to do so. Now, course I is extending the student's time to degree by one term. The delay factor of I is 4, since the longest prerequisite chain it belongs to is four courses long. It is the only course that extends the time to degree, so the sum of the delay factors is also 4. This metric allows institutions to identify sequences that systematically delay students, particularly transfers, and informs more flexible curricular planning.

Curriculum-level metrics - extracting specific curricular design patterns

Beyond examining specific courses and the entire curriculum, there are situations in which one might want to calculate any of the metrics reviewed thus far for a subset of courses. These subsets are typically specific curricular design patterns (Heileman et al., 2017), such as courses connected to the Calculus sequence, introductory engineering courses, or the mechanics sequence (i.e., Statics, Dynamics, and Strength of Materials; Padhye et al., 2024). For these design patterns, we can calculate the subcomplexity, also referred to as pattern complexity, which is the structural complexity of a subset of the curriculum network (i.e., a subgraph). However, more can be done with these subnetworks.

For example, the underlying process for calculating the transfer delay factor can be revised to quantify the proportion of complexity associated with a student's extended time to degree, referred to as complexity explained. To accomplish this, we would calculate the transfer delay subcomplexity (α_td; Reeping & Grote, 2022). This calculation involves forming a subgraph of courses that are connected directly or indirectly to the courses extending the students' time to degree, which is encapsulated in the Is indicator function, and summing up the crucialities of those courses - which could be weighted or unweighted - to get the complexity of the network.

Here, c(v_ij) is the cruciality of the course, and I_s(vij, v_{*j > te}) is the indicator function identifying courses that are connected to courses that extend the student's time to degree, t_e. For example, consider a transfer student entering a program structured as shown in Figure 2, with an expected graduation time of 3 semesters (i.e., t_e = 3). By design, the student cannot complete the program because course I is extending completion beyond the expected two-term mark. If we assign cruciality scores, we find that c(B) = 11, c(D) = 7, c(G) = 5, and c(I) = 4. So, α_td= 11 + 7 + 5 + 4 = 27.

Once we calculate the transfer delay subcomplexity, we can divide it into the overall structural complexity to calculate the complexity explained, α_e, which provides a quantitative way to describe the complexity attributable to that design pattern.

In our example, the explained complexity is 27/52, or 52%. This means that 52% of the overall complexity is associated with completion delays. While initially used to assess course-level delays, it can also be applied more broadly to evaluate how specific curricular design patterns contribute to program-wide complexity (Reeping & Grote, 2022).

Instructional complexity

Compared to the structural complexity component of curricular analytics, the instructional complexity has received much less attention. Across the reviewed papers, we identified a small subset of metrics that capture the curriculum's latent, non-structural features. In the original framework, instructional complexity was entirely proxied by individual-course pass rates to capture other aspects of the curriculum, including student support structures, course difficulty, and instructional quality (Heileman et al., 2018). Most papers did not expand on this idea and continued to use previous grades, grades of D and F in addition to withdrawals (DFW) rates, or pass rates - all of which refer to the same underlying concept.

The use of pass rates to measure instructional complexity primarily serves to simulate student progression through a given program. For example, Molontay et al. (2020) used discrete-event simulation to model the expected graduation time based on both the curriculum's topological structure and course-level completion probabilities. The model uses the partial derivative of a function f(p₁, p₂, …, p_n) representing the expected graduation time when one of the pass rates is changed, p_i.

The value of D_i must be estimated empirically or, more realistically, through simulation. This metric complements traditional structural metrics by incorporating student performance, allowing us to pinpoint bottlenecks to student progress by simulating their effect on student flow through the curriculum.

An extension of the idea of pass rates that does not require simulations is student mobility turbulence (SMT; Basavaraj, 2020). This metric captures volatility in student progression by analyzing patterns of program withdrawals and major changes, which can indicate structural barriers or curriculum inefficiencies. The SMT metric is formulated as follows.

where drop rate (DR) denotes the number of students who withdrew from the program at the end of the i^th term; changed major (CM) represents the number of students who changed their major at the end of the i^th term (assuming into the program); C₁ and C₂ are weighting coefficients; N is the total number of students who withdrew from the program or changed majors (however, it seems more likely it should refer to the total number of enrolled students if both DR and CM are students leaving). The term, n, is the number of terms under analysis (typically 8). The authors suggest that the weights C₁ and C₂ are determined based on empirical observations of how each factor contributes to student turbulence, with typical values set as C₁ = 1 and C₂ = 0.5 in prior applications. For example, if we consider the SMT using a collection of programs, if a cohort of 100 engineering students has 50 students leaving the engineering programs (DR) and 10 major changes among them (CM) over three terms, with C₁ = 1 and C₂ = 0.5, their SMT would be (1 × 50 + 0.5 × 10) /100 = 0.55, suggesting moderate turbulence in student progression. Using the proposed definition provided in (Basavaraj, 2020), we find that SMT would be (1 × 50 + 0.5 × 10) /60 = 0.92 - a significantly higher turbulence value. While no strict thresholds are defined, lower SMT values typically indicate smoother, more stable progress toward degree completion. In contrast, higher values reflect increased academic instability, which can delay graduation. For example, if Program A has a higher SMT than Program B, it suggests that students in Program A experience more disruptions and are progressing less efficiently toward their degrees (Basavaraj, 2020).

There is a smattering of metrics that can further contextualize the instructional complexity of the curriculum beyond pass rates. For example, another metric defined in the literature, grade anomaly (GA; Waller, 2022), is a measure of students' performance in specific courses relative to their performance in their other courses. It is formulated as:

Here, GA_ij is the grade anomaly of student i in course j, which is the difference between the student's grade from the focal course, G_ij, and their overall GPA, GPA_ij. Note that grade anomaly is at the individual student level. At the course level, course grade anomaly (CGA) is the mean discrepancy between the students' GPAs and their grades for a given course:

where N is the number of students.

Expanding on the concept of CGA is the course toxicity metric (Thompson-Arjona, 2019). Course toxicity quantifies the impact of taking particular courses together on student success. It is calculated by comparing the probability of passing a course individually with that of passing it when taken alongside another course. The toxicity score T(v_ij, v_kj) of taking courses v_ij and v_kj together is defined as:

where X_ij is defined as the event that a student passes course v_ij on the first attempt. This metric captures how sensitive the expected graduation time is to changes in the success rate of course i, allowing institutions to prioritize interventions in courses with high pass-through impacts. The model complements traditional structural metrics by incorporating dynamic, data-driven estimates of course completion and simulating their effect on student flow through the curriculum.

Moving beyond measures of student progression, the literature has also examined how closely students follow recommended course pathways. For example, the conformity score (CS; Backenköhler et al., 2018) quantifies the alignment between the courses students take and those recommended by academic advisors or degree plans. Formally, this metric is defined as follows.

where C_sel^i,t represents the set of courses the student i selects in term t, C_rec^i,t represents the set of courses recommended for student i in term t; k is the number of recommended courses to consider (typically 4-6), τ is the set of all terms, and s_t is the set of students enrolled in term t. A higher CS suggests stronger adherence to the recommended pathway. Personal preferences, scheduling conflicts, or course availability can lead to lower scores when students do not follow the course recommendations. For example, consider a scenario in which three courses are recommended by advisors (k = 3) :{Math101, Physics101, Chemistry101}. A student then enrolls in {Math101, Physics101, Biology101}, where two of the three courses are the same. The CS for this term would be 1- (3 - 2)/3 = 0.67, indicating moderate adherence to the recommendations. This metric can help evaluate the effectiveness of advising strategies and uncover patterns in which students are navigating the curriculum differently than intended.

Blending structural and instructional features

In our sample, we found one metric that combined instructional and structural factors, curriculum stringency (Basavaraj, 2020). This metric uses the prerequisite relationships in the form of a sum of proportions relative to the number of vertices, evoking a similar idea to curricular rigidity, summed with a factor called DM_psf.

The DM_psf factor in the equation is a "difficulty metric consider[ing] the average number of times a student had taken the exam (Avg_taken), the average number of times a student had failed (Avg_failed), and the average grade of the exam (Avg_grade)" (Basavaraj, 2020, p. 38). In this case, "exam" refers to program-specific exams outside the context of specific courses.

Curriculum stringency combines the structural features of the network with instructional features, such as student performance on non-course-related exams. Though for many programs, such features do not necessarily apply, so DM_psf = 0, and the metric becomes entirely structural.