Beyond CIPP: Validating the CIPPRI framework: A mixed-methods stakeholder-driven evaluation of an electrical engineering program in an emerging economy

Abstract

Conventional Context–Input–Process–Product (CIPP) evaluations have been widely adopted across higher-education quality-assurance regimes; however, they tend to subsume reflective and improvement-oriented practices within process or product metrics, rendering these mechanisms structurally invisible. This study advances and empirically validates an extended evaluation framework—CIPPRI—which disaggregates Reflection (R) and Improvement (I) into two dimensions analytically distinct from the original four. The framework was applied to evaluate a Bachelor of Engineering (Electrical Engineering) program at a Thai public university through a sequential explanatory mixed-methods design. Quantitative data were collected from 240 internal and external stakeholders (students n = 120, alumni n = 80, employers n = 25, instructors n = 11, and administrators n = 4) using a six-dimension, 42-item Likert-scale instrument. Construct validity was established through Item-Objective Congruence (IOC = 0.67–1.00), exploratory factor analysis (KMO = 0.91; Bartlett χ² = 5,842.34, p < .001), and confirmatory factor analysis supporting a six-factor solution (CFI = 0.94, TLI = 0.93, RMSEA = 0.058, SRMR = 0.047). Internal consistency was strong (Cronbach α = 0.89–0.94; McDonald ω = 0.91–0.95). Quantitative data were analyzed using descriptive statistics, Kruskal–Wallis tests, and post-hoc Dunn–Bonferroni comparisons; qualitative data from semi-structured interviews (n = 18) and focus groups (n = 4) were thematically analyzed. The overall program was rated favourably, with the process dimension highest (Weighted M = 4.21, Pooled SD = 0.79) and the input dimension lowest (Weighted M = 3.83, Pooled SD = 0.77), the latter constrained by obsolete laboratory infrastructure and limited smart-grid resources—a bottleneck consistent with the broader literature on emerging economies. Crucially, while reflection and improvement scored highly in self-report (M = 4.15, M = 4.05), thematic analysis revealed these practices were rated as latent rather than systemically institutionalized. Kruskal–Wallis tests indicated significant divergence across stakeholder groups for the Input (H = 14.62, p = .006, ε² = 0.06) and Improvement (H = 11.34, p = .023, ε² = 0.04) dimensions. The study contributes (a) a psychometrically validated six-factor evaluation instrument; (b) empirical evidence that explicit reflection-improvement dimensions surface evaluation gaps invisible to traditional CIPP; and (c) a transferable framework for engineering programs navigating Industry 4.0/5.0 transitions in resource-constrained settings.

Keywords: Program evaluation, CIPP model, reflective practice, continuous improvement, engineering education, mixed-methods, confirmatory factor analysis, Thailand, quality assurance.

This article is published under the terms of the Creative Commons Attribution License 4.0