In the context of the digital transformation of lifelong education, adaptive learning systems (ALS) based on deep learning algorithms are becoming a key tool for personalizing educational trajectories. Despite the significant growth in related publications, the issue of the multilevel effectiveness of such systems under rigorously controlled experimental designs remains insufficiently explored. The objective of this study is to develop and empirically validate a multilevel model for evaluating the effectiveness of deep neural network-based adaptive learning systems using a randomized controlled trial (RCT) design. A total of 842 university students participated in the study and were randomly assigned to an experimental group (n = 421) and a control group (n = 421). The experimental group used an adaptive learning system built on a hybrid Transformer-LSTM architecture with an attention mechanism, whereas the control group studied through a traditional digital learning platform without adaptive algorithms. Hierarchical Linear Modeling (HLM) was employed to assess individual, group-level, and institutional effects. The results demonstrated a statistically significant improvement in academic performance (β = 0.47, p < 0.001), cognitive persistence (β = 0.39, p < 0.01), and metacognitive regulation (β = 0.42, p < 0.01) in the experimental group. The effect size (Cohen’s d = 0.68) indicates moderate-to-high practical significance. Multilevel analysis revealed that 27.0% of the variance in learning outcomes was attributable to between-group differences, highlighting the necessity of multilevel statistical modeling. The findings support the hypothesis regarding the high effectiveness of deep learning-based adaptive systems and provide a methodological foundation for their large-scale implementation in higher education institutions.
D. Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861-874, 2006.
C. Hodges, S. Moore, B. Lockee, and T. Trust, “The difference between emergency remote teaching and online learning,” Educause Review, vol. 27, no. 1, pp. 1-12, 2020.
B. S. Bloom, “The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring,” Educational Researcher, vol. 13, no. 6, pp. 4-16, 1984.
G. Westerman, D. Bonnet, and A. McAfee, Leading Digital. Boston: Harvard Business Review Press, 2014.
J. Hattie, Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. London: Routledge, 2009.
E. Wenger, Artificial Intelligence and Tutoring Systems. Los Altos, CA: Morgan Kaufmann, 1987.
J. Bruner, Toward a Theory of Instruction. Cambridge, MA: Harvard University Press, 1966.
A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253-278, 1995.
M. Martin and H. Rubin, “A new measure of cognitive flexibility,” Psychological Reports, vol. 76, pp. 623-626, 1995.
C. Piech et al., “Deep knowledge tracing,” in Advances in Neural Information Processing Systems (NeurIPS), pp. 505-513, 2015.
A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), pp. 5998-6008, 2017.
J. Nunnally and I. Bernstein, Psychometric Theory, 3rd ed. New York: McGraw-Hill, 1994.
S. D’Mello and A. Graesser, “Dynamics of affective states during complex learning,” Learning and Instruction, vol. 22, no. 2, pp. 145-157, 2012.
S. W. Raudenbush and A. S. Bryk, Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd ed. Thousand Oaks, CA: Sage, 2002.
A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge: Cambridge University Press, 2007.
S. W. Raudenbush, A. S. Bryk, and R. Congdon, HLM 7: Hierarchical Linear and Nonlinear Modeling. Lincolnwood, IL: Scientific Software International, 2011.
L. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika, vol. 16, no. 3, pp. 297-334, 1951.
T. Cook and D. Campbell, Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin, 1979.
R. Little and D. Rubin, Statistical Analysis with Missing Data, 2nd ed. New York: Wiley, 2002.
R. Slavin, “Evidence-based education policies,” Educational Researcher, vol. 31, no. 7, pp. 15-21, 2002.
Y. Bengio, I. Goodfellow, and A. Courville, Deep Learning. Cambridge, MA: MIT Press, 2016.
H. Goldstein, Multilevel Statistical Models, 4th ed. Chichester: Wiley, 2011.
P. Austin, “Balance diagnostics for comparing groups,” Statistics in Medicine, vol. 28, pp. 3083-3107, 2009.
J. Shadish, T. Cook, and D. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin, 2002.
B. J. Zimmerman, “Becoming a self-regulated learner,” Theory Into Practice, vol. 41, no. 2, pp. 64-70, 2002.
T. Snijders and R. Bosker, Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage, 2012.
S. Dennis and R. Vander Wal, “The cognitive flexibility inventory,” Cognitive Therapy and Research, vol. 34, no. 3, pp. 241-253, 2010.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
European Parliament, “General Data Protection Regulation (GDPR),” Official Journal of the European Union, 2016.
R. E. Mayer, Multimedia Learning, 2nd ed. Cambridge: Cambridge University Press, 2009.
J. Sweller, “Cognitive load theory,” Psychology of Learning and Motivation, vol. 55, pp. 37-76, 2011.
F. Faul et al., “G*Power 3: A flexible statistical power analysis program,” Behavior Research Methods, vol. 39, pp. 175-191, 2007.
S. Hochreiter, “The vanishing gradient problem during learning recurrent neural nets,” International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 6, no. 2, pp. 107-116, 1998.
D. Haladyna and M. Rodriguez, Developing and Validating Test Items, 3rd ed. New York: Routledge, 2013.
J. Tondeur et al., “Teachers’ readiness for digital innovation,” Educational Technology & Society, vol. 20, no. 3, pp. 1-14, 2017.
P. Pintrich et al., “A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ),” University of Michigan, 1991.
H. Akaike, “A new look at the statistical model identification,” IEEE Transactions on Automatic Control, vol. 19, no. 6, pp. 716-723, 1974.
J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum, 1988.
R. Preacher and A. Hayes, “Asymptotic and resampling strategies for assessing mediation,” Behavior Research Methods, vol. 40, no. 3, pp. 879-891, 2008.
Z. C. Lipton, “The mythos of model interpretability,” Communications of the ACM, vol. 61, no. 10, pp. 36-43, 2018.
H. Qian, M. Wang, and Y. Xiong, “Digital transformation in higher education: A systematic review,” DOAJ Journal, vol. 9, pp. 56-70, 2023.
