A Hybrid Deep Learning Model for Facial Emotion Recognition: Combining Multi-Scale Features, Dynamic Attention, and Residual Connections

Muthana Salih Mahdi, Zaydon Latif Ali, Ahmed Ramzi Rashid, Noor Khalid Ibrahim and Abdulghafor Waedallah Abdulghafour

References:

1. M. K. Chowdary, T. N. Nguyen, and D. J. Hemanth, "Deep learning-based facial emotion recognition for human–computer interaction applications," Neural Comput. Appl., vol. 35, no. 32, pp. 23311-23328, Nov. 2023, [Online]. Available: https://doi.org/10.1007/s00521-021-06012-8.
2. P. Yu, X. He, H. Li, H. Dou, Y. Tan, H. Wu, and B. Chen, "FMLAN: A novel framework for cross-subject and cross-session EEG emotion recognition," Biomed. Signal Process. Control, vol. 100, article 106912, 2025, [Online]. Available: https://doi.org/10.1016/j.bspc.2024.106912.
3. A. L. Cîrneanu, D. Popescu, and D. Iordache, "New trends in emotion recognition using image analysis by neural networks, a systematic review," Sensors, vol. 23, no. 16, p. 7092, Aug. 2023, [Online]. Available: https://doi.org/10.3390/s23167092.
4. E. Hato, "Extracting descriptive frames from informational videos," Iraqi J. Sci., vol. 64, no. 4, pp. 4260-4277, 2023, [Online]. Available: https://doi.org/10.24996/ijs.2023.64.8.43.
5. S. Hazmoune and F. Bougamouza, "Using transformers for multimodal emotion recognition: Taxonomies and state of the art review," Eng. Appl. Artif. Intell., vol. 133, article 108339, 2024, [Online]. Available: https://doi.org/10.1016/j.engappai.2024.108339.
6. Y. M. Mohialden, N. M. Hussien, Q. A. Jabbar, M. A. Mohammed, and T. Sutikno, "An internet of things-based medication validity monitoring system," Indones. J. Electr. Eng. Comput. Sci., vol. 26, no. 2, pp. 932-938, 2022, [Online]. Available: http://doi.org/10.11591/ijeecs.v26.i2.pp932-938.
7. A. Basim and A. Sadiq, "Enhancement of low light images using residual deep learning," in Proc. Int. Conf. Innov. Intell. Informatics, Netw. Cybersecurity, Cham, Switzerland: Springer Nature, Oct. 2024, pp. 119-132, [Online]. Available: https://doi.org/10.1007/978-3-031-81065-7_8.
8. M. S. Mahdi and Z. L. Ali, "A lightweight algorithm to protect the web of things in IoT," in Emerging Technology Trends in Internet of Things and Computing, Cham, Switzerland: Springer Int. Publ., Mar. 2022, pp. 46-60, [Online]. Available: https://doi.org/10.1007/978-3-030-97255-4_4.
9. H. R. Shakir, "Secure selective image encryption based on wavelet domain, 3D‑chaotic map, and discrete fractional random transform," Int. J. Intell. Eng. Syst., vol. 16, no. 6, pp. 85-98, 2023, [Online]. Available: https://doi.org/10.22266/ijies2023.1231.80.
10. A. R. Rashid, Z. L. Ali, and G. H. A. Alshmeel, "Developing an artificial intelligence-based system to detect fraud and corruption in government," in Proc. 2022 Int. Congr. on Human–Computer Interaction, Optimization and Robotic Applications (HORA), Baghdad, Iraq: IEEE, Jun. 2022, pp. 1-8, [Online]. Available: https://doi.org/10.1109/HORA55278.2022.9799900.
11. M. Jagadeesh and G. Baranidharan, "Dynamic FERNet: Deep learning with optimal feature selection for face expression recognition in video," Concurrency Comput. Pract. Exp., vol. 34, no. 5, May 2022, [Online]. Available: https://doi.org/10.1002/cpe.7373.
12. B. Fu, Y. Mao, S. Fu, Y. Ren, and Z. Luo, "Blindfold attention: Novel mask strategy for facial expression recognition," in Proc. ACM Comput. Graph. Interact. Tech., New York, USA, 2022, pp. 1-10, [Online]. Available: https://doi.org/10.1145/3512527.3531416.
13. V. X. Chi and P. C. Vinh, "Facial sentiment recognition using artificial intelligence techniques," EAI Endorsed Trans. Context-Aware Syst. Appl., vol. 10, no. 3, pp. 112-120, 2023, [Online]. Available: https://doi.org/10.4108/eetcasa.v9i1.3930.
14. Y. Chen and M. Zhang, "Research on face emotion recognition algorithm based on deep learning neural network," Appl. Math. Nonlinear Sci., vol. 8, no. 1, pp. 59-67, 2023, [Online]. Available: https://doi.org/10.2478/amns.2023.2.00533.
15. H. Shahzad, S. Bhatti, A. Jaffar, M. Rashid, and S. Akram, "Multi-modal CNN features fusion for emotion recognition: A modified Xception model," IEEE Access, vol. 11, pp. 12345-12356, 2023, [Online]. Available: https://doi.org/10.1109/ACCESS.2023.3310428.
16. X. Wang, Y. Wang, and D. Zhang, "Complex emotion recognition via facial expressions with label noises self-cure relation networks," Comput. Intell. Neurosci., vol. 2023, article 987654, 2023, [Online]. Available: https://doi.org/10.1155/2023/7850140.
17. S. Wang, J. Qu, Y. Zhang, and Y. Zhang, "Multimodal emotion recognition from EEG signals and facial expressions," IEEE Access, vol. 11, pp. 23456-23468, 2023, [Online]. Available: https://doi.org/10.1109/access.2023.3263670.
18. R. Tshibangu and J. R. Tapamo, "Improving facial emotional recognition using convolution neural network with minimal layers," in Electronics, Communications, and Networks, Amsterdam, The Netherlands: IOS Press, 2024, pp. 672-682, [Online]. Available: https://doi.org/10.3233/FAIA231252.
19. J. Pan, W. Fang, Z. Zhang, B. Chen, Z. Zhang, and S. Wang, "Multimodal emotion recognition based on facial expressions, speech, and EEG," IEEE Open J. Eng. Med. Biol., vol. 5, pp. 396-403, 2024, [Online]. Available: https://doi.org/10.1109/OJEMB.2023.3240280.
20. Z. Wu and D. Pan, "The application and optimization of deep learning in recognizing student learning emotions," Trait. Signal, vol. 41, no. 1, pp. 391-399, 2024, [Online]. Available: https://doi.org/10.18280/ts.410133.
21. Face expression recognition dataset, Kaggle, 2017, [Online]. Available: https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset.
22. Facial emotion recognition dataset, Kaggle, 2018, [Online]. Available: https://www.kaggle.com/datasets/chiragsoni/ferdata.
23. Facial Emotion dataset, Kaggle, 2019, [Online]. Available: https://www.kaggle.com/datasets/mh0386/facial-emotion.