BERT-ALPHA: BERT for Arabic Language Processing with Hybrid Architecture

Waseem Saad Nsaif, Saja Salim Mohammed, Musaab M. Jasim, Ahmed Abbas Brisam, Hassan Hadi Saleh, Dheyab Salman Ibrahim, Israa Mishkal, Hassan Al-Mahdawi and Salam Abdulkhaleq Noaman

References:

1. W. S. Nsaif, H. M. Salih, H. H. Saleh, and B. T. Al-Nuaimi, “Chatbot Development: Framework, Platform, and Assessment Metrics,” The Eurasia Proceedings of Science Technology Engineering and Mathematics, vol. 27, pp. 50-62, 2024, [Online]. Available: https://doi.org/10.55549/epstem.1518314.
2. W. S. Nsaif, H. M. Salih, H. H. Saleh, and B. T. Al-Nuaimi, “Conversational agents: An exploration into Chatbot evolution, architecture, and important techniques,” The Eurasia Proceedings of Science, Technology, Engineering & Mathematics, vol. 27, pp. 246-262, 2024.
3. W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” arXiv preprint arXiv:2003.00104, 2020.
4. K. Darwish, N. Habash, M. Abbas, et al., “A panoramic survey of natural language processing in the Arab world,” Communications of the ACM, vol. 64, no. 4, pp. 72-81, 2021.
5. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
6. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” arXiv preprint arXiv:1806.03822, 2018.
7. O. Obeid, N. Zalmout, S. Khalifa, et al., “CAMeL tools: An open-source toolkit for Arabic NLP,” in Proceedings of LREC 2020, pp. 7022-7032, 2020.
8. Y. Liu, M. Ott, N. Goyal, et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
9. M. Huang and X. Zhu, and J. Gao, “Challenges in building intelligent open-domain dialog systems,” ACM Transactions on Information Systems, vol. 38, no. 3, pp. 1-32, 2020.
10. L. Li et al., “Deep context modeling for multi-turn response selection in dialogue systems,” Information Processing & Management, vol. 58, no. 1, p. 102415, 2021.
11. X. Feng et al., “Language model as an annotator: Exploring DialoGPT for dialogue summarization,” arXiv preprint arXiv:2105.12544, 2021.
12. T. N. Alruqi and S. M. Alzahrani, “Evaluation of an Arabic chatbot based on extractive question-answering transfer learning and language transformers,” AI, vol. 4, no. 3, pp. 667-691, 2023.
13. M. Boussakssou, H. Ezzikouri, and M. Erritali, “Chatbot in Arabic language using seq-to-seq model,” Multimedia Tools and Applications, vol. 81, no. 2, pp. 2859-2871, 2022.
14. T. Ait Baha et al., “The impact of educational chatbots on student learning experience,” Education and Information Technologies, vol. 29, no. 8, pp. 10153-10176, 2024.
15. N. Habash, Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers, 2010.
16. M. T. Diab et al., “Challenges in processing dialectal Arabic,” Computational Linguistics, vol. 40, no. 1, pp. 55-85, 2014.
17. M. Abdul-Mageed et al., “QADI: An annotated dataset for Arabic dialect identification,” in Proceedings of NAACL-HLT 2021, 2021.
18. I. Zeroual et al., “Tashkeela: A novel dataset for Arabic diacritization,” Language Resources and Evaluation, 2019.
19. O. F. Zaidan and C. Callison-Burch, “The Arabic Online Commentary Dataset: An annotated dataset of informal Arabic with high dialectal content,” in Proceedings of ACL 2011, 2011.
20. H. Bouamor et al., “A multidialectal parallel corpus of Arabic,” in Proceedings of LREC 2014, 2014.
21. H. Mubarak et al., “Building a dialogue system for Modern Standard Arabic and dialects,” Computational Linguistics, vol. 46, no. 3, pp. 555-579, 2020.
22. I. Mishkhal, N. Abdullah, H. H. Saleh, N. I. R. Ruhaiyem, and F. H. Hassan, “Facial Swap Detection Based on Deep Learning: Comprehensive Analysis and Evaluation,” Iraqi Journal for Computer Science and Mathematics, vol. 6, no. 1, Art. 8, 2025.
23. J. Devlin et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
24. W. Antoun et al., “AraBERT: Transformer-based model for Arabic language understanding,” arXiv preprint arXiv:2003.00104, 2020.
25. Y. Zhang et al., “DialoGPT: Large-scale generative pre-training for conversational response generation,” in Proceedings of ACL 2020, 2020.
26. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 1, pp. 1-67, 2020.
27. K. Darwish et al., “Addressing tokenization challenges in Arabic NLP,” Transactions on Asian and Low-Resource Language Information Processing, vol. 20, no. 3, pp. 1-18, 2021.
28. I. Jibril et al., “QARiB: A contextualized model for Arabic NLP tasks,” in Proceedings of the 2022 International Conference on NLP and AI, 2022.
29. I. Sutskever et al., “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems, 2014.
30. K. Cho et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in Proceedings of EMNLP 2014, 2014.
31. S. Alhumoud et al., “Enhancing Arabic chatbots using Transformer-RNN hybrid architectures,” Journal of Artificial Intelligence Research, vol. 68, no. 4, pp. 115-132, 2023.
32. A. Farghaly and K. Shaalan, “Arabic natural language processing: Challenges and solutions,” ACM Transactions on Asian Language Information Processing, vol. 8, no. 4, pp. 1-22, 2009.
33. T. Al-Hanai et al., “Optimizing Arabic dialogue systems using reinforcement learning and transformer architectures,” Computational Intelligence, vol. 38, no. 2, pp. 312-330, 2022.
34. A. A. Hussein, K. M. Hussein, H. H. Saleh, and I. H. Farhan, “Survey towards a sustainable information and communication technologies (ICT) in Iraq,” Journal of Physics: Conference Series, vol. 1530, no. 1, p. 012089, May 2020.
35. T. N. Alruqi and S. M. Alzahrani, “Evaluation of an Arabic chatbot based on extractive question-answering transfer learning and language transformers,” AI, vol. 4, no. 3, pp. 667-691, 2023.
36. J. Lafferty et al., “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in Proceedings of ICML, 2001.
37. O. Obeid et al., “CAMeL tools: An open-source toolkit for Arabic NLP,” in Proceedings of LREC 2020, 2020.
38. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
39. F. Fernández-Martínez, C. Luna-Jiménez, R. Kleinlein, D. Griol, Z. Callejas, and J. M. Montero, “Fine-tuning BERT models for intent recognition using a frequency cut-off strategy for domain-specific vocabulary extension,” Applied Sciences, vol. 12, no. 3, p. 1610, 2022.
40. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
41. H. H. Saleh, W. Nsaif and L. Rashed, “Design and implementation of a web-based collaborative E-learning model: A case study — Computer Science Department Curriculum,” in 2018 1st Annual International Conference on Information and Sciences (AiCIS), Nov. 2018, pp. 193-200.
42. C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Proceedings of the Workshop on Text Summarization, 2004.
43. I. Chalkiadakis, G. W. Peters, and M. Ames, “Hybrid ARDL-MIDAS-Transformer time-series regressions for multi-topic crypto market sentiment driven by price and technology factors,” Digital Finance, vol. 5, no. 2, pp. 295-365, 2023.
44. D. Peras, “Chatbot evaluation metrics,” Economic and Social Development: Book of Proceedings, pp. 89-97, 2018.
45. S. K. Yuwono, B. Wu, and L. F. D’Haro, “Automated scoring of chatbot responses in conversational dialogue,” in 9th International Workshop on Spoken Dialogue System Technology, pp. 357-369, Springer, Singapore, 2019.
46. S. M. Suhaili, N. Salim, and M. N. Jambli, “Service chatbots: A systematic review,” Expert Systems with Applications, vol. 184, p. 115461, 2021.