The fast growth of malware and the continuous development of new techniques to hide malicious behavior have made traditional detection methods, such as signature-based and heuristic approaches, less effective. Today’s cybersecurity needs systems that are accurate, scalable, and capable of dealing with new and unknown types of attacks. Deep learning models such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) have shown strong ability in analyzing API call sequences and detecting malware behavior. However, when these models are trained on small or unbalanced datasets, they often suffer from overfitting and weak generalization. In addition, many previous studies did not include external features from services like Hybrid Analysis, or they used costly transformations such as converting binaries into images, which reduced the practical value of their systems. To address these problems, we propose a hybrid CNN–LSTM framework that combines API sequence analysis with external features. API calls are integer-encoded to preserve their order, while extra attributes such as malicious ratios, detection counts, and threat scores were collected through the official API of Hybrid Analysis system. A variety of optimization methods were used such as dropout, token dropout, batch normalization, L2 regularization, adaptive learning-rate scheduling with AdamW and stratified dataset splitting. These techniques increased the generalization, training stability and decreased overfitting. It was tested on 3 datasets, Dataset 1 (MalBehavD-V1 + Oliveira 3,500 samples), Dataset 2 (the same dataset with external features with Hybrid Analysis API), and Dataset 3 (a large scale dataset with 2025 85,594 balanced samples). The proposed model achieved 95.43% accuracy on Dataset 1, 97.49% on Dataset 2, and 99.52% on Dataset 3, consistently surpassing standalone CNN and LSTM baselines in accuracy and AUC. These results demonstrate that combining behavioral API sequences with lightweight external features yields a precise, scalable malware detection system suitable for practical cybersecurity deployments.
P. Maniriho, A. N. Mahmood, and M. Kim, “EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls,” arXiv preprint arXiv:2407.13355, 2024.
A. Hussain, A. Saadia, M. Alhussein, A. Gul, and K. Aurangzeb, “Enhancing Ransomware Defense: Deep Learning-Based Detection and Family-Wise Classification of Evolving Threats,” 2025.
Y. Song, D. Zhang, J. Wang, Y. Wang, Y. Wang, and P. Ding, “Application of Deep Learning in Malware Detection: A Review,” Journal of Big Data, vol. 12, no. 1, p. 75, 2025.
S. S. Bamber, A. V. R. Katkuri, S. Sharma, and M. Angurala, “A hybrid CNN–LSTM approach for intelligent cyber intrusion detection system,” Computers & Security, vol. 148, p. 104146, Oct. 2025.
P. Thakur, V. Kansal, and V. Rishiwal, “Hybrid deep learning approach based on LSTM and CNN for malware detection,” Wireless Personal Communications, vol. 136, pp. 1879-1901, Jun. 2024.
M. Alshomrani, A. Albeshri, A. A. Alsulami, and B. Alturki, “An Explainable Hybrid CNN–Transformer Architecture for Visual Malware Classification,” Sensors, vol. 25, no. 15, p. 4581, 2025.
L. Qian and L. Cong, “Channel Features and API-Frequency-Based Transformer (CAFTrans) for Malware Identification,” Sensors, vol. 24, no. 2, p. 580, 2024.
G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, S. V. G., and P. Poornachandran, “CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection,” Procedia Computer Science, vol. 233, pp. 492-503, 2024.
C. Li, Q. Lv, N. Li, Y. Wang, D. Sun, and Y. Qiao, “A Novel Deep Framework for Dynamic Malware Detection Based on API Sequence Intrinsic Features,” Computers & Security, vol. 116, p. 102686, 2022.
C. Miller, T. Portlock, D. M. Nyaga, and J. M. O’Sullivan, “A Review of Model Evaluation Metrics for Machine Learning in Genetics and Genomics,” Frontiers in Bioinformatics, vol. 4, p. 1457619, 2024.
M. D. Rahman, SMOTE API Call Sequence Dataset, Kaggle repository, 2025. , [Online]. Available: https://www.kaggle.com/datasets/marahmanju/smote-api-call-sequence-dataset.
L. Cardoso, V. Santos, J. Ribeiro, R. Kawasaki, R. Prudêncio, and R. Alves, “Enhancing Classifier Evaluation: A Fairer Benchmarking Strategy Based on Ability and Robustness,” arXiv preprint arXiv:2504.09759, 2025.
R. J. Kolaib and J. Waleed, “Crime Activity Detection in Surveillance Videos Based on Developed Deep Learning Approach,” Diyala Journal of Engineering Sciences, vol. 17, no. 3, pp. 98-114, Sep. 2024.
S. Beddar-Wiesing, A. Moallemy-Oureh, M. Kempkes, and J. M. Thomas, “Absolute Evaluation Measures for Machine Learning: A Survey,” arXiv preprint arXiv:2507.03392, 2025.