Proceedings of International Conference on Applied Innovation in IT
2022/03/09, Volume 10, Issue 1, pp.1-6

Cost-Effective High Performance Distributed GPU Cluster for Deep Learning Tasks


Kirill Karpov, Dmitry Kachan, Maksim Iushchenko, Ivan Luzianin, Eduard Siemens


Abstract: The expenses on computational resources for modern Deep Learning computing can be extremely large. However, most of them are spent on the chassis and not on the GPU units themselves. Since modern mass market

Keywords: Tensor Flow, DNN Training, Performance, Horovod, HPC, High-Performance Computing

DOI: 10.25673/76925

Download: PDF

References:

  1. "RDMA Aware Networks Programming User Manual," Mellanox Technologies, Tech. Rep. Rev 1.7, 2015. [Online]. Available:.https:// network.nvidia.com/related-docs/prod software/RDMA Aware Programming user manual.pdf
  2. W. Gropp, E. Lusk, and A. Skjellum, Using MPI: Portable Parallel Programming with the Message Passing Interface. The MIT Press, 11 1999. [Online]. Available: https://doi.org/10.7551/mitpress/7056.001.0001
  3. "Nvidia collective communication library (nccl) documentation," Nvidia Corporation, Specification, 2020. [Online]. Available: https://docs.nvidia.com/deeplearning/sdk/nccl-developer-guide/docs/index.html
  4. A. Sergeev and M. Del Balso, "Horovod: fast and easy distributed deep learning in tensorflow," arXiv preprint arXiv:1802.05799, 2018.
  5. Horovod, "Horovod," https://github.com/horovod/horovod, 2017.
  6. "Intel NUC 9 Extreme/Pro Kit, Techincal Product Specification," Intel Corporation, Specification 1, Dec. 2019. [Online]. Available:https://simplynuc.com/wp-content/uploads/2020/01/NUC9QN TechProdSpec.pdf
  7. "Intel NUC 9 Kit, User Guide," Intel Corporation, Specification 1, May 2020. [Online]. Available: https://www.intel.com/content/dam/support/us/en/documents/intel-nuc/nuc-kits/NUC9xyQNX UserGuide.pdf
  8. "ConnectX-5 EN Card," Mellanox Technologies, Tech. Rep., 2020. [Online]. Available: https://www.mellanox.com/files/doc-2020/pb-connectx-5-en-card.pdf
  9. Z. Zhang, C. Chang, H. Lin, Y. Wang, R. Arora, and X. Jin, "Is network the bottleneck of distributed training?" in Proceedings of the Workshop on Network Meets AI & ML, 2020, pp. 8–13.
  10. J. George, "Qperf 0.4.11 (1)-linux man page," Qperf (1): Measure RDMA/IP Performance.
  11. D. Merkel et al., "Docker: lightweight linux containers for consistent development and deployment," Linux journal, vol. 2014, no. 239, p. 2, 2014.
  12. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean,
  13. M. Devin et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems," arXiv preprint arXiv:1603.04467, 2016.
  14. N. J. Gunther, Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services, 1st ed. Springer Publishing Company, 2010.
  15. F. Alkhoury, D. Wegener, K.-H. Sylla, and M. Mock, "Communication efficient distributed learning of neu-ral networks in big data environments using spark," in 2021 IEEE International Conference on Big Data (Big Data), 2021, pp. 3871–3877.


    HOME

       - Call for Papers
       - Paper Submission
       - For authors
       - Important Dates
       - Conference Committee
       - Editorial Board
       - Reviewers
       - Last Proceedings


    PROCEEDINGS

       - Volume 12, Issue 1 (ICAIIT 2024)        - Volume 11, Issue 2 (ICAIIT 2023)
       - Volume 11, Issue 1 (ICAIIT 2023)
       - Volume 10, Issue 1 (ICAIIT 2022)
       - Volume 9, Issue 1 (ICAIIT 2021)
       - Volume 8, Issue 1 (ICAIIT 2020)
       - Volume 7, Issue 1 (ICAIIT 2019)
       - Volume 7, Issue 2 (ICAIIT 2019)
       - Volume 6, Issue 1 (ICAIIT 2018)
       - Volume 5, Issue 1 (ICAIIT 2017)
       - Volume 4, Issue 1 (ICAIIT 2016)
       - Volume 3, Issue 1 (ICAIIT 2015)
       - Volume 2, Issue 1 (ICAIIT 2014)
       - Volume 1, Issue 1 (ICAIIT 2013)


    PAST CONFERENCES

       ICAIIT 2024
         - Photos
         - Reports

       ICAIIT 2023
         - Photos
         - Reports

       ICAIIT 2021
         - Photos
         - Reports

       ICAIIT 2020
         - Photos
         - Reports

       ICAIIT 2019
         - Photos
         - Reports

       ICAIIT 2018
         - Photos
         - Reports

    ETHICS IN PUBLICATIONS

    ACCOMODATION

    CONTACT US

 

DOI: http://dx.doi.org/10.25673/115729


        

         Proceedings of the International Conference on Applied Innovations in IT by Anhalt University of Applied Sciences is licensed under CC BY-SA 4.0


                                                   This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License


           ISSN 2199-8876
           Publisher: Edition Hochschule Anhalt
           Location: Anhalt University of Applied Sciences
           Email: leiterin.hsb@hs-anhalt.de
           Phone: +49 (0) 3496 67 5611
           Address: Building 01 - Red Building, Top floor, Room 425, Bernburger Str. 55, D-06366 Köthen, Germany

        site traffic counter

Creative Commons License
Except where otherwise noted, all works and proceedings on this site is licensed under Creative Commons Attribution-ShareAlike 4.0 International License.