Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers

Long Cheng, Ying Wang, Qingzhi Liu, Dick H.J. Epema, Cheng Liu, Ying Mao, John Murphy

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Large data centers are currently the mainstream infrastructures for big data processing. As one of the most fundamental tasks in these environments, the efficient execution of distributed data operators (e.g., join and aggregation) are still challenging current data systems, and one of the key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer data locality optimization to reduce network traffic or on network-layer data flow optimization to increase bandwidth utilization. However, the techniques in the two layers are totally independent from each other, and performance gains from a joint optimization perspective have not yet been explored. In this paper, we propose a novel approach called NEAL (NEtwork-Aware Locality scheduling) to bridge this gap, and consequently to further reduce communication time for distributed big data operators. We present the detailed design and implementation of NEAL, and our experimental results demonstrate that NEAL always performs better than current approaches for different workloads and network bandwidth configurations.

Original languageEnglish
Pages (from-to)1494-1510
JournalIEEE Transactions on Parallel and Distributed Systems
Volume32
Issue number6
DOIs
Publication statusPublished - 20 Jan 2021

Keywords

  • big data
  • coflow scheduling
  • data centers
  • data locality
  • distributed operators
  • metaheuristic
  • SDN

Fingerprint Dive into the research topics of 'Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers'. Together they form a unique fingerprint.

Cite this