Huge Data Transfer Across Distributed Data Centers

A challenge rise on how to schedule the huge data transfers at different importunity levels, in order to fully utilize the available inter-data center bandwidth. The Software Defined Networking (SDN) paradigm has issued recently which decouples the control plane from the data paths, enabling potential global optimization of data routing in a network. This paper aims to design a dynamic, highly efficient huge data transfer service in a distributed data center system, and engineer its design and solution algorithms nearly within SDN architecture. We model data transfer demands as delay tolerant migration requests with different completing deadlines. The tractability provided by SDN, we enable dynamic, optimal routing of distinct clump within each huge data transfer (instead of treating each transfer as an infinite flow), which can be impermanent stored at intermediate data canter’s to mitigate bandwidth contention with more urgent transfers. An optimal clump routing optimization model is formulated to solve for the best clump transfer schedules over time. To derive the optimal schedules in an online fashion, three algorithms are discussed, namely a bandwidth-reserving algorithm, a dynamically-adjusting algorithm, and a future-demand-friendly algorithm, targeting at different levels of optimality and scalability. We build an SDN system based on the Beacon platform and Open Flow APIs, and carefully engineer our huge data transfer algorithms in the system. Extensive real-world experiments are carried out to compare the three algorithms as well as those from the existing literature, in terms of routing optimality, computational delay and overhead.