Hadoop Acceleration using Open Flow Technique

This paper is about of how Hadoop can control its network resources to improve performance using OpenFlow. . OpenFlow is a very popular example of software-defined network technology. Our study will explore the use of OpenFlow to provide better link bandwidth for shuffle traffic and decrease the time which Reducers have to wait to gather data from the Mappers. Hadoop's distributed compute framework which is called as MapReduce, exploits the distributed storage architecture for Hadoop's distributed file system which will deliver a reliable parallel processing services for arbitrary algorithms. The shuffle phase of Hadoop's MapReduce are the computation involves movement of intermediate data from the Mappers to Reducers. Reducers often are delayed due to inadequate bandwidth between them and thereby lower the performance of the cluster. Our experiments shows the decrease of execution time for a Hadoop job and when the shuffle traffic can be used for more of the available bandwidth on a link. Our approach illustrates how high performance of the computing applications can improve performance by controlling their underlying network resources. The work presented in the paper is for a starting point for some experiments being done as part of SC12 SCinet Research Sandbox which will be quantified the performance advantages of different versions of Hadoop that uses OpenFlow to dynamically adjust the network of local and wide area Hadoop clusters.