Integrated Deadline Scheduling with MapReduce Tasks in Hadoop

The MapReduce model implemented by Hadoop is the standard platform for applications processing Big Data. Schedulers are critical in enhancing the performance of MapReduce/Hadoop with limited computing resources in presence of multiple jobs with different characteristics and performance goals. The commonly used schedulers include the First In First Out (FIFO), Capacity, Fair Scheduler etc. But none of the above schedulers consider the strong dependency between the map and reduce tasks. They fail to jointly optimize the placement of map and reduce tasks. They even ignore the data locality for reduce tasks and the deadline constraints for jobs. The Fair Scheduler is not fair to reduce tasks and might cause starvation when allocating excess resources to reduce task without jointly optimizing with map task. The delay scheduling might lead to under utilization of resources at times. Considering all of the above problems, we propose a resource aware Integrated Preemptive Deadline Scheduler with the MapReduce Task Coupling which jointly optimizes the map and reduce tasks by coupling their progress and ensures deadline completion of jobs. The integrated approach ensures fast and under deadline completion of jobs. It mitigates job starvation and makes better slot utilization for reducing the total completion time of jobs. Thus the integrated scheduler improves data locality, slot utilization, mitigates starvation and ensures under deadline completion.