Improving Job Scheduling in Hadoop

Hadoop is an open-source software framework. Hadoop is used for storage and large-scale processing of data-sets on clusters of commodity hardware. MapReduce is a one type of programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. Scheduling is required in hadoop for Time Optimization and CPU Utilization. By default Hadoop uses FIFO, and optional 5 scheduling priorities to schedule jobs from a work queue. Various scheduling algorithms such as FIFO, Fair and Capacity are good as some time means if load is low then Fair scheduler are good one but if load increased at some level then capacity scheduler are good choice. All this scheduler having some problem. If jobs are scheduled in FIFO order, it easily causes the starvation of small jobs in the event of resources being utilized by large jobs, while Fair Scheduler is inefficient when handling large jobs and it leads to sticky slots problem. For resolving such issue some customized algorithms has been developed by enhancing existing algorithms. Now our work is to enhance the scheduling algorithm for task allocation by minimizing the process time and more resource utilization. We will improve TDWS (Tencent Distributed Data Warehouse Schedulling) algorithm by including parameters such as CPU Utilization, Free Memory (RAM), Load average of the resources.