Big Data Processing: Improve Scheduling Environment in Hadoop

Now a days, dealing with datasets in the order of terabytes or even petabyte is a reality. Therefore, processing such big datasets in an efficient way is a clear need for many users. In this context, Hadoop Map Reduce is a big data processing framework that has rapidly become the de facto standard in both industry and academia. The main reasons of such popularity are the ease-of-use, scalability, and failover properties of Hadoop Map Reduce. However, these features come at a price: the performance of Hadoop Map Reduce is usually far from the performance of a well-tuned parallel database. Therefore, many research works (from industry and academia) have focused on improving the performance of Hadoop Map Reduce jobs in many aspects. For example, researchers have proposed different data layouts; join algorithms, high-level query languages, failover algorithms, query optimization technique, and indexing techniques. We will point out the similarities and differences between the techniques used in Hadoop with those used in parallel databases. And try to improving scheduling environment in Hadoop.