Fault Tolerant Distributed MapReduce Engine

Hadoop is open source implementation engine to handle single point of failure problems occurred in Hadoop job management (Master node), Distributed Map Reduce Engine for Hadoop exists to handles the drawback of failure of master node. System proposed about a standby master node which is synchronized with active master node and receiving incremental updates from active master node. Dominant issue in existing system is that if after completion of synchronization, active master fails immediately then standby node will not know until next synchronization due to which task updates will be lost on active master and standby node will also wait for next synchronization cycle due which switching latency will increase. In this paper, we propose scheme in which standby master node keeps sending Heartbeat message to Active master node in order to detect failure. When active master node stops responding to that message, standby master node comes to know that master node has failed. Then, switching will takes place from active master to standby node and standby node will start acting as active node. We have also analysed the effect of faster synchronization cycle between active and standby node.