A Dynamic Replication to scale back Response-Time of I/O Task Execution Performance on Cluster Computing

Extraordinary large datasets of high performance computing applications require improvement in existing storage and retrieval mechanisms. Moreover, enlargement of the gap between processing and I/O operations’ throughput will bound the system performance to storage and retrieval operations and remarkably reduce the general performance of high performance computing clusters. File replication may be a thanks to improve the performance of I/O operations and increase network utilization by storing several copies of each file. Furthermore, this will lead to a more reliable and fault- tolerant storage cluster. In order to improve the response time of I/O operations, we have proposed a mechanism that estimates the required number of replicas for each file based on its popularity. Besides that, the remaining space of storage cluster is taken into account within the evaluation of replication factors and therefore the number of replicas is tailored to the storage state. We have implemented the proposed mechanism using HDFS and evaluated it using Map Reduce framework. Evaluation results prove its capability to enhance the reaction time of read operations and increase network utilization. Consequently, this mechanism reduces the general reaction time of read operations by considering files’ popularity in replication process and adapts the replication factor to the cluster state.