High Impact Factor : 4.396 icon | Submit Manuscript Online icon |

An Approach to Solve Small File Problem in Hadoop by using Dynamic Merging

Author(s):

Suraj Rajendra Sawat , Jspm's Imperial Collage of Engineering and Research Center,Wagholi Pune; Shubham Bhandari, Jspm's Imperial Collage of Engineering and Research Center,Wagholi Pune; Suraj Chougale, Jspm's Imperial Collage of Engineering and Research Center,Wagholi Pune; Deepak Pandit, Jspm's Imperial Collage of Engineering and Research Center,Wagholi Pune; Suraj Sawat, Jspm's Imperial Collage of Engineering and Research Center,Wagholi Pune

Keywords:

Big Data Problem, HADOOP Cluster, HDFS, Map Reduce, Parallel Processing, Small Files

Abstract

size of the data application in now’s firms has been spreading at an immoderate common from last few donkey’s years. Simultaneously, the emergency to process and breakdown the complete volumes of statistics has also extended. Hadoop dispensed report system (HDFS), is a candid fountain implementation of Apache, display for flowing on profit ironmongery to spindle applications property diffusive datasets (TB, PB). HDFS production is supported on by myself skipper (call Node), which stale the metadata for diffusive quantity of vassal. To get maximum ability, name Node deliver all of the metadata in its RAM. So, whilst putting with considerable enumerate of insignificant defile, call Node frequently get a impasse for HDFS as it might exit of remembrance. Apache Hadoop makes use of Hadoop ARchive (HAR) to distribute with unimportant march. However it isn't always so effective for numerous-name Node surrounding, which exact system rifle flaking of metadata. In this courier, we've got scheme triturate summary shield creation, New Hadoop ARchive worn sha256 as the constituting, which is a modification of existent HAR. Is suggest to situation greater reliableness which also can condition car peeling of metadata. In preference to worn one NameNode for shop the metadata, NHAR makes use of manifold NameNodes. Our occasion manual that NHAR lessen the freight of a sincere NameNode in symbol quantity. This companion the crowd greater scalable, extra lusty and less headlong to deterioration unlikely of Hadoop Archive.

Other Details

Paper ID: IJSRDV5I40312
Published in: Volume : 5, Issue : 4
Publication Date: 01/07/2017
Page(s): 277-279

Article Preview

Download Article