Web Crawling Using Location Aware Technique |
Author(s): |
| Praveen Kantha , BRCM College of Engineering & Technology, Bahal, Bhiwani (Haryana); Amit Kumar, BRCM College of Engineering & Technology, Bahal, Bhiwani (Haryana) |
Keywords: |
| web crawling model, Mobile Agent, database. |
Abstract |
|
Most of the modern search engines are based in the well-known web crawling model. A centralized crawler or a farm of parallel crawlers is dedicated to the process of downloading the web pages and updating the database. However, the model fails to keep pace with the size of the web, and the frequency of changes in the web documents. For this reason, there have been some recent proposals for distributed web crawling, trying to alleviate the bottlenecks in the search engines and scale with the web . In this work, facilitating the flexibility and extensibility of Mobile Agent , we consider adding location awareness to distributed web crawling, so that each web page is crawled from the web crawler logically most near to it, the crawler that could download it the faster. We evaluate the location aware approach and show that it can reduce the time demanded for downloading to one order of magnitude. |
Other Details |
|
Paper ID: IJSRDV1I5052 Published in: Volume : 1, Issue : 5 Publication Date: 01/08/2013 Page(s): 1251-1253 |
Article Preview |
|
|
|
|
