High Impact Factor : 4.396 icon | Submit Manuscript Online icon |

Web Crawler


Shreyash S Pawar , Pcp; Ms. T. R. Shinde, Pcp; Priyanka K Barkund, Pcp; Saniya M Kadmude, Pcp


Web Crawler


A Web crawler starts with a URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl list. URLs from the frontier are recursively visited according to a set of policies. If the crawler is performing archiving of websites it copies and saves the information as it goes. Such archives are usually stored such that they can be viewed, read and navigated as they were on the live web.

Other Details

Paper ID: IJSRDV7I110001
Published in: Volume : 7, Issue : 11
Publication Date: 01/02/2020
Page(s): 4-5

Article Preview

Download Article