Web Crawler |
Author(s): |
| Shreyash S Pawar , Pcp; Ms. T. R. Shinde, Pcp; Priyanka K Barkund, Pcp; Saniya M Kadmude, Pcp |
Keywords: |
| Web Crawler |
Abstract |
|
A Web crawler starts with a URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl list. URLs from the frontier are recursively visited according to a set of policies. If the crawler is performing archiving of websites it copies and saves the information as it goes. Such archives are usually stored such that they can be viewed, read and navigated as they were on the live web. |
Other Details |
|
Paper ID: IJSRDV7I110001 Published in: Volume : 7, Issue : 11 Publication Date: 01/02/2020 Page(s): 4-5 |
Article Preview |
|
|
|
|
