Building a Web Crawler
A web crawler is a script that goes out and crawls the web. Pretty surprising, eh? There are many crawlers out there, including the famous GoogleBot that attempts to index every site it bumps into and nefarious spambots that look for and exploit vulnerabilities. The logic behind them seems simple:…