Web Crawlers are Difficult
In my last post I explained what a web crawler was and some of the basic steps that someone could break up the logic into. I also dived into some of the difficulties I had when building out my own crawler. Granted, some of the difficulties were self-imposed: I was using PHP, had a limited timeframe,…