Group items tagged

This a great website not only describing how web crawlers work, but how you can create on your own. Descriptions and pictures really help to create one if your stuck too.

That is a good tutorial -- thanks, Gordon. My own PHP skills are good enough to build this, though at the moment I don't need to. I had forgotten the synonym "scrapers," too. Useful quotation: "One typical task that Google performs is to pull all the links from a page and see which sites they are endorsing."

I was still a little confused on what a web crawler was exactly. I thought this site was helpful.

Yes, it is! I've already given you credit for one link this week, but that is a good explanation -- even though the page it was last updated in 2005 and the search engine it gives as an example is Alta Vista. :) Alta Vista doesn't even exist anymore; it was acquired by Yahoo awhile ago, I think. Here's the Wikipedia page: https://en.wikipedia.org/wiki/AltaVista

Ya I saw the date, but felt the explanation was good.

Defines web crawler and describes web crawling software

Potential problems with data collected by a web-crawler that would not, in theory, be present if the data were collected by hand.