This a great website not only describing how web crawlers work, but how you can create on your own. Descriptions and pictures really help to create one if your stuck too.
That is a good tutorial -- thanks, Gordon. My own PHP skills are good enough to build this, though at the moment I don't need to. I had forgotten the synonym "scrapers," too. Useful quotation: "One typical task that Google performs is to pull all the links from a page and see which sites they are endorsing."
In theory that's a good resource, Jimin, except that it's wrong. :) The "server name" could be anything, and has very little to do with the domain name. It is true that you can usually log in to a server (a remote computer) by giving whatever program you're logging in with the domain name, but that doesn't mean that the server itself has the same name as the website. That page is also very, very wrong in calling the the top-level domain (.org etc.) the "domain name." It's important to note that that page was almost certainly written by a librarian, not a tech professional. (Of course, I'm an English PhD, not a tech professional myself, but still.) And when I looked at the source code, I could tell that it was hand-coded in HTML, which indicates to me that it's probably many years old.
Wish there were a "dislike" button. :)