team_IC05_ECP

June 09, 2005

Search Engines Explained (Part 3)

In Part 1 of Search Engines Explained, I talked about how the Search Engines (SE) find and index your web site and Part 2 talked about the basics of how SE process the search query.  Today we'll wrap things up with a discussion of the final result, the Search Engine Result Page (SERP).

Regardless of the SE used, the SERPs will display two main search listings: the free crawler-based listings and the sponsored listings.  Depending on the SE you currently use, there may be more listings available, such as directory listings, image listings, and shopping listings.

June 08, 2005

Search Engines Explained (Part 2)

Now that you have a basic understanding of how Search Engines (SE) find and index web content, we'll take a look at what happens when a searcher types in their search keywords.

It's truly amazing that you can type in a few keywords and within seconds be presented with hundreds of relevant web sites.  Imagine going to the library and telling the librarian "used cars".  The librarian will probably look at you for a couple of seconds, then start asking you questions so they can get you the exact information that you are searching for.  SE do not get this chance.  They have to return results (commonly referred to as Search Engine Result Pages or SERPs) with only the information they are given.

une 06, 2005

Search Engines Explained (Part 1)

We have all heard of them, we use them everyday, but do you really know how a Search Engine (SE) works?  Understanding how a SE finds and processes web pages, what goes on when someone types in a search query, and how those results are displayed is crucial information for your business.

This week I am going to explain each of these important processes and how you can use this information to your advantage.

The Crawling Search Engine

This is the type of SE that we normally think of when we hear the term Search Engine.  Other types include Directories (Yahoo and DMOZ) and Metacrawlers (Dogpile and Momma) which I will discuss in future posts.

The crawling Search Engine has to perform three basic tasks:

1. Finding web pages and storing their content
2. Analyzing the page content
3. Processing searchers queries

2. Analyzing the Page Content

Once your web page has been found and stored, the SE will inspect every word and tag and translate it into a mathematical representation in its database  This process is different for each SE and it is strictly confidential.  If this translation were ever to be released, then SE marketers could change the structure of their web pages in order to artificially inflate the page's relevance and ranking.

The complexity of the analysis, as well as the different processes used by the different SE, make a detailed discussion on this topic difficult.  What is important to know is at this point the SE do not look at the real web pages, they look at these mathematical representations when they match the searchers keywords to the documents in the database.

One more point I'd like to make.  When the SE analyzer comes across a link, the SE will feed that link back into its scheduler program to have its crawler visit that page. 

Pro Tip: To find out which of your pages have been found by
Google, type the following into Google's search field:
site:YourDomainName.com

3. Processing Searchers Queries

A discussion of Information Retrieval Theory is well beyond the scope of this blog.  But there are some very important issues that you need to be aware of.

Everyone is familiar with the plain text-based search feature of SE.  Here you would type into the search field something like "public golf courses in New Jersey".  But SE are advancing into many different areas of search results.  For example, type in your telephone number and see what comes back.  Even FedEx tracking numbers are interpreted by the SE giving you just the information you are searching for.

This concept, called semantic analysis, which tries to determine the searchers intention when typing in search keywords is an area that all major SE are competing in.  Each SE wants to deliver the best results to its visitors or risk losing that business to a competitor.

But the SE have to be careful.  Some SE track searching habits and use that information to improve their service.  This comes very close to privacy infringement.  Some users won't mind being tracked if their search results improve, but a lot of other people think that it's none of the SE's business to track their search history.  For more information, read this article that was posted June 3rd on CNN.

In the next blog post, we'll look at how the SE return the results in the way that they do.