Search Engine Considerations


Note: This article was written back in 1998 and is woefully out of date.
Here’s a newer version which isn’t quite as out of date.

In the past few months I’ve received a number of questions from clients
about why they can’t find their site when they use a search engine.
There are a number of reasons why this might be, including the site is
too new, the topic too broad or their search query is too vague. Sometimes
there’s no good reason, and a resubmission of the site takes care of the
problem. Many search engine gurus suggest resubmitting your site once
a month to avoid disappearing.

In this article I’m going to give you a basic overview of search
engines. There’s more information on search engines that I could
ever put into one article, and if you’re interested in this topic
I suggest you look at by Danny
Sullivan. It’s just one of the search engine resources you can
find on my resources page at

While search engines are a very common method that people use
to information on the web, they’re not the only way. The latest
statistics (as reported in the Wall Street Journal) put the number
of web pages at 350 million and growing exponentially, so it’s
going to become harder and harder to get your site into the top
ten most relevant pages in a given search. Especially if your
type of business is well represented on the Web; i.e. graphic
designers, X-Files collectibles, or sadly, web site designers.
In other words, don’t ignore search engine considerations, but
don’t lose sleep over them either. (That’s my job.)

Before we start, some quick definitions. (Don’t worry, this
will be painless.) Search engines are applications that “search” the
web, using programs called spiders or robots to gather information
from web pages and bring it back to a search engine’s database.
When you enter a search query, (i.e. Tibet,) the engine generally
searches against its own index, not the entire web. This is why
some search engines won’t show your pages for up to a couple
of months even if you submit the pages to them. Also, services
like Yahoo! are not search engines, but directories. This means
there is a team of humans–those living, breathing bipeds–who
take submitted URL’s (uniform resource locators or “website
addresses”,) review them and add the page to their directory
if they see fit. There are several major search engines and directories,
and everybody has a favorite. Some of the most popular are Yahoo!
(, Alta Vista (,
HotBot, (, Excite! (,
Lycos, (, Webcrawler, (,
and Northern Light (you guessed it,

When you enter your query the engine looks through its database
and pulls up documents–web sites in most cases–that it thinks
meets your criteria. The number of documents returned depends
on how broad (Tibet) or narrow (Tibetan rap bands) your query
is, bringing the results to your browser window, generally 10
or so at a time.

Search engine administrators are constantly trying to improve
their products so that they return more targeted and relevant
results while many website creators are constantly trying to
tweak their websites so they come up higher in the ranking. These
two goals often come into conflict as many website owners continually
try to push the limits of acceptable behavior while the search
engines are trying to keep the playing field level. Some tricks
website owners use fall into the category of spamming. Some examples
include over-repetition of keywords, using same color text and
background (so the search engine sees the text but not the viewer)
or using keywords that don’t reflect the site’s topic. For instance,
sites may add a mention of “Monica Lewinsky” because
they know people will perform searches on her, (so to speak).
Search engines have moved to disallow pages using these techniques.

However, what passes for spamming isn’t universal. Also, many
of the engines continually change their own rules to further
frustrate spammers. They believe that if people searching for
information feel the search engine isn’t returning relevant documents,
they’ll stop using that engine for another one, which then brings
down the search engine’s value to advertisers. (Doesn’t it always
come back to money?)

Tips: Although every search engine has its own mechanism for
determining relevance, there are some tips you can use to improve
your ranking. Perhaps the most important aspect of your page
is the title. “Welcome!” or “My
Company’s Home Page
” are not good. Nobody searches for these words. (If
they did, they would receive over a million irrelevant documents.) “Tibet:
The History of a People
” or “New Techniques
for Fighting Halitosis
” are more likely to bring people to your site
who are interested in these topics.

Meta-Tags: These tags aren’t seen by visitors, but many search
engines use them for relevancy ratings. It allows the creator
to give the search engine a brief description of the page, the
keywords, who the author is, and any other information the creator
wants to pass along. Certain search engines ignore meta-tags
since some website designers use inappropriate keywords. For
example, a children’s clothes shop may use “child safety” to
lure parents to their site. Many sites use adult-oriented keywords
to attract visitors even if their site is about real estate.
(Adult topics are still the most popular queries.)

Since some search engines count the number of words on the page
as part of their relevancy testing, a short page that mentions “New
England Patriots” twice might be deemed more relevant than
a longer page that mentions next year’s Super Bowl champs five

Duplicate pages: I’m not sure if this is considered spamming,
but I’ve had some luck creating duplicate versions of the same
page, changing around some keywords or shortening the page, and
submitting both pages. (I generally don’t submit them on the
same day as some search engines consider that to be spamming.)
Sometimes it’s just interesting to see where almost identical
pages match up.

For example, if your media relations company has offices in
Boston and L.A. you could duplicate your home page and put Boston-related
keywords on one page, saving it as boston.html and L.A.-related
keywords on the other, saving it as la.html. Submitting both
pages might get you a higher rating if someone searched under “Boston

Search Engine specific pages: If a search engine ignores meta-tags
it might be good to duplicate a page, take out the tags and submit
it specifically to that search engine.

There are many other variables that affect relevancy including
JavaScript, use of frames or tables, subject matter, the waxing
and waning of the moon….If you are still hungry for information
please go to [Defunct]
and check out some of the links there.

Remember, search engines aren’t the only methods of bringing
people to your site. Traditional media, banner swap programs,
online malls, and web rings are just a few alternative methods
to alerting people to your offerings.

–Rich Brooks

President, flyte new media