Visitors will only find a site by word of mouth (which includes email, blogs and forums these days), through a search engine or adverts. Generally speaking it’s better to get a good number of the right visitors than lots of random visitors. You’ve probably read about pumping up search rankings and other ways of blagging visitors—here’s what I set out to do:
Search engines are useful but probably won’t be enough on their own.
This site is simple and fast, the message is clear and it’s easy to find things. There are no pop-up windows, animations, award logos, software badges, affiliate buttons, banner ads or contacts from a link exchange. The sound clips are MP3 and don’t need a special player. You don’t need to register anywhere to use the site.
Some “professional” sites are impossible to use because of all the junk. User interface experts say a third of visitors don’t wait more than 8 seconds for the first page.
A link exchange won’t help unless it’s a high value site targeted at your audience. You don’t want to waste your visitors’ time sending them to a low value list.
I told my best contacts when I launched the site and word of mouth was far more powerful than I expected. These visitors are more likely to be tuned-in to what you’re doing. The word of mouth network is definitely worth spending some time on.
I recommend using a web statistics tool to tell you who’s visiting, where they’re from and what they’re looking at. This helps you keep the site relevant. Most web hosts offer one and there are several freeware statistics sites. (I’ve used Matrix Stats which cost about £10 a year and works well, Urchin which is very good, AWStats which is OK and Webalizer which is poor. Urchin is used by Google Analytics.) Stats will show:
Search engines will only find a site if you tell them it’s there or if other indexed sites link to it. It’s worth remembering the blindingly obvious, that searching always uses the words that are there, not what you meant to say, and not pictures, diagrams or logos. Some directories ask for keywords and you’ll probably use them in your page titles and in HTML headers anyway. It took me a while to get this right.
There’s a lot of folklore about search engine friendly HTML techniques and contradictory advice on just about every angle. I designed the site for users rather than search engines and I don’t think I’m missing any magic tricks that would get better rankings. Here are some common sense steps.
If trial searches for your own site fail, look at the HTML source for the top result pages and see if it gives any clues.
There are other ways to write your pages; this is just my best guess from the research I’ve done, and it works. I have many top page ranks and many top 10 page ranks on Google.
Meta name tags are used by some search engines and not by others, you can safely forget about them. I have tried with and without—information in meta name tags is effectively ignored by all the big search engines.
None of the major search engines will index a site quickly unless you pay, and their free services vary widely. Things were changing rapidly as I went along and it’s not easy to make sense of the search engine network. In some cases it’s hard to work out how to submit a site and what the terms are. Some different nationality sites have different options, so it’s worth trying variations (e.g. excite.com and excite.co.uk). The trend is to charge for submissions but many are still free. I didn’t use a mass submission service; the free ones tend to be a front for junk mailers.
|Search engine||Type||How submission works|
|index||Indexes a site from the main URL, also uses the Open Directory Project directory. Google shares its index with many other search engines.|
|All The Web||index||Indexes a site from the main URL (also offers Lycos chargeable options).|
|AltaVista||index||Says it indexes specific pages (not everything under one main URL) and only 5 pages at a time. The AltaVista crawler (Scooter) visited after about two months and indexed the whole site, not just the pages I submitted.|
|Ask Jeeves||index||Simple submission by email —one main URL and a description of the site. You will get hits from web.ask.com anyway.|
|MusicDish||directory||A specialist directory but they don’t change an entry once it’s up—you can link an MP3. No good for direct traffic but scores as a Google link.|
|Open Directory Project (dmoz)||directory||This is the directory used by Google and many others. It took nearly a year to get onto ODP.|
|Dogpile||Uses information from other search engines.|
|Excite||Has links to the Open Directory Project (so I assume it uses that directory) and a free option that indexes specific pages you submit. It says 1 page at a time but I couldn’t get it to work at all.|
|Yahoo||Appears to be free but if you sell anything you must use the chargeable Yahoo Express service.|
|Lycos||I could only find free options from the UK site. The index option worked but the directory option didn’t. Otherwise Lycos is part of a (mostly chargeable) group including Overture, Infoseek, Infospace, MetaCrawler, WebCrawler, HotBot, InSite, DirectHit, Go and others, sharing a mix of indexes, directories, data and portals.|
Although I didn’t submit to Yahoo, Excite and Lycos, their crawlers visited and I get searches referred by them and many others (e.g. MSN and AOL).
This is the sequence of events with Google and ODP from July to November 2002, as an illustration of submission lead time. I also had a number of hits by FAST WebCrawler and other spiders but it’s difficult to track the source and timing because their search engine network is complex (Lycos above).
|6 July||0||-||-||-||0||First URL submission|
|6 August||31||-||10||10||0||Nothing searchable yet|
|25 August||50||-||-||10||1||Some home page words searchable|
|27 August||52||0||-||10||1||Submitted to the Open Directory Project (ODP)|
|1 September||57||4||5||15||1||More Googlebot visits|
|2 September||58||5||16||31||1||More Googlebot visits|
|11 September||67||14||-||31||1||Subscribed to Google free web and site search|
|21 September||77||24||-||31||1||Resubmitted URL today|
|23 September||79||26||1||32||1||Another Googlebot visit.|
Cached page same as 25 August
|24 September||80||27||-||32||2||Cached page updated!|
|26 September||82||29||1||33||1||Still just the home page indexed|
|28 September||84||33||20||Site indexed (Google says about 24 pages).|
Checked the pages indexed, initially 20, then 24
|6 October||-||39||25||58||24||More Googlebot visits (regular update)|
|8 October||-||41||2||60||24||More Googlebot visits|
|9 October||-||42||1||61||24||More Googlebot visits|
|Oct/Nov||61||32||More pages indexed by Google|
|7 November||-||71||1||62||32||More Googlebot visits (regular update)|
|8 November||-||72||61||123||32||More Googlebot visits|
|9 November||-||73||1||124||32||More Googlebot visits|
|23 November||-||87||-||124||32||Still not on ODP (but listed in July 2003)|
My Google index was originally updated about once a calendar month: the crawler visited for a day or two and about 4-8 weeks later the index was updated. This meant the index could lag the site content by up to 3 months. These days updates are generally on Google within a week or so. Here are a few tips for checking your Google visibility.
The most useful online guide I found about search engines, submission, etc. is www.selfpromotion.com Robert Woodhead’s site. It has good advice and coverage (although I’m sceptical about search ranking tricks) and provides some useful tools.
Here’s a summary of what worked best.
By the end of 2002 my hit rate was good enough and although I keep tweaking I don’t plan any major work on HTML or search engines. It’s now simply a matter of keeping the site up-to-date.
Here’s the kind of growth rate you can expect on a popular site without advertising. Slow but steady.
Getting a foothold on the web can be a slow and frustrating process but you only have to do it once. The web crawling strategy of Google has changed considerably since 2002 so your experience may be completely different but the basic principles are still the same.