Web Spiders

[paulselhi]

Is there a way via the stats that we can see when or wether we have been crawled by a spider or do we need a third party program?

WHAT ???

Ok quick lesson

Some search engines run a program often called a web spider that follows web site links to build up it's search engine database

When a spider visits your site it should "crawl" from link to link getting info about your pages and hopefully listing you in it's search engine database

[alphadesk]

Download your access_log file. Open with wordpad do file>find> googlebot. You can see how many times google has crawled you. If you know the names of the other spiders you can do the same for them. Here are some others.
Gulliver
Scooter-3.0.gdelta
SplatSearch.com
ZyBorg
Acoon Robot v1.50.001 (www.acoon.de)
Lycos_Spider_(T-Rex)
teomaagent crawler
Archie

[Ddr]

Hey Alpha

All of those spiders crawling around gave me the chills

[alphadesk]

Here's a good list of some of the most common.
http://www.akintermedia.co.nz/seo/spiderlist.htm

Another
http://www.spiderhunter.com/spiderlist/spiderinfo.txt

And you might want to check this page out.
http://www.spiderhunter.com/ Looks like they have a script that will extract all spider info from you logs.

[paulselhi]

this will keep me busy for a while, just have to get these blonde au-pairs to stop pestering me for raw uncompromising sex

[alphadesk]

Be strong pauli. Keep your nose to the grind.

[paulselhi]

i surfed from those sites and got to the searchengine forum page-loads of great tips

I have used simcoweb's doorway generator and i think the results are a bit tacky - no offence simco- i mean they just don't look right with the forced keywords and from what i have been reading they may get penalised (steady girls )

so what i am going to do is take these pages off the web and use the index page the gnerator creates and labouriously create
seperate pages for all the important keywords !!! no no there must be a better way help alpha!!