While we all know about the search engines and the way their spiders trawl the internet looking for new content and indexing sites every second of every day, do you know how to attract spiders to your site? Have you ever even considered it?

While on the surface the internet may look like a fairly sedate place with sites listed, people accessing them and then moving on, under the surface their is so much going on. Each and every search engine on the internet will have their own spiders which they release onto the net, bringing back information about sites and content. It is these spiders which will dictate how and when you are listed, what pages are listed and which are ignored. While they do a great job, they do need help and guidance because they will not hang around trying to unravel your site if you don’t assist them!

By simply connecting your site to the outside world, either by some kind of natural link or paid link (even though they are not strictly allowed) the spider should find you, and once they find you they will remember where you are. However, the more new content you can add to your site on a regular basis, the more often they will come back. They like new food, new content to “feed on” and the more food you give them the more often they will return.

However, there will often be pages which you do not want indexed, whether these are admin or backend pages. There are some simple commands available which will tell the spiders not to index certain pages – all you have to do is tell them!

Treat the spiders well and they will come back time and time again, however if you starve them or feed them the same food as others (i.e. duplicate content) they will soon forget about you and not visit as often as you would like. Do not under estimate the power of the spiders, but do not be afraid!