Website Toolbox Pro

Googlebot And Webserver Questions

SEO Comments »
I am having problems with web server(s) getting bogged down by google bots. I am looking for more information and help on this specific topic.

The site that the google bots are looking up is HUGE- dozens of pages with hundreds of links. Google appears to be crawling the site at 5 minute intervals and all of this indexing (or whatever its doing) is taking up resources nad costing big money. I am afraid if I put a robots.txt file that the site will not come up as high on the google search page.

Any help or info? I have found plenty of info that tells me "this is what a web crawler is" but nothing goes past that.

Continue reading Googlebot And Webserver Questions »

How Do I Get My Site On Google?

SEO Comments »
I am getting ready to up up my website, and would like it listed on Google, can I do this for free, and do I need a robots.txt file and if I do what need to go in to it? so far My robots.txt file looks like this:

User-agent:*
Disallow:/cgi-bin/
Disallow:/privatefolder/
Disallow:/downloads/
Disallow:/important/members.html

is that all I need?

I have never used a robots.txt file.

Continue reading How Do I Get My Site On Google? »

Robots.txt File And The Search Engines

SEO Comments »
What should I do with the robots.txt file in order to have my site found with the search engines?

Continue reading Robots.txt File And The Search Engines »