Webmaster Q&A

I am having problems with web server(s) getting bogged down by google bots. I am looking for more information and help on this specific topic.

The site that the google bots are looking up is HUGE- dozens of pages with hundreds of links. Google appears to be crawling the site at 5 minute intervals and all of this indexing (or whatever its doing) is taking up resources nad costing big money. I am afraid if I put a robots.txt file that the site will not come up as high on the google search page.

Any help or info? I have found plenty of info that tells me "this is what a web crawler is" but nothing goes past that.

I am getting ready to up up my website, and would like it listed on Google, can I do this for free, and do I need a robots.txt file and if I do what need to go in to it? so far My robots.txt file looks like this:

User-agent:*
Disallow:/cgi-bin/
Disallow:/privatefolder/
Disallow:/downloads/
Disallow:/important/members.html

is that all I need?

I have never used a robots.txt file.

What should I do with the robots.txt file in order to have my site found with the search engines?

A few questions.

1. How do seach engines find things about what a website is about besides useing the <meta> tag?

2. How can you get the top result on a search engine if your basic thing is put in. (Ex. Landmark Missionary Baptist Church, right now the (just about) only way to find out site is to type in a nearby location)

3. Is the only way that search engines find things are by robots and by people submitting URLs?

4. How does your URL affect the search results?

5. How much does it cost to pay search engines to be the first results?

6. Is there a simple way to submit your site to a whole bunch of search engines at once?

7. Is there any other ways besides the <meta> tag to add tags to your result?

If you want to add more, feel free!

I have sumbitted my blog in Google Webmaster Tools, but its asking to verify my home through a meta tag or a HTML file. How can I upload these files (sitemap.xml, robots.txt and meta tag or HTML file verification)?