Crawl Delay Settings

The search engines need to crawl websites regularly to index new content and check for changes to existing content and also to make themselves aware of content that has been removed from the web. Sometimes the amount or frequency of time which the search engine robots spend on a particular website can cause that website server to experience load problems. For that reason most of the major search engines allow webmasters the ability to slow down the crawl rate to help web servers to keep up with the load.

When performing search engine optimization on your website, using this feature is not usually needed and isn't generally recommended either, but it is available should the need arise to use it. Smaller websites which do not update their content regularly will likely never need to slow down the crawl rate. They probably won't receive any benefit from using a crawl delay, because search engine robots will adjust their crawl rate to an appropriate frequency based on the content that it finds on a website each time that it visits that same website.

For large websites that have lots of web pages that are updated more frequently, they may need to be crawled more often and deeper too so that all of their content can be indexed by the search engines.

If you feel that your site needs to set a crawl delay you can do so by setting the crawl delay inside of your robots.txt file. In the robots.txt file within the generic user agent section you can add this directive as shown:

User-agent: *
Crawl-delay: 1

You can only use positive, whole numbers as values. The higher the value, the less frequent your crawl rate will be. Most webmasters will want to use the lowest value possible, which is a value of 1. Never use a value of more than 10 because that will severely impair a search engine robot from effectively crawling the content of your website.

This is how crawl delay settings effect your crawl rate:

No crawl delay specified - Normal

1 - Slow

5 - Very Slow

10 - Extremely Slow