Aproval level: 85%
Insertion Date: 2009-04-21
Total downloads: 51
Downloads this week: 0
Times recommended: 0

Spider robots are usually search engines that look in your website and link the content. But this has a negative side, as they consume a lot of your website`s bandwidth. The reason why a program like Robots txt generator exists is to control and restrict these spiders and their access to your website. First and foremost Robots txt generator is a web utility, totally free, very small and simple to use (it doesn`t even need installing), that lets you limit the number of consultations a spider can do, per minute or hour, within your website. The program database comes equipped by default with a list of 300 spider bots, but you can add new ones if you want, and also specify different restrictions for each bot. For example, in Robots txt generator you can let a Google Bot have access, but not a Google Image Bot.