Allows/disallows using robots.txt and <META NAME="robots">
exclusions. Use "no", for example for link validation of your server(s).
Command may be used several times before "Server" command and
takes effect till the end of config file or till next Robots command.