Bot Handling Tips
Published on .
The TaylorNet is stuck is a constant storm of bot traffic. Many of the bots are benevolent, just quietly spidering away and respecting robots.txt but some are not. Either way, they generate a lot of traffic. (Google is notably much better at knowing when to crawl.) Here are a few things that I have found important when it comes to bots:
- Assume that everything will be discovered unless you use
rel=nofollow,<meta> noindexor userobots.txt, and assume that by using those you will help bad bots discover things. So use them, but also make sure whatever it is, is prepared for bot traffic. Add the relevant mark up, but also add BasicAuth or something similar. - Use Basic Auth for WordPress, like the example below. At some point your needs may surpass the limits this directives create, but until they do it will prevent headaches.
<Files wp-login.php>
AuthUserFile /home/user/passwd
AuthName "private"
AuthType Basic
require valid-user
</Files>
<Files xmlrpc.php>
AuthType Basic
AuthName "private"
AuthUserFile /home/user/passwd
require valid-user
</Files>