Bot Handling Tips
The TaylorNet is stuck is a constant storm of bot traffic. Many of the bots are benevolent, just quietly spidering away and respecting
robots.txt but some are not. Either way, they generate a lot of traffic. (Google is notably much better at knowing when to crawl.) Here are a few things that I have found important when it comes to bots:
- Assume that everything will be discovered unless you use
<meta> noindexor use
robots.txt, and assume that by using those you will help bad bots discover things. So use them, but also make sure whatever it is, is prepared for bot traffic. Add the relevant mark up, but also add BasicAuth or something similar.
- Use Basic Auth for WordPress, like the example below. At some point your needs may surpass the limits this directives create, but until they do it will prevent headaches.
<Files wp-login.php> AuthUserFile /home/user/passwd AuthName "private" AuthType Basic require valid-user </Files> <Files xmlrpc.php> AuthType Basic AuthName "private" AuthUserFile /home/user/passwd require valid-user </Files>