Since it has been a few days without issues, I modified the response to bot code a bit further. Now you will no longer see forbidden the link simply will not do anything.
This step fully stops the bots from pulling any of our bandwidth whereas the prior step lowered their bandwidth and cpu usage by about 75%. Again hopefully this will help with speed particularly for the times the bots are pulling 10 pages a second.
Ok if you ever see a link and it goes nowhere let me know.
Thanks
Kato
|