Spiders and System Performance

I’ve noticed a couple of occasions where the server has been slogged to the floor. This would appear to be due to search engines unfriendly spiders spamming the site with lots of requests in a short period. Altavista is particularly dastardly for this, and is presently in the banned list that doesn’t even know the server is alive any more. FastSearch was the one that collapsed us tonight.

So… I am trialling some throttling software that tracks the number of requests a user makes in a certain period and throttles high users back. Hopefully normal users won’t notice any problems, but drop me a mail if you do get weird things happening…

Steve.