I think it might be the usual misbehaving spider causing the problems, but I shall reserve judgement for now.. I know there is a particularly badly behaved spider on the net that tries to grab hundreds of diary pages in one go and causes massive server problems...

We shall see whether my changes to robots.txt again fixes it... Please note that no diary entries will end up on Google and such like though because all the spiders are disabled.