Xedri's Comment

Hi Xedri,

Yeah, the front screen also uses a connection to the database, for pretty much everything, the genre selector which shows the different genres requires a connection, the Recent Updates requires a connection once every ten minutes (its a fairly long (well, 5 seconds or so) process to work out the latest updates, so its done offline every 10 minutes). The Top10 and the Choice of the Week don’t require a database connection – but because the Genre Selector does, and in fact the whole look of the front page is changeable based on information in the database – that screwed that page up too… Sorry.

My information thus far indicates that the server first failed somewhere around 17:00 EST, for approximately half an hour. It then recovered somewhat, until about 21:00 EST, where it remained dead until I discovered it at around 02:00 EST.

We currently believe the problem was caused by the database software itself not quitting when it had finished, and since the system is configured so that the database cannot request so many resources as to kill the machine, it eventually ran out. Its likely that a tweak we made that day (and tested successfully!) had the unfortunate side effect of tickling a bug in the database software that did not come to light until many hours later. The tweak has been reverted and hopefully the problem won’t re-occur. Either way, if it does, the system SHOULD now let you know its currently too busy rather than just error like that.

More information as we find it out.

Steve.

Down and Out!

Anyone trying to access the site at the sites busiest time (1am GMT) will have noticed a rather annoying phenomenon… You couldn’t. Any time you turned around there was an Internal Server Error or if you tried to login you were constantly denied…

The overall reason for this is simply that for some reason the database server became too busy to serve any more connections. Normally this would be a good thing (though would mean we need to invest in some more hardware!) since it would mean the server was being used. However, since its configured to accept more than double the normal usage, we’re still not sure what exactly happened.

The previous night we were spidered by Altavista in a MOST UNFRIENDLY manner, so if anyone from AV is reading, fix your goddamn spiders so it doesn’t request 1000 pages from us all at once. Its not nice, it locks everyone else out…

Last night was NOT about that. The database failed for some reason, and wound itself up in a knot to the point where it couldn’t close any of its processes down properly, resulting in more and more being opened, each time someone made an entry or read an entry. Pretty quickly that expired all the resources and everything else stopped. The database processes were still open this morning when I got here… They shouldn’t have been, but my efforts to kill them off gracefully were not rewarded either. Brute force almost didn’t work and the next step was a complete reboot… Fortunately that was not necessary in the end.

Investigations will proceed into what happened and we’ll let you know if we find anything concrete. Apologies to anyone who suffered as a result, and I will get to replying to all your emails today.

Steve.

ATTENTION NEW USERS!

A script error has meant that any new users have not been able to access their diaries properly since creating them. This has been since some time early this morning, but is now fixed. If you had never customized your diary, you would see Internal Server Error’s when trying to access your diary, and when anyone else was trying to access it.

I apologise for any inconvenience and stress caused to new users!!

Steve.

Database Upgrades

The underlying database has been upgraded to a newer and reportedly much faster MySQL version. Unfortunately, due to some technical limitations of the operating system, it took longer than expected, (about half an hour, compared to the 5 minutes it should have been!) but its now back online and should stand a lot more load in the future. It seems a bit snappier now, but maybe I’m imagining it :)

If you get any weird errors, I’m on vacation and Matt’s dealing with everything…

Heheh. No, really, any weird errors, please mail the regular support@deardiary.net address.

Steve.