Friday Sept 16 NZT Site Outage - Now with Incident Report

Here is this incident report for this issue.

This issue was triggered during our automated backup of the game database. We are not absolutely sure what happened yet but the backup caused the database software to panic.

Normally this would lead to the server aborting all open transactions, restarting the database and then retrying all the transactions once more, continuing where it left off. Unfortunately the DB recovery code was not well tested enough and the server managed to get into a state in which it thought it was recovered, but yet every transaction returned a database error.

While our monitoring did detect the issue, nobody was awake to see problem and attempt to fix the issue.

When I woke up this morning I restarted the realm and this brought everything back up.

Here are the steps we will be taking in response to this issue.

1) Investigate the cause of the original issue and attempt to fix it. We will be asking the developers of our database software questions about what occurred. This should prevent this specific issue happening again.

2) Fix the code that recovers the database after a failure. This should prevent problems occurring with the database in the future.

3) Investigate our options for adding support for sending an SMS message to our developers so that if something goes wrong during the middle of the night we can investigate it immediately.

4) Potentially add an automatic restart to our monitoring so that if the realm continues to be in bad state for more than 5 minutes, automatically restart it. This is something that we have been considering but are reluctant to do because we might end up causing more problems if there are false positives.

While we are in beta, server problems are expected, but longer term downtime is not acceptable. We apologise for this issue and hopefully with these steps it will not happen again!