Sept 10 3:17pm NZT Server downtime

"

Covertghost wrote:

Thanks for the prompt response and explanation guys.

Your communication has been awesome.

Crashes are to be expected in early beta.

"

Jonathan wrote:

This problem was caused by a segmentation fault in the master server along with a failure in the systems designed to automatically restart some parts of the realm in response to the problem.

While the website daemons reconnected, the instance spawners got into a state in which they were not authenticated with the restarted realm and also failed to restart.

In addition, our monitoring system didn't have any email events set up for the "Incorrect number of instance spawners" event.

Here are the three changes we will be making in response to this incident:

1) Investigate and fix the cause of the initial crash.

2) Fix the logic that restarts an instance spawner if it is not correctly authenticated with the current master.

3) Make sure we send emails on more of the monitoring events.

While we can never promise that the realm will not crash during beta, the realm shouldn't ever be down for a sustained period. We apologise for this incident and hopefully these steps will mean that it will not happen again!

"

Jonathan wrote:

This problem was caused by a segmentation fault in the master server along with a failure in the systems designed to automatically restart some parts of the realm in response to the problem.

While the website daemons reconnected, the instance spawners got into a state in which they were not authenticated with the restarted realm and also failed to restart.

In addition, our monitoring system didn't have any email events set up for the "Incorrect number of instance spawners" event.

Here are the three changes we will be making in response to this incident:

1) Investigate and fix the cause of the initial crash.

2) Fix the logic that restarts an instance spawner if it is not correctly authenticated with the current master.

3) Make sure we send emails on more of the monitoring events.

While we can never promise that the realm will not crash during beta, but the realm shouldn't ever be down for a sustained period. We apologise for this incident and hopefully these steps will mean that it will not happen again!