Incident Report for 1.0.2 deployment downtime

Exemplary professionalism, paired with honesty and truth. Sounds a bit melodramatic but I mean it.

Keep up Your passion and love for this game, You really make me happy with it. :)

Wasn't necessary but I liked it as I used to admin some servers. Don't you just hate it when that stuff happens ;-)

"

Jonathan wrote:

Nobody checked that the migration took a reasonable amount of time.

Very interesting read. Although I have to say that the above quoted sentence is a no-go from a QA Department. As computer scientist myself, checking the logs even if a process was successful is a no-brainer.

I was working as a financial controller for a very huge company in europe and, because of my background, was also responsible for some parts of the servers and backup systems. We ran backups every day. During my training days in the company I was "never" told to check the logs, but I did anyway and one day discovered a minor problem, which could have lead to data loss.

So I guess the QA department just didnt do their jobs right. Checking logs is very important, especially in these kind of processes. I hope you guys learned a lesson.

On the other hand: A total downtime of less then an hour is absolutely no tragedy. You guys still did a nice job.

Jonathan

That was more than any other Dev ever said from any gaming company regarding downtime :D

Much appreciated!

I won't deny there are a few things I'd like to hear explanations for from GGG.

But not this, and certainly not an apology.

my take on all this..


this is partly why other big devs shut their systems down to 12 hours when doing big patches, it hides the fact that they are human just like you guys and lets them seem like uber-pro devs with everything running smoothly. However its saving face at the expense of player access to the game, it sucks for the players to be locked out for long periods of time in an online only game. We get a patch, the server goes down, 5 mins later were im a map killing, when the worst thing in the world happens and the entire patch goes tits up we have to wait half an hour... big deal, Id take the half hour wait over 12 hour downtimes, no question.

Same with significant updates every 2 weeks rather than 2 years, ok we then get the thing where balance changes are made more often, but thats so much better than not having anything change in the game for huge periods of time.

keep doing what youre doing, computing and snags go hand in hand, always. I think this is the future of game development, people dont have all day to wait for a patch, people dont have years to wait for new content. We need a game thats on its toes and I think were more than willing to accept the rough edges that come with that.

I really like the way you communicate. I know a couple of companies that can learn a lot from you about transparency and all that good stuff ;)

OK, here is something I have to say. And I'm not attacking GGG, nor am I defending the "bigger companies". Come on, everybody knows who we are talkin about here, so lets just say "Blizzard", ok?

Anyway, you cannot compare GGG's downtime and server structures and database sizes with Blizzards. It's just ridicolous, and only shows that you have no idea what you are talkin about.

The reason why Blizzard have significant longer downtimes is based on the fact that their databases, servers etc are probably A LOT larger then GGG's.

If GGG also had several online games with servers ALL around the world and a total of probably 20 million players, GGG's downtime would also be several hours. And if something like yesterday happened then, they probably wouldn't just have rolled back and fixed the issue in less then an hour. They also, like Blizzard, would probably just leave the servers offline until they find the error.

Again, I'm not defending Blizzard here, but you guys compare a giant company with hundrents and hundrets of servers, extremly huge multiple databases to a company with lot less servers ... it's not fair.

just my 2 cents ...

Keep up the amazing work GGG!

You guys kick ass!