Incident Report for March 20 Downtime

was a bit irritating but eh, watched a stream,
people joking around and fiddled with my stash for awhile.

after bout 20 mins i was like damn,
someone must literally be away from the computer for no response yet. yup.

quickly fixed once the issue was brought to attention.
well done. and thanks for the report.
keeping us in the loop keeps us...not mad.

What's great is we got an honest response in a timely fashion, not corporate double-speak.


Thanks guys! We love this game!

As someone who studied engineering I find systemic breakdowns very interesting. A detailed description of how an error or failure gets around the systems put in place to detect problems just reinforces that no matter how thorough you are when something "wants" to go wrong it will.

Specific to this incident:
If Jonathan was not at GDC the failure would not have happened
If Thomas was available at his desk it would not have happened
If the list had not been moved to the live server it would not have happened
If the list had been tested it would not have happened
If the list did not have the bugged code it would not have happened
If this had happened a week or two later the new employee would have been available

The amount of odd coincidences and otherwise benign things that happen at just the "right" time is always fascinating.

On topic: The amount of server uptime that GGG maintains is something you guys should be very proud of. Also thank you for doing an incident breakdown its a very informative read.

Awesome support.

@Thomas / Jonathan:

I am a network engineer by trade (working for Akamai in the states) and I'm also a long-term mmo player. I would like to say simply that as far as incident responses go 1 hour is really top-notch, world-class support. My team where I work has a 20-minute SLA with our customers which include HUGE financial firms, retailers, social media sites, streaming media sites, etc. The difference between my team and yours?

a 10 billion dollar market cap.

Seriously you guys, you're doing world class work, getting shit done very quick, and keeping the servers up WAY better / more reliably than most of your non-free counterparts. Furthermore I'd like to add that you guys diagnosed / fixed a problem in under an hour while vpn'ing around the globe using shitty hotel wi-fi. The fact that you were able to do this / push a fix into prod in under and hour...Jesus.

Stop apologizing, this is work that you should be proud of, not ashamed of.

If anyone gets in trouble for this, shoot me a PM and I'll hire you on the spot.

It's all good, personally I would understand.

You guys are the best!

As a fellow admin, dont feel damned sorry you found a bug in so little time. Especially with a game where a downtime is like finding the holy grail copulating with the fountain of youth in el dorado....

*edit*
I should set up a reminder that some of my next pay goes towards you guys.....

chillex!
The way i see it <1h resolve time is nice. at my work we have 30mins response time <2h to solve in our contracts for mayor incidents in the electronical patient file system (work at mental health care).

so basicly good job.

This is why I always feel like I want to send more money your way.

This post just reeks of awesomeness