3.7.4 Deployment Downtime Incident Report

Being so open and forward about ones mistakes is a rarely seen feat indeed;
Respect GGG and in particular Jonathan...

I am new to PoE, started in Legion. This is the first time i have ever seen such transparent communication after server downtime. Keep it up, and things happen. No worries!

"

SlaserX wrote:

"

"...our QA processes didn't catch the issue"

I've been playing PoE for nearly 7 years. I keep seeing this same sentence over and over again. This is what stops me from giving you more money.

Please tell me what other gaming company takes the time to write up incident reports this detailed with an apology as well? Your $10 a year budget on this game going away isn't effecting them in the slightest so stop acting like it does.

Interesting, thanks for the explanation
Seems the information-system between "producers" also needs to be enhanced. Perhaps these "producers" this time didnt speak the same language?
Would have been worse if this was affecting console as well, i suppose

Very mysterious incident

Not that I'd ever hope to have to see these posts again, but they're a huge reason I initially stuck with PoE: The open communication about issues, the small technical details about what can and does go wrong so quickly.

So a huge thank you for posting this.

"

BurninBodies wrote:

"

SlaserX wrote:

"

"...our QA processes didn't catch the issue"

I've been playing PoE for nearly 7 years. I keep seeing this same sentence over and over again. This is what stops me from giving you more money.

Please tell me what other gaming company takes the time to write up incident reports this detailed with an apology as well? Your $10 a year budget on this game going away isn't effecting them in the slightest so stop acting like it does.

I would, but they QA their patches before pushing them live, and don't have major incidents like this, much less over and over again.

Insult me all you want, I am right.

Thanks for this post.

This is one of those reasons why GGG is ahead against many other companies.

We love you GGG!

That it wasn't flagged or caught in QA is unfortunate and perhaps symptom of a tendency to rush things at GGG that crops up in many places. But rather than QA or devs specifically, it would seem that it's again process and therefore management that was lacking here. People automatically blame QA but when you don't allocate time, resources and tools for QA to do it's job, QA obviously will fail to catch things. Maybe you can get 90% quality for 50% of the work, but often times those remaining % matter a whole lot.

However having to make decisions quickly about live servers is not easy and doesn't leave much time to consider everything so while the fossil implications might seem obvious in retrospect, it's understandable that it wasn't considered at the time. 29 minutes is a worst case scenario, from time to realise the servers are up, download the patch, login etc, very little was lost and the rollback was definitely the right call. Saying that as someone who wouldn't have lost any fossils and lost a map's worth of farming.

Much respect to Jonathan for being open and taking responsibility, but I don't think many people would blame him here and while GGG has it's faults, the response to the stackening was as good as one might hope.

Really hoping that GGG evolves in how it values quantity and quality as a company however.

"

SlaserX wrote:

I would, but they QA their patches before pushing them live, and don't have major incidents like this, much less over and over again.

Insult me all you want, I am right.

Which game has updates that perfect?