Incident Report for Item Loss Bug

This is bullshit no matter how you try to spin it. There's no legitimate process by which you can straight-up remove an implicit or mod from an item. That already tells you something wrong has been happening. The actual deletion of items is harder to sort out, but 300 items affected is small enough to be able to go over each one manually.

"

Nuro wrote:

I'm extremely disappointed that you aren't restoring items that were obviously destroyed by the bug.

If a player posted about an item disappearing in a bug report or support ticket, and you can see it was legitimately destroyed, it should be fairly obvious that they did not perform the action. If an expensive item was destroyed, such as a Shav's or 6-link, it should be equally obvious. If the item had a microtransaction on it it should be perfectly obvious. If the stats on the item are completely nonsensical... I really can't even put to words how obvious that is.

300

Given what you stated, and that Chris stated this small but semi-specific upper bound, would make me believe that GGG knows what items were deleted.

I grok that GGG doesn't want to set item restore precedants. Unless the server has rewritten over the deleted data, GGG should make the effort to restore those items. If they simply can't, or the effort would be onerous, it would be good customer service to make some other sort of amends to the 300 or less affected.

"

InfiniteMonkeys2 wrote:

"

Zoen wrote:

"

Unfortunately, due to the nature of the bug, we are unable to restore affected items.
Memory corruption can corrupt data in such a way that it looks the same as a regular user action. Unfortunately, in our logs, legitimate item deletion is indistinguishable from loss due to this bug. We are unable to restore items lost in this way.

To those complaining that items aren't being restored:

"

Lost Data: In the event that any information or data (including without limitation in respect of in-game characters, achievements, Virtual Items, Points or general POE gameplay) relating to you or your use of any of the Website, Materials and Services held by Grinding Gear Games or any third party on behalf of Grinding Gear Games is lost, corrupted or otherwise no longer reasonably available or accessible as judged by Grinding Gear Games in its sole discretion, you agree that to the greatest extent permissible at law Grinding Gear Games will have no liability to you of any nature relating to any such information or data.

*cough*

The fact that this is present in the TOS doesn't invalidate the fact that refusing to restore items lost directly due to a bug you created is total bullshit. Also, accepting TOS doesn't mean shit in the EU (because nobody reads them anyway).

2 points.

1) ToS are legally binding, despite the fact that no one reads them. They aren't for companies to exploit and abuse consumers, as that sort of practice would utterly collapse in court, but they're meant to protect the companies when exactly this sort of shit happens. This is precicely what the ToS does and is for.

2) We are talking about a random line of code in a random grid reference being randomly set to 0.
There will be no logs of this change. It's a glitch. It sucks, but in order for them to verify that items were illegitimately lost due to this bug, I imagine they would have to go pouring through megabytes upon megabytes per character, per item change, per day to find the specific discrepancy, because a change like this would completely bypass any coding trackers they have.

This is like looking for a needle in a haystack the size of Ireland.

Now factor in that when you even FIND the offending data block, you have to arbitrarily decide whether that change was indeed a result of the glitch or an 'intended mechanic' of one form or another, AND that people will call in with non-legit complaints, rollbacks or just straight up fraudulent claims, and that you have hundreds of thousands of other legit changes in the same data to weed through, AND accounting for the real and incredibly likely possibility of human error in the tracking of these glitches - no. I do not blame GGG for saying that this is not possible. It is possible, but there are entire companies dedicated to tracking down and recovering these sorts of embedded software glitches, and even they have a bright red disclaimer saying "Sometimes, borked shit just can't be fixed and we're sorry about that." With all the conditions considered, it is so remotely impractical that it may as well be impossible. Much like sending a probe into a stable orbit with a tin can, a piece of string and a couple carrier pigeons. Possible. But remotely so.


As an aside, GGG are doubly damned if they do acquiese to something like this, because if they set a precedent in trying to amend for some people, then they're going to get screwed down the line every time a glitch like this happens and people demand reperations, regardless of it's possibility. From a PR standpoint, it is much less damaging, both in the short and long term; if an adequate fix cannot be reached, it is better to have everyone afflicted be left on an equal feild. If everyone can't be unfucked in a reliable fashion, then no one will be.

It sounds harsh, but if you really think about it and the repercussions around it - it's definitely the best, and some would say the only option.

I'm really glad GGG stuck to their guns again. HOPEFULLY we don't have a Kiwihalt 3.0, but honestly...these sort of glitches are in the long term, unavoidable, and if this is your policy of damage control, then stick to it. People won't be happy with it every time, but they will accept it. They have to. And in a few weeks, this will be mostly forgotten. In a few months, only those affected will remember, and only a handful would likely still care.

It sucks. It really does. But there's honestly no other way for this to play out that doesn't completely torpedo GGG for the future.



If after all that you still want to bitch about the unfairness of it - let's take a moment to commend GGG on how they discovered, isolated and fixed this in an incredibly timely fashion and prevented it from becoming an irreparable epidemic. A few weeks seems like a long time, but bugs like this can exist for years without being traced in mainstream programs.
You think people would be happy about having the entire server rollback a month? For a fraction of a percentile of people who lost items in a league only a small subset of the playerbase even play in and even fewer care about?

This incident was kept relatively isolated. It could very easily have been far more serious. Can you imagine if someone had an entire character deleted, should the 'nulling glitch' have falled on the right reference?

People need to understand there's not really anything GGG can do here to make everyone happy. It's impossible for them to know all the items that were affected by this bug. It's not really fair to fix only the obvious cases, since there are plenty more people affected by the bug in a way where, database-wise, it looks the same as if they changed/deleted the item themselves. If GGG tried to compensate everyone who was affected by this, there'd be plenty of people lying about it to try and get free stuff, and GGG would have no way to differentiate them from people legitimately affected by the bug.

This really sucks, and it's unfortunate a coding error is harming players, but really the only thing GGG could do is fix the bug, and they did that ASAP. There's nothing more GGG can do here that wouldn't be harmful or extremely unreasonable.

So you guys knew about this bug for over a week.. And no warning was given to players, and now they are left with the consequences. I hope at the very least this is a lesson to improve your code so that you can distinguish between item loss due to bugs and deletion of items.

we will see..

"

Taelis wrote:

So you guys knew about this bug for over a week.. And no warning was given to players, and now they are left with the consequences. I hope at the very least this is a lesson to improve your code so that you can distinguish between item loss due to bugs and deletion of items.

I can tell you don't understand how data is stored, so I'll try and keep this very simple:

That's not going to happen. You are asking someone to not get sick. In order to not get sick, you have to live in a sealed, conditioned container, isolated from all outside contact of every kind with everything passing through in or out to be screened, checked for contamination and filtered. Even the air.

Yes, it's possible to ensure that you never get sick. But are you really going to do it? No - it's impractical.

It's exactly the same with this data. "Sure, we can ensure that it we will be able to tell if it got bugged or not. But the data will be essentially unusable."

TikoXi, GGG should absolutely take responsibility for all of their fucking glitches. The argument that "if they take responsibility now, they will also have to in the future" as a reason to not do so is completely sociopathic of you. If their business model is to never take responsibility, they deserve to fail as a business.

Further, GGG wasn't transparent about this issue until it was fully fixed. The fucking responsible thing to do would be that as soon as they had two reports (or one fully-confirmed report), make an announcement about the bug and how they were trying to fix it (but that they would not be able to restore items, so people should not play if they cannot handle losing items). All week, people were going onto the bug-report forum to report their bug only to find out for the first time that this was a known bug that lots of people were experiencing.

Quite frankly, everyone that GGG thinks actually was affected by this change should be given a free perfect +1 acuity.


(I was not affected by this bug, I am just completely morally outraged at GGG)

[Post Edited by Support]

"

TikoXi wrote:

"

Taelis wrote:

So you guys knew about this bug for over a week.. And no warning was given to players, and now they are left with the consequences. I hope at the very least this is a lesson to improve your code so that you can distinguish between item loss due to bugs and deletion of items.

I can tell you don't understand how data is stored, so I'll try and keep this very simple:

That's not going to happen. You are asking someone to not get sick. In order to not get sick, you have to live in a sealed, conditioned container, isolated from all outside contact of every kind with everything passing through in or out to be screened, checked for contamination and filtered. Even the air.

Yes, it's possible to ensure that you never get sick. But are you really going to do it? No - it's impractical.

It's exactly the same with this data. "Sure, we can ensure that it we will be able to tell if it got bugged or not. But the data will be essentially unusable."

My beef isnt so much that it happened, its that they knew it was happening and did nothing to warn people (not even an announcement of HEY dont use the mtx! Hey were disabling this because of reports).

Another frustration with this is had this resulted in the creation of insane items (Uniques with extra affixes.. Gems lv 30+ ect..) you can BET that GGG would be all over "fixing" that problem.

Also i'm not asking anything of the sort. they should have measures in place that allow them to roll back characters. and if they have those measures they should be able to compare items from one rollback point to live and determine what was affected (especially since most people will report exactly what was affected. And then there's the obvious like shav's missing its unique affix)

Also: For the record I was not affected personally by this (i havent even logged in since buying my supporter pack.) but if I had been and this was the response I would be pretty pissed, and unlikely to support them in the future with my money.

This is all water under the bridge now. That said, this happened last year, too.

I'd like to see better disaster recovery plans myself.

While I love the agile approach to the game development here, bugs like this cause very long-lasting player pain due to the nature of an RPG. DR plans to address this sort of problem (regarding item loss/corruption, not just this bug) without the player pain would be quite nice to see.

I feel that with the growth of GGG, more effort could be put into such a DR plan.