Desync is out of control

This has been in the development manifesto for some time, and it's been about a month since I posted this (This is a long post):

Spoiler
"
DemonSkye wrote:
What drives me sort of nuts about this is that it's EASILY fixed, and it essentially relates to "trust" of your client.

First off, let me begin with my qualifications. I'm a network engineer working for the world's largest CDN. On any given day about 25-39% of all the internet traffic IN THE WORLD passes through our servers. I've had to troubleshoot errors from tiny errors at layer 2 of the OSI model all the way up to layer 7. I've had about 10 years experience in the networking-tech field.

As for my qualifications in POE. I have 3 (primary) characters, A level 80 Chin-Sol puncture ranger, a level 86-flicker crit dagger shadow, and a level 83 Scion, which I'm not currently using due to changes in aura nodes and EK just not having the clear speed of my other builds.

As is obvious, my PRIMARY build would be my shadow. I do the most dps (>100k with flicker and 4 power charges + melee splash, or >190k with added fire damage to a single target), clear the fastest, and generally have the most survivability.

I've died a lot on my shadow. And basically every time it goes something like this: Flicker into a group of level 73-75 mobs, watch their health drop to 20-30%, desync, stop hitting anything, end up stunned, die in the middle of a pack of 60000 mobs that should have died in 1-2 more hits. I know I'm not hitting the mobs when I don't hear the sound connecting on my headphones. When that happens I smash my /oos macro and hope that it reacts before I'm dead.

So, from a network engineering perspective, what's happening here.

1. Packets are being sent to the POE servers. They're being routed here, there, and everywhere by the internet. That's just the way things work. Generally packets make it (about 99.95% of the time) and sometimes they need to be retransmitted. Let's assume that desync is NOT caused by packets dropping, as GGG has commented that it's a sync issue, not a QOS issue.

2. Packets are received by the server a calculation about that packet is performed in a matter of milliseconds, and a response is dispatched

3. The response makes it to your computer, and it verifies that you are somewhere that you can be / that your attack connected / whatever.

So, what's the problem with this?

1. It places NO confidence in the client. Essentially POE is checking every single movement that the client is making, rather than allowing the client to determine whether or not you have taken a hit, the server decides. On a good day, when I'm playing in Boston, MA and my friends are playing in Singapore, I have a ping of about 350-400ms as it's likely me connecting to the SEA servers. That's totally acceptable. What's NOT acceptable is me using flicker strike once and missing with every hit of it, and then teleporting back a half a screen. What happened?

2. My computer sent packets to the GGG servers that service the Pacific Islands and SE Asia, and in the 350-400ms round trip they took, my client chose 3 random targets with flicker. Meanwhile the server chose 3 different targets to attack (flicker + multi). The net decision is: Well, we can't decide what happened, so you didn't use the attack and let's move you back to the last point we agreed. This happens CONSTANTLY. Using movement skills (WB, Flicker, and LS for example) causes me to desync ALMOST immediately.

3. This approach also doesn't factor in outside sources of network congestion, latency, or issues regarding lag spikes (local, or at any of the hops between me and your server). If you do a simple ping-of-death on a website like google and ping it for a period of 24-48 hours every 3-5 seconds, about 98% of your pings should come back within 10-50ms, 1.5% should come back between 50-150ms, and .4% will be well above that at about 150-700ms, and the outliers will probably be about 700ms - 3s. The point I'm getting at is that even with normal connections you can expect STRONG latency spikes in otherwise normal traffic on the internet. It's the nature of the beast.

4. This approach allows for us to "dodge projectiles" and melee attacks that we see coming, but let's face facts and compare this diablo 3. I can dodge spears or fireballs thrown at me just fine... why? Because there's no server-side check to make sure of if I dodged it, and if I didn't let's rewind to the last thing that worked. I'm either standing in fire, or I'm not. It's decided on my client and passed to the server. The server is not telling me if I'm standing in fire. That's why POE cannot fix this. It was a design decision, and it was strongly likely based in anti-cheating. Putting too much information client-side allows the user to cover up what's really happening and allows for a chance to change that information before it reaches the server.

I think that the end of point 4 needs to be reiterated. I would NEVER cheat in POE. The only time I want to cheat is when I die to desync... which wouldn't be caused if the client had more trust placed in it.

There is NO discussion on this. That IS what's happening. You can claim that it's a million other things, but until you realize that the clients need to be trusted, but checked in ways that don't cause user interference, that this game WILL suffer. It will lose you users. It will cost you money. And it will hurt your bottom lines.

This has been an issue for years now, and has sat near the top of your development manifesto for the better part of a year.

The reality is that testing in a 0-latency office is not going to produce desync. Hire someone in Boston to run Wireshark and packet capture EXACTLY what's going on. Find triggers, make bug reports, and stop dodging the issue. This is the BIGGEST problem holding this game back from being the GOTY. And you guys can't / won't throw the resources necessary at it to handle it. Which is saddening.


Videos of serious Desync:
Spoiler

https://www.youtube.com/watch?v=j8wTE71aXFI
https://www.youtube.com/watch?v=a7SI0S5b6fE
https://www.youtube.com/watch?v=oWOL_HTMT3o
http://www.youtube.com/watch?v=ECylSvEGqbg&
http://www.twitch.tv/ephixagames/c/4710896 -- Rank 1 death...
http://www.twitch.tv/willywonka_hc/c/4661939
http://www.twitch.tv/qstep/c/4797990


Examples of bullshit logic that don't need to be stated in this thread
Spoiler

It's part of the game, deal with it
If only that were applied everywhere in life we'd still be wiping our ass with leaves in caves.

Don't play using a build that is prone to desync.
Fuck you. If they didn't want us to use a skill, they should remove it from the game. This happens in other games when a skill is bugged. Stop treating Desync like it's not a bug. It is.

It allows us to do X/Y/Z, and for me, that's worth it
For 1: This isn't about you princess. This is about making a game that's fun for EVERYONE. 2. Other games that don't have desync don't suffer from these issues and have similar features. I can use leap slam in Diablo 3 to avoid a fireball and not rubber band back into creeps.

I never desync, therefore it's not a big issue
Great, either your ISP has better routing, or you live closer to Dallas, I'm glad it works well for you, but that's not universal. You're not hungry but that doesn't mean we've wiped out worldwide hunger.


You're confusing desync and lag
I'm a network engineer, I know what latency is.





I'd like to begin this thread by saying 9 things.

1. I do not suck at video games.
2. I enjoy playing hardcore in games. I have a level 70 hardcore char in EVERY class in Diablo 3 that can solo in at least Torment 4.
3. I am employed as a network engineer, still working for the largest CDN in the world.
4. I also work as a programmer, and as a programming teacher, where I teach roughly 25,000 students in C++ and Java (Online, obviously).
5. I really love playing path of exile, it's an AWESOME game that really puts a gigantic focus on customization and player-choice.
6. The above fact is totally ruined by desync. Not latency. Desync.
7. I have a 75/50 internet connection (FIOS) and my average ping to the POE servers is around 50-65ms
8. I desync roughly every 30-50 seconds. I blamed this on my build (flicker crit) for a very long time. I have started running with Ele Buzzsaw and I still have a GREAT deal of issues with desync.
9. I have died more times to desync (which is a bug) than to every legitimate gameplay mechanic combined. Calling this a "security feature" is a SLAP IN THE FACE to the players that support you.

As I stated in my post on the development manifesto, it comes down to trusting the client. If you trust the client about nothing else, you HAVE to trust it about it's current position. Code in cheat protection around the limits of how far a character should be able to move under certain circumstances... but at the end of the day, you HAVE to trust that the client knows where the player is. If you don't, you ruin the gameplay experience for the player.

This really isn't an issue that you can continue to hide from.



Combined between my 2 primary characters I have about 750 deaths, couple in all my other characters (level 39 - 72) and I'm at over 1000. And there's no doubt that a good amount of those I've earned on my own. I've probably died to Atziri double bosses a solid 15-20 times. Swapping gear on Dominus to cull is probably another 15-20. And sometimes I'd run out of frenzy charges in the middle of a swarm of monsters on my flicker crit char. Those all happened. I totally deserved those deaths. The other probably...600-700 times that I've died? I didn't deserve those. That's not what I entered into my client. And each one of those times that I've died despite taking action that would have prevented me from dying.




Based solely on the information I've provided so far I can tell you a few things as fact.

When I'm playing any other MMORPG / MOBA / RTS / Action game (be it Diablo 3, Wildstar, League, Dota2, etc) I don't experience the type of lag / desync that I experience in Path of Exile. Are there any plans to even look at this issue? Is there anything at all being done about this?
Last edited by demonskye#3413 on Nov 14, 2014, 10:13:26 AM
This thread has been automatically archived. Replies are disabled.
You must have missed the part where the devs said that putting complete trust in the client encourages hacking and cheating, and will not be done.

Sure - they COULD put a ton more trust in the client side of the game, but they won't.


http://www.pathofexile.com/forum/view-thread/889669



All online games have this situation. The server has to dictate whether things happen or not, but there's a 50-250ms delay before data gets to the server and back. There are three ways that games can solve this:

Trust the client. This means people can cheat, but the results are instant. We will not do this.

Wait until data arrives back from the server before doing anything. This is a very common strategy in RTS and MOBA games. If you click to move, the unit will only start moving once the server says so, which is 50-250ms later. If you are close to the server, you'll quickly get used to the lag and everything feels pretty good. If you're far away (New Zealand, for example), it feels like you're playing drunk. Every time you issue an order, nothing happens for quarter of a second. This does not work well for Action RPGs - the immediacy and pace of combat requires that actions start to execute instantly.

Start predicting the result of the action as though the server said yes, immediately. When the server later gets back to you with a result, factor it in. This is what Action RPGs including Path of Exile do. It means that when you click to move, or click to attack, it occurs instantly and feels great. The problem is what happens when the server decides that the action can't have occurred - that's when the game gets badly out of sync.
Last edited by JuniorHakensheath#1699 on Oct 28, 2014, 7:09:24 PM
"
JuniorHakensheath wrote:
You must have missed the part where the devs said that putting complete trust in the client encourages hacking and cheating, and will not be done.

Sure - they COULD put a ton more trust in the client side of the game, but they won't.


http://www.pathofexile.com/forum/view-thread/889669



All online games have this situation. The server has to dictate whether things happen or not, but there's a 50-250ms delay before data gets to the server and back. There are three ways that games can solve this:

Trust the client. This means people can cheat, but the results are instant. We will not do this.

Wait until data arrives back from the server before doing anything. This is a very common strategy in RTS and MOBA games. If you click to move, the unit will only start moving once the server says so, which is 50-250ms later. If you are close to the server, you'll quickly get used to the lag and everything feels pretty good. If you're far away (New Zealand, for example), it feels like you're playing drunk. Every time you issue an order, nothing happens for quarter of a second. This does not work well for Action RPGs - the immediacy and pace of combat requires that actions start to execute instantly.

Start predicting the result of the action as though the server said yes, immediately. When the server later gets back to you with a result, factor it in. This is what Action RPGs including Path of Exile do. It means that when you click to move, or click to attack, it occurs instantly and feels great. The problem is what happens when the server decides that the action can't have occurred - that's when the game gets badly out of sync.


Yes, I also read that post, and if you believe it you're just plain deluded.

Placing "ANY" amount of trust in the client does not enable cheating. Placing full reliance of a server doesn't automatically disable cheating either. I can use google to find websites with undetected bots for this game. How is desync preventing that? Furthermore, placing trust in the client with server-side checks on bounds makes a LOT more sense than just distrusting EVERYTHING the client says and making sure the server is okay with every move. It's insane. For people like me on the East Coast (I'm a 12ms ping from the largest Colocated DataCenter in Boston, which happens to be where I spend a good portion of my time) I have to connect to Dallas (173.192.12.40 goes to a Dallas DataCenter). Bare minimum with no packet loss I have a 60ms ping time. This is acceptable, but for a lost packet to timeout and be re-requested and to retransmit we're talking about 250+ms, which is still acceptable... but it often isn't even that clear. It usually ends up being more like 500-2500ms. Which just isn't good enough to play with.

Again, I don't care if the client doesn't trust where my attacks are, and those need to sync, at the VERY least you need to trust where I say I am. I don't run into the middle of a pack of mobs and then not attack. I'm sorry, I just don't do that. The client saying differently doesn't change that.

Lastly, let's use D3 as a broilerplate example again. Why can I dodge attacks and projectiles in that game in real time, and yet cheating isn't an issue there?

It's possible, shit it's not even THAT hard, you just need to trust your client, implement some logical kind of anti-cheat (one that punishes users with slower internet / that live further from Dallas is INSANE) and move forward with a MUCH better game. You can't have this on your development manifesto and do nothing about it. That's just paying lip service and admitting that a task is above your skill level. You made a MMOARPG, settling an issue of client trust and the latency issues it resolves should really be a no-brainer.
Bump for attention
actually, thanks to action prediction system (which is btw the only possible system), you are slightly out of sync ALL OF THE TIME.

desync problems are easily avoidable in case your computer and connection is ok. I dont remmember when I died to desync for the last time. People reporting constant desync dieing suffer from at least one of these issues:

1) lag and packet loss caused either by crappy system or connection (on client or ISP side)
2) high latency caused by long distance from server (usually coupled with #1 ^)
3) keeping on doing something stupid that makes deync go out of control - like using movement skills through the door, trying to run through enemies in crowded areas etc
4) inability to make survivable builds and blaming desync for their own incompetence

desync is something we have to pay for having such complex game mechanics, and for not having game completely full of hackers, cheaters and duped items..
IGN: Eric_Lindros
CET: Timezone
Last edited by Ludvator#6587 on Oct 30, 2014, 10:03:46 AM
"
Ludvator wrote:
actually, thanks to action prediction system (which is btw the only possible system), you are slightly out of sync ALL OF THE TIME.

desync problems are easily avoidable in case your computer and connection is ok. I dont remmember when I died to desync for the last time. People reporting constant desync dieing suffer from at least one of these issues:

1) lag and packet loss caused either by crappy system or connection (on client or ISP side)
2) high latency caused by long distance from server (usually coupled with #1 ^)
3) keeping on doing something stupid that makes deync go out of control - like using movement skills through the door, trying to run through enemies in crowded areas etc
4) inability to make survivable builds and blaming desync for their own incompetence

desync is something we have to pay for having such complex game mechanics, and for not having game completely full of hackers, cheaters and duped items..


1. In the OP I mentioned that I have a very capable system. I've shown screenshots of my FPS / Latency. When I die, I'm getting 200fps and no latency. I die, POE has a lag spike of like .971 reported then it goes back down to 50.
2. Boston > Dallas is kinda far.
3. Why have movement skills in the game? I rarely use them, but don't see any correlation to using them and my desync killing me.
4. Yeah, that's clearly not the issue.
Bump.
Don't know why you are bumping.

Desync is not a bug...they know about it. it can't be fixed....any future "fix" is going to make you rubber band more.

Go read the manifesto.
Ancestral Bond. It's a thing that does stuff. -Vipermagi

He who controls the pants controls the galaxy. - Rick & Morty S3E1
"
lagwin1980 wrote:
Don't know why you are bumping.

Desync is not a bug...they know about it. it can't be fixed....any future "fix" is going to make you rubber band more.

Go read the manifesto.


I've read the manifesto, reject their summary because it's essentially bullshit that they "can't" fix it. They refuse to. There's a clear-cut difference between the two. No other ARPG suffers from these issues and no other ARPG is suffering from wide-spread hacking the way they're concerned about. Plain and simple it's an engineering problem. This is literally what I do for work, I know what I'm talking about.
"
demonskye wrote:
"
lagwin1980 wrote:
Don't know why you are bumping.

Desync is not a bug...they know about it. it can't be fixed....any future "fix" is going to make you rubber band more.

Go read the manifesto.


I've read the manifesto, reject their summary because it's essentially bullshit that they "can't" fix it. They refuse to. There's a clear-cut difference between the two. No other ARPG suffers from these issues and no other ARPG is suffering from wide-spread hacking the way they're concerned about. Plain and simple it's an engineering problem. This is literally what I do for work, I know what I'm talking about.


Yeah, I am sure they "refuse to fix desync", a problem that drives some players away, because they hate money right?
“Demons run when a good man goes to war"

Report Forum Post

Report Account:

Report Type

Additional Info