What is Desync? (A thorough explanation)

"

adania wrote:

To the OP - Very nice post, explained it well and I feel like I've learnt a more in-depth knowledge of de-syncing from it.

Doesnt alter the fact its worse in POE than any other game I ever played (except Mortal Online which sucked satans blue balls)

The Desync at times in this game is appalling, feels like playing on a dial up modem circa 1995. In fact even D2 back in the day had less desync. SMITE which I play and is much faster paced has almost zero noticable desync.

Add to that that in fact POE is actually quite slow compared to many ARPGs and I do wonder why desync is quite so pronounced.

"

CrystalisQualinthi wrote:

You offered what appears to me to be an incomplete analysis of the situation. I'm not a programmer, but I *am* an IT professional and I understand bandwidth and things of that nature. Thus, if desync is caused by the client/server having their own version of the game that get out of sync with each other due to lack of communication between the two units, why would bandwidth optimization followed by faster sync between the client and server devices NOT alleviate the problem?

No it doesn't, and I already explained why (this has nothing to do with how fast or how much data is being sent from the server to the client, its to do with the server and client having incompatible states and the method of having to sync them)

States become incompatible due to things like packet loss and variances due to delay, GGG cannot do anything about this, its a result of being on the internet. Sync speed has nothing to do with it, you see that massive "jump" (when the game is out of sync) because its updating a lot of things that are out of sync, not because the bandwidth is too slow. If you have 10 units in the wrong position (because they are out of sync), it doesn't matter how fast you get the synced data from the server, it will still have to do a refresh which would suddenly move those 10 units in the proper position.

The problem is those 10 units being out of position, not how fast data is being transferred there and back

This isn't a bandwidth problem, its how GGG have implemented their netcode problem

Being an IT professional has little to do with understanding how stuff like this works. What matters is you understanding programming and networking (in the context of programming), which you clearly don't

"

deteego wrote:

This isn't a bandwidth problem, its how GGG have implemented their netcode problem

Being an IT professional has little to do with understanding how stuff like this works. What matters is you understanding programming and networking (in the context of programming), which you clearly don't

Interesting, I would think most broadband connections tend to have very little packet loss and packet ordering issues of that nature? I know how those factors can affect voice services and certainly could see how it would affect a real time application like the game. However, I have to wonder how often you lose packets or have a real variance of response time from the server.

Doing a ping test against the website IP, I can see zero packet loss and the ICMP packets are taking roughly 40 - 42 MS response each time. Would there be a difference in packet sizes that are being sent from the client to the server, which in turn could affect the order in which they arrive?

See, once you explain the actual problem in technical terms (such as packet loss and out of order packets) it made a lot more sense to me. I'm just uncertain how much of either of those are factors in modern internet connections.

"

CrystalisQualinthi wrote:

Interesting, I would think most broadband connections tend to have very little packet loss and packet ordering issues of that nature? I know how those factors can affect voice services and certainly could see how it would affect a real time application like the game. However, I have to wonder how often you lose packets or have a real variance of response time from the server.

Doing a ping test against the website IP, I can see zero packet loss and the ICMP packets are taking roughly 40 - 42 MS response each time. Would there be a difference in packet sizes that are being sent from the client to the server, which in turn could affect the order in which they arrive?

See, once you explain the actual problem in technical terms (such as packet loss and out of order packets) it made a lot more sense to me. I'm just uncertain how much of either of those are factors in modern internet connections.

Packet loss happens rarely, but its bound to happen. The network is made up of many links from the client to the server and if one of the routers from the client to the server is overloaded, they will drop packets (this is networking 101 which you learn at university). Its basically the black swan theory (http://en.wikipedia.org/wiki/Black_swan_theory) in practice. You are basically certain that packets will get lost at some point, you don't know when, but it is going to happen and you have to account for it

Now the thing is, normally if you have the client executes command -> server confirms and then sends reply confirmation -> client receives and then reacts (usually displays whatever that command is graphically), this isn't an issue, because if the packet is lost than no confirmation command is received, and so the client will quickly try again (this obviously does impact on speed on delay, but as I said earlier, if you have a 50ms connection its barely noticable). There are other ways to confirm if the command is received or not, but there mere fact that the client is always asking for confirmation means they are always going to stay in sync

On the other hand with GGG's model, if a packet containing a command (lets say move 5 metres to the north) is lost, then what happens is that the client still moves you 5 metres to the north (because its not asking for a confirmation from the server, remember they are keeping seperate states), however the server (since it never received the packet) still thinks you are in point A (which was before you moved those 5 metres to the north). So every single command after that is going to be out of sync with the server. At some point, the server realizes that the client does something incredibly stupid/is not possible, and would deduce the client is out of sync, and its at this point the server will resend all data about the game state (which means all the positions, healths, debuffs of all the units on the level) to the client, so it can update itself (and this is when you see the jump). The /oos command which GGG removed did this forced sync update on demand, and the command was removed because of the amount of stress it put on GGG's servers

Now the thing is, nothing I have said has anything to do with time or speed. The packets can be near instant, but its completely irrelevant to packets getting lost (which is what causes the problem), and its the packets that get lost which causes the desync for PoE, because essentially loses a "command" (which is inside the packet) which the client still thinks is valid, and the server is not. This isn't an issue in most modern games, because they always ask for confirmation if the server received the packet, and so they don't get out of sync at all

The compression argument is a furphy anyways for numerous (other) reasons (like disregarding the fact that this wont fix the problem at all). The most obvious one is there is a limit to compression due to entropy, and the further you get to that limit the much more computationally expensive it becomes for the server (encrypting stuff on the fly takes a massive amount of CPU), and the mere fact that compression takes up so much CPU power that it may not be worth it to compress stuff at all

There are other reasons why the states can get out of sync, like packets coming in the wrong order or really large delays, it gets quite complicated at this point and it depends how specifically GGG coded PoE

Everyone should be thanking deteego instead on OP, that was a very thorough explanation. I'm a software developer myself, but not having to deal with networking, especially real time, i never had a clear view of how these things are handled.

Great post OP, I only skimmed it, but understood the major points.

I wonder how Jitter and Packet Loss plays into it all.... Obviously negatively, but still.

Most of Path of Exile's sync problems have nothing to do with packet loss, they're simulation mismatches. That's why it happens way more often in rooms with many monsters and even more often than that (and to a larger degree) when said room of monsters tries to navigate through a doorway or other small opening.

Also, despite the overall networking scheme, it is not clear to me why they can't do partial locational updates of monsters near the player/actively attacking. It appears they do this with health values (ever notice you hit a monster and the health bar goes down a quarter second or so after the fact?) and if true that implies they have some sort of tech for that. Sending across location updates should be fairly cheap in a game like this (2 byte monster ID + 2 byte x 2 axes, can ignore the up direction since the game doesn't support multilayer dungeons). That certainly wouldn't magically fix the core issue but it'd be significantly less bad and at least you wouldn't die by invisible non-animating monsters.

"

Mysterial wrote:

Most of Path of Exile's sync problems have nothing to do with packet loss, they're simulation mismatches. That's why it happens way more often in rooms with many monsters and even more often than that (and to a larger degree) when said room of monsters tries to navigate through a doorway or other small opening.

Yeah as I said in my post, it could be mismatch in position due to delay (which is what causes simulation matches). Simulation mismatches can happen due to many reasons, it can be packet loss and/or delay. Like I said, I have no idea what actually specifically causes the game to go out of sync, but its either of those things (or both)

Point is, in either case, the things that cause this are out of GGG control. Packet loss GGG can't control, nor can they control delay to their clients

"

Mysterial wrote:

Also, despite the overall networking scheme, it is not clear to me why they can't do partial locational updates of monsters near the player/actively attacking. It appears they do this with health values (ever notice you hit a monster and the health bar goes down a quarter second or so after the fact?) and if true that implies they have some sort of tech for that. Sending across location updates should be fairly cheap in a game like this (2 byte monster ID + 2 byte x 2 axes, can ignore the up direction since the game doesn't support multilayer dungeons). That certainly wouldn't magically fix the core issue but it'd be significantly less bad and at least you wouldn't die by invisible non-animating monsters.

Of course its possible, heck they can solve the current issue by force syncing every interval, however apparently its ridiculously expensive in terms of bandwidth and/or processing power for the servers. I am assuming that when a forced sync happens, it basically sends the entire state that is relevant (in PoE's case it might be every single monster for the current instance, their HP/debuffs, position etc etc)

All I know is that /oos command was removed specifically because it was causing way too much stress on the servers, and that due to how certain abilities work, partial updates may not be technically the correct answer for this type of game

In fact I am not sure partial updates would work, some spells can theoretically go on forever (ice spear) which would mean that you would have to technically speaking partially update the whole level, which isn't any better than what they are doing now. It could actually create more problems because you are now creating a 3rd state (partially updating the screen is creating a 3rd state, which is a separate state for the rest of the level of the client, and also obviously a different state for the server).

In any case, this is all kinda speculation because we have no idea how specifically GGG's netcode works, im sure if there was a simple solution to this they would have already done it

Was there any insight given from GGG about how the consistency maintenance is working in the current network code?

If the current concept cannot be fixed in the long term, maybe a rewrite should be considered (as was pointed out before that takes a LOT of time/money).

In my opinion the main problem is that the desyncing prevents the introduction of new gameplay mechanics, which PoE is currently severely lacking.

Well, whatever the perspective I hope we can all agree that authentication issues trump desync fixes for the time being (and within reasonable temporal parameters according forthwith pursuant to the applicable juncture herein)?