Reasons Why Desync Happens (Thoroughly Explained /w Cited Commentary On Fixes & Focuses Info)

"

SilasOfBorg wrote:

Scrotie -- as usual -- is bang-on.

Actually I redid Point #2, because I feel that I didn't do the topic justice. You can read my edit above (edit: except new page lol).

"

ScrotieMcB wrote:

Actually I redid Point #2, because I feel that I didn't do the topic justice. You can read my edit above (edit: except new page lol).

Heh.. the original point was sound enough. Fewer bytes on the wire = more efficient usage of existing bandwidth. Another aspect, as well, comes in compressing the information to something brief enough to fit independently in an MTU (shoot for 1400 bytes I guess?), now you have one packet with <all the information required for X> instead of two -- much more robust.

"

ScrotieMcB wrote:

What GGG Really Should Be Focusing On With Desync

1. One-way UDP for temporary things, two-way TCP for commands and permanent changes
Right now Path of Exile using TCP for everything.
Read this: UDP vs. TCP.
That author goes a little too far, since TCP would still be smart for certain things (player mouse clicks and keyboard presses, inventory management), and in today's world we should assume all routers are already using TCP in some form at all times (cells sharing WiFi)... but the enthusiasm is good

what would you consider "UDP-worthy" ? you mention mouse and keyboard is not, as, i assume, in your opinion their order and continuity are too important. what part of the data transferred between the client and server is not important enough to be ordered, and for that matter can be considered temporary? the server can't even entrust the map to the client, and the reason being maphack, but would you be satisfied if the map would just appear on your client in random sequence - one part missing in the middle and another appearing? off the top of my head, i cannot think of a single thing that is less (or more for that matter) important than keyboard and mouse, and not important enough to be out of order. there are no temporary things, only one big continuity. in my opinion, in places where desync matters most, continuity matters most as well. then again, i am not a game designer.

"

2. True packet information entropy (both brevity and compression)
Right now I doubt GGG has taken efforts to make packets compact, and is instead focusing on making them readable by humans, which greatly increases bandwidth consumption.

This isn't exactly non-standard for the industry; for example, the standard HTTP header* that comes attached to web pages you view looks something like this (which would be shrunk about 15% with lossless compression):

Example HTTP header

GET /tutorials/other/top-20-mysql-best-practices/ HTTP/1.1
Host: net.tutsplus.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120
Pragma: no-cache
Cache-Control: no-cache

However, if we convert it to this, it's 32% smaller, through brevity alone, loses zero meaning, and further compressible to 39% smaller:

Brief HTTP header

GT /tutorials/other/top-20-mysql-best-practices/ H/1.1
Ho: net.tutsplus.com
UA: Mo5.0 (Win; U; WinNT 6.1; en-US; rv:1.9.1.5) Gko/20091102 FF/3.5.5 (.NET CLR 3.5.30729)
At: t/html,a/xhtml+xml,a/xml;q=0.9,*/*;q=0.8
AL: en-us,en;q=0.5
AE: gz,def
AC: ISO-8859-1,utf-8;q=0.7,*;q=0.7
KA: 300
Cn: k-alv
Ck: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120
Pr: no-c
CC: no

Read this: Entropy, Compression, and Information Content.
Reducing the size of packets using these methods would help with bandwidth issues (both for GGG and the players who have them) and also help ease the packet flow issues between TCP and UDP that where mentioned by the author in the TCP vs UDP article.

* Although I've run packet capture on PoE before (to determine that PoE is 100% TCP), I haven't decrypted the packet contents, so I'm using this as a blind example. It's very hard to tell what GGG is doing on this front.

http is designed to burst information and then disconnect or rest. it was not designed for compact data transfer, or continuous and fast data transfer. some other protocols, such as IP, were designed to be compact. so compact, in fact, that IPv6 seems inevitable. but writing an IP reader, that would translate it's contents into a readable form is as easy as it gets. just use a table correspondence between the values in the IP header and readable sentences that explain each possible value, and you even have the RFC as a reference. i wouldn't think it would be too hard to do a similar thing to anything else that you have a reference to, including (for GGG) a protocol GGG might have written for the game. http is a different tool for a different job. i don't even think that any great overhead in the protocol exists at all, since, unlike http, there is no need to send some header information over and over again - different purpose means different design. i highly doubt GGG made a protocol similar to http. and i highly doubt that in the multiplayer gaming industry it is standard or common. but then again i am not a game designer.

"

3. Smart sync rates
Players in town need sync information a lot less often than players in the middle of combat with rhoas. The amount of anticipated combat stress on a character should guide how often synchronization packets are sent out, allowing for more sync attempts in heated situations at the expense of situations where they are not needed, allowing better performance with the same total server bandwidth.

i think that OP made a good job going over it all generally, and i cannot go into more detail since i am not a game designer. OP also listed possible scenarios that would be beyond the game's control. going into specifics in this thread just seems off topic, especially since it is all speculation and for all we know, they are doing it just the way you suggest. i would obviously assume that when the game AI on the server needs to update something (or another player), which in combat situations happens quite often, one can imagine, sync attempts are made. any data transfer is in fact a sync attempt. and if the mobs are not moving, and neither is the player, it is not a combat situation, which means no sync attempts necessary.

"

SilasOfBorg wrote:

HeavyMetalGear, I realize you mean well. But please don't make statements like this:

"

"If the software were written well enough, Sync Errors would never happen even with sub-par hardware.

Why, because it holds some truth? I've played countless other FREE and NOT FREE online games that either have way more players playing, beg for players to have more computer performance, etc. and have never experienced desync as bad I do in PoE in my 14 years of on-and-off online gaming, regardless if using a Wireless connection or not.

That's the reason why GGG is hesitating on fixing this issue because it would require a huge overhaul somewhere in the code/writing of the software because somewhere along the lines there's an error, and then an error behind that error, and another error in the code behind that error . . . The list just goes on.

I never said it was easy to do/fix. I said had they been more careful in the writing of the software we wouldn't have the desync issue as bad we do now. That's not implying that we would be 100% free of desync; I am saying it would be a lot less.

Now, they have this massive issue on their hands and it's a conundrum within an aneurysm to fix! The devs brains lock up when trying to deal with it.



That said, Thank You for reviewing this Thread.

"

0nin wrote:

"

HeavyMetalGear wrote:

However, we can list the things which can make those Desync happen more or less often:

Your computer can't keep up with the global simulation: It can be because your computer is too slow, or because your ping to the Server is high. In any case, you can't exchange data fast enough with the Server to keep up with the current state.
Your connection bandwidth is insufficient: Some games exchange far too much data to keep the clients in sync. If you can't send or receive enough to keep up with all the data necessary, you will Desync.
Your connection's quality is poor, and you lose a lot of Packets: If you play on WiFi with a bad range, you are likely to lose Packets. You will lose information the Server sent, or the Server won't get your changes. If the game doesn't handle this correctly, it can lead to Desync. Note that lost Packets can happen even if you have a great connection. It can just happen anywhere between you and the Server.

"

0nin wrote:

Translation:
Having slow internet causes desync

Truly you are wise beyond our understanding.

It's a thing called elaboration. That's why the Thread has 'Thoroughly Explained' in its title line.

Just saying, "Having slow internet causes desync" is not good enough! And for the cherry on top, regardless of good or bad internet connect, wireless or not, desync is still a problem for many players during gameplay, notably because it's an issue on the developers end, not always on our end.

P.S. Sarcastic remarks do no good for important issues like this. Please offer some feedback that can be of more help to GGG/PoE next time.

"

Dao wrote:

what would you consider "UDP-worthy" ? you mention mouse and keyboard is not, as, i assume, in your opinion their order and continuity are too important. what part of the data transferred between the client and server is not important enough to be ordered, and for that matter can be considered temporary? the server can't even entrust the map to the client, and the reason being maphack, but would you be satisfied if the map would just appear on your client in random sequence - one part missing in the middle and another appearing? off the top of my head, i cannot think of a single thing that is less (or more for that matter) important than keyboard and mouse, and not important enough to be out of order. there are no temporary things, only one big continuity. in my opinion, in places where desync matters most, continuity matters most as well. then again, i am not a game designer.

First things first: All UDP packets would have on them a simple timestamp (server time, since server is only UDP sender). All client calculations would pretty much start with "IF ThisTimestamp > LastTimestamp" -- so getting an out-of-order UDP packet wouldn't revert any gamestate, it would simply be discarded. Since we're not anticipating desyncs longer than a minute (timeout before that), a short that incremented each millisecond, looping back to zero after max, would do the trick.

The importance of the UDP protocol here is that we don't care about order here, we just care about the newest information, and the older information can be completely ignored -- hence no need for ordered delivery with the oldest packet always arriving first. (It also helps that lost packets won't be retransmitted, since they'd be moot by the time they got there anyway, and a waste of bandwidth.)

That said, TCP should be used for things like these:

• Player commands: User uses SkillA at position x,y
• Zone changes: entering entrances/exits, portals, waypoints, etc.; player is dead, character league transfers; login/logout, character selection
• Inventory: Unidentified unique Glorious Plate from ground to user inventory; Items X, Y, Z available at vendor; transfer Kaom's Heart from inventory to stash
• Character maximum/permanent stats: Max life, max mana, player's current level, XP to next level, passive point allocations, remaining refund points; name, class
• Party "permanent" stats: Name, class, level of party members; party member death; players leaving the party
• Simple idle/timeout detection

Many of these server-to-player TCP communications would require a client TCP acknowledgement; you don't want to start streaming UDP position data while the client doesn't know you've changed zones yet, or the client doesn't know it has a new party member.

Pretty much everything else -- everything that would fluctuate often during combat -- would be UDP:

• Current position: of player, of monster, of projectiles, of items on ground (only goes TCP once picked up); minimap info
• Character state: Life, ES, mana, XP
• Party info: % life, % ES, current position (including minimap)
• Monster/minion data: Monster type, facing direction, alive/corpse, white/magic/rare/unique, affixes
• Animation data: Identifier, position, current progess, end time
• Buff data: Curses, auras, charges, etc.; level of effect, time remaining

None of these would require a TCP acknowledgement, because newer UDP information is anticipated to be en route anyway.

I probably didn't cover all of them, because a bunch of minutae is involved, but the core concept is that if it doesn't change much, use TCP, and if it changes very often, use UDP. Note that some are kind of arguable; for example, you could argue sending the monster type, white/magic/rare/unique and affixes data TCP since it doesn't change much, but this might delay, then you'd need an ACK to start on the UDP, which might slow things down, so I opted to just resend it many times (this is why compactness is important).

"

Dao wrote:

i highly doubt GGG made a protocol similar to http. and i highly doubt that in the multiplayer gaming industry it is standard or common. but then again i am not a game designer.

In my experience, the emphasis is on debugging clarity, which leads to very human-readable data. It's rare to see a strong emphasis on compactness, and instead to see a focus on understanding how the client misinterpreted a received communication.

"

Dao wrote:

any data transfer is in fact a sync attempt. and if the mobs are not moving, and neither is the player, it is not a combat situation, which means no sync attempts necessary.

However, players are moving in town -- even when the player playing the game isn't. Obviously we don't need update on that, nor on the player's progress from point A to point B after a command to move to point B, as much in town as we need it in non-town areas. We can send less packets and rely more on the client to predict player positioning.

Nor do we need it as much when a player moves from point A to point B in a non-town area when not in any danger whatsoever. Which means you need some kind of monster proximity/quantity test to measure danger.

ScrotieMcB, I don't know if GGG themselves has all this background information you do (I know some myself,) but you should be on their team as an ideasman to help fix this issue.

Let's just hope someone on the Development Team gets these messages.

There are many other Threads on Desync, but this Thread is more significant with a wealth of information within just a few comments made thus far.

EDIT: I am moving some excerpts you made I will quote front-and-center in my Thread.

I can't help but feel there is an "obvious" (well, at least plausible) solution for this everybody is missing.

Like, there should be a way to create a new protocol between the POE client and server to deal with most of this shit.

Hmm, first a question:

Does the POE server, 100% know whether a "POE packet" is legit or not?
Basically, when the server gets a POE packet, can he know 100% whether it was made legitimately by the POE client, or whether it's a fraud by some hacker trying to cheat?

That could be pretty important since it could help make a better algorithm for this, if the server can assume the stuff he gets is legit and plan accordingly

"

gonzaw wrote:

Like, there should be a way to create a new protocol between the POE client and server to deal with most of this shit.

LOL no. These are internet protocols, not proprietary ones. You can't just invent your own internet protocol and have it work; it requires coordination across the entire internet, and programming in every router along the way. You're pretty much stuck with the existing methods of data transport.

"

gonzaw wrote:

Does the POE server, 100% know whether a "POE packet" is legit or not?
Basically, when the server gets a POE packet, can he know 100% whether it was made legitimately by the POE client, or whether it's a fraud by some hacker trying to cheat?

I don't know all the nitty-gritty on this, but PoE uses encrypted communications. This means that spoofed packets would have to know how to encrypt the communication the exact same way the client would. This raises the bar to the point that botters prefer pixel-based detection (capturing data from the video card and analyzing pixels to determine the location of monsters and items) to network interception. And there's really no point in raising the bar higher than that, because PoE still needs to run on a variety of video cards, and it's the video card being compromised, not the communication.

It is worth pointing out that only the TCP transmissions would require encryption. UDP would be entirely server-to-client and thus wouldn't really require spoof prevention, unless Kripparrian gives out his IP address on stream or something stupid like that, and someone tries to man-in-the-middle his screen. So I guess it's possible, but I'm not really worried.

This is a senseless, pointless thread. GGG have already explained why desync happens the cause of said desync and the possible solutions they are working on these kind of threads only serve as hyperbolic gathering points for 3rd party citations. You want to fix desync? do you live in auckland? Otherwise 3rd party (fixes) really are senseless. But it's cure there's always a hero in every game w/ fixes to every solution. Oh, and i'll add unless you're not playing on your region gateway and have dial up connection desync isn't a problem.