Reasons Why Desync Happens (Thoroughly Explained /w Cited Commentary On Fixes & Focuses Info)

"

gonzaw wrote:

...wat?

But how would the game work then?

You just see a little portion of the map and every time you move the portions you visited are erased? So you can't complete the minimap in the client?

No, the client stores the previously received data and continues to use it. It's just that the new data is delivered as it's explored.

"

gonzaw wrote:

Why is the exit thing necessary? So they don't run bots that run straight to the exit all the time and finish everything fast?
...is that a problem at all?

It could be. Especially as a hack for actual players (not bots).

Necro'd to say that apparently all the worrying I was doing about sending the map data to the client piecemeal was a lot more than GGG was doing... because they were and are sending it all to the client at once during the instance loading screen.

As a result, maphack and zoomhack are both current available as UI hacks, and map data can be (and for all I know is being) integrated into bots for improved navigation.

This thread is definitely going places...

This "solution" about using both TCP and UDP protocols for handling anything makes me die a little inside. Instead of using one socket, do you really suggest using two simultaneously?! It won't solve anything, as it only increases complexity substantially for no significant gain.

This suggestion totally holds no critique. It's more like offering for a broken cart to have 4 more wheels to "fix" it. The cart is already good to be carried with only 4 wheels, why doubling that Jay Wilson style?

In any case, this "proposal" looks like an attempt to treat developers like little kids pretending you know it all better. And for someone to then point a finger at themselves and say - here, look, it was me who originally suggested using both UDP and TCP (so I am actually the one to blame for the broken game, sorry guyz. I crapped again just like it was with life nodes, when I claimed that was totally my idea, GG GGG)

And about the link to some random guy speaking about Internet Communications 101, with all this packet compression and stuff - why is it even remotely relevant? Do you think Path of Exile developers don't know this already? For me it's like some kindergarten guy advising how to better solve equations to a math professor, and trying to involve as many of his kindergarten friends as possible to strengthen his point of view. The article in question even contains spelling errors, how can it be treated seriously?

Grinding Gear Games already made awesome work to make Path of Exile. Yes, game engine has some limitations and problems, mainly due to chosen game features, but actually GGG have very strong arguments to have it exactly this way, and they actually stated all this in Dev Manifesto, but let's reiterate:

- forever abandoning hit rating could mask all possible desyncs so well, that you can say the issue is fixed completely (just as in D3), but the game would feel lame without this mechanic (so to say, I feel chance to hit should be buffed a little, Resolute Technique is too strong now, but it's a different topic)
- treating client side better could mask desyncs very well also (like in WoW), but could open up possibilities to cheating (WoW used to employ Warden for detecting cheaters for afterwards banning), and it's pretty understandable that GGG would rather spend their very limited resources on actually developing a better game for everyone

Proposed "hybrid protocol" will not solve anything aside from new jaw-dropping effects like seeing drops that were already picked up, not all loot showing up after monster dies, or no loot at all.

Client gets UDP message with server time 995 stating that some white item drop.
Client gets UDP message with server time 990 stating that Kaom's Heart dropped.
Client gets TCP message with server time 1000 stating that Kaom's Heart was picked up by someone.

How do you plan handling this very situation, when you auto skip second UDP message due to timestamp, and what should client then do when TCP got message about something that it never even know was around? If you intend to ignore the message, it will open up possibilities for hidden errors ("best-case" default behavior is never error prone), and if you intend to log everything, you'll gonna need even more memory for that. Short Allocation loot mode anyone? How can one pick what was never seen?

TLDR: Synchronizing two simultaneous data streams will definitely lead to more problems then what they can (probably) fix. All this hybrid protocol thing sounds like an ass talking to me. I have ten years of working programming experience, and things like that just never work in production code and are first targets to elimination already at design stage.

"

Daefecator wrote:

This "solution" about using both TCP and UDP protocols for handling anything makes me die a little inside. Instead of using one socket, do you really suggest using two simultaneously?! It won't solve anything, as it only increases complexity substantially for no significant gain.

This suggestion totally holds no critique. It's more like offering for a broken cart to have 4 more wheels to "fix" it. The cart is already good to be carried with only 4 wheels, why doubling that Jay Wilson style?

The way which packets are handled by the bastard routers of the bastard internet between server and client is indeed an area which presents opportunity for gain, so yes, I am suggesting two sockets here. Which wouldn't be one cart with twice as many wheels, but two carts doing different jobs through specialization of labor.

"

Daefecator wrote:

In any case, this "proposal" looks like an attempt to treat developers like little kids pretending you know it all better.

I really am not set out to give GGG a hard time, but you do know that they actually have been sending the entire map seed to the client on instance load for months now, allowing maphack this entire time... a security violation that I would have lolnoob'd gonzaw for suggesting?

They're fixing that now, but after that, I really can't assume anything anymore.

Nevertheless, you have a point about unverified speculation. There's a difference between armchair programming and prototyping; maybe I should become more interested in the latter.

"

Daefecator wrote:

Proposed "hybrid protocol" will not solve anything aside from new jaw-dropping effects like seeing drops that were already picked up, not all loot showing up after monster dies, or no loot at all.

Client gets UDP message with server time 995 stating that some white item drop.
Client gets UDP message with server time 990 stating that Kaom's Heart dropped.
Client gets TCP message with server time 1000 stating that Kaom's Heart was picked up by someone.

How do you plan handling this very situation, when you auto skip second UDP message due to timestamp, and what should client then do when TCP got message about something that it never even know was around?

The UDP packets would not deliver notification of actions (this monster activates Ice Spear, that item dropped); they deliver notifications of positions (this monster is located here, Ice Spear projectile is located there, that item is located somewhere else). This has been the message consistently throughout the entire thread.

UDP packets deliver the entire packet, or no packet at all (unless using a specialized UDP variant such as UDPLite). Therefore, the 995 positions packet would include the position of the 990 Kaom's that dropped. The 990 UDP packet would be safe to drop. And no 1000 TCP message would be sent to the client anyway; it's not going to their inventory, so instead they'd get another UDP positions packet where the Kaom's is gone.

"

SilasOfBorg wrote:

Good lord people, stop making it more complicated than it really is.

MrMrMr nailed it. Desync cannot be "solved" without a trusted client, something GGG would be 150000% foolish to do.

Here's a concrete example:

Assume that at time T, client and server are 100% in sync.

At that exact time, you attack a mob that is running past you. The client calculates (correctly) that you attack fast enough to hit that mob, and you're close enough to hit them mob. So it does not try to issue any movement commands, plays an attack animation, and sends the "player attacked mob #12398" to the server.

This information gets to the server at T+100ms. By that time, the mob has moved *just* far enough so that its out of range of your melee attack. If you were holding shift down, you just miss. If you weren't holding shift down, the server interprets this as wanting to chase down the mob and starts moving you towards it.

Either way, the simulation is now out of sync (de-synced).

This is why some people ALWAYS attack with shift down -- without it, desync can turn what YOU think is "attack that mob" into "chase that mob into the next room and die".

This is only one situation that causes desync, there's plenty more (stun!)

I HATE posts insinuating that desync is just something that GGG can magically fix with better coding. They can mitigate it somewhat, but the magnitude of coding wizardry that needs to happen behind the scenes to do this without compromising simulation integrity is not to be underestimated.

HeavyMetalGear, I realize you mean well. But please don't make statements like this:

"

if the software were written well enough, Sync Errors would never happen even with sub-par hardware

Scrotie -- as usual -- is bang-on.


p.s. in before some smartypants mentions that I am technically incorrect, in that the problem could be solved with sufficient memory and computing power -- "all" that they'd have to do is keep the last second or two of simulation 'slices' (lets say one per 10ms), receive commands like "player attacked mob #12398 at time T", back up the simulation to time T, apply the command at that point, and then very quickly re-run the sim to catch it back up to the current time. Even then, that would only work for single-player.

The solution would be completely trust the server and do not make predictions. Each action would be done ONLY after the server says it can be done.
U may say: oh, i tell my char to move and it starts to move only after 100 ms?? yeah thats right. I thinks its better this way than being killed by desync.

If all game is on server side, there is nothing to be in sync with

The problem was already named "trusted client".

The server says too much "no, desync!" or "you are dead since 2sec, you know that, don't you?" :)
95% of my deaths, no, 99.99% are caused by desync. To say it nicely, it sucks.

I hope they will come to their senses and give all legit players a trusted client and gfys to the hackers.

I dont give a sh!t about this witch hunting, i want fun while playing and only die because I suck and not the connection! :D

Geez!

Will never happen, but i tried!

The short fix is "path of lifenode", "unwavering stance", "no damage passives", "trade insane gear" and have a father called "papparian".

I'm just kidding, but you can't deny a little bit truth.

*sadface*

"

ScrotieMcB wrote:

UDP packets deliver the entire packet, or no packet at all (unless using a specialized UDP variant such as UDPLite). Therefore, the 995 positions packet would include the position of the 990 Kaom's that dropped. The 990 UDP packet would be safe to drop. And no 1000 TCP message would be sent to the client anyway; it's not going to their inventory, so instead they'd get another UDP positions packet where the Kaom's is gone.

Well, I am sorry to burst your bubble here, as I see You really want to improve the game as much as I do, but still...

Two sockets will cause much more problems. It is exactly making a cart have twice more wheels, as both protocols are self-sufficient. You use UDP for things like internet radio, and TCP pretty much anywhere else. TCP is exactly a phone call, and UDP is exactly an SMS service. So basically in order to speak you suggest reading SMS messages from your recipient on top of an already established call? How sane is that? It is not. That is why I tell You this will just add more problems on top of what you already have. For my ten years of programming experience I invented enough rocket science technologies, and I tried a very similar thing just-for-lols. It never worked properly on top of it's over-complex. A single TCP socket is significantly more stable and reliable. Do you have working prototype of what You suggest, or You just blow an air right here?

To explain a bit about the situation You quoted from my last post here, You examine the situation in whole, that's why you were able to make an educated guess of how this should have been resolved. Unfortunately, the server is not guaranteed to have all this data in time. What should it do at time 991? You just suggested it to include the data, that was not generated yet (packet 995) to be sent as a single packet. So ok, now specify the point up to which the packet should buffer it's data? And please remember, that real underlying UDP (as well as TCP) packets are limited in size. Not that they are very small, but limits should be taken very seriously especially in what you are suggesting - deliver all or nothing in a single packet - if data overflows the packet size, it's split into several UDP packets automatically by underlying socket implementation, but failure to deliver even one packet of such meta-packet will silently cause failure to deliver all the meta-packet wholly (as no retransmissions are queried or made) thus transmission reliability is drastically reduced.

You also skipped the raised question about Short Allocation. What will you say to people that suddenly have not even a lag spike any more, just sudden UDP packet loss (which can not be predicted or shown via simple latency graph) - and loose valuable drop allocated to them, but they didn't see it at all? All the looting after this point becomes moot. People that have more reliable connections with less UDP packet losses on the way to them are automatically favored. And all this I repeat once more on top of having much more complex connection protocol.

What is the point of having significantly more complex protocol, that will cause at least a) synchronization issues, and b) UDP packet loss (totally unpredictable)?

Yes, I agree, that something should be done to resolve severe de-synchronizations of game states between server and client. Implementing this new shiny dual-protocol hybrid monster will open much more probabilities for game states to desync even further so actually this dual thing substance looks like chocolate, but smells like... Well, not like a teen spirit exactly.

As for the whole map being sent from the server to client at load time - well, it's nothing new. It's not like "OMG, the sky is falling" situation as you pretend it to be via use of multiple facepalm memes, and acting like you would have done it at least 10x times better. At most you could suggest using another eight-wheeled cart to solve possible maphacking. Sure, let's fantasize together - basically one will need to implement torrent-like downloadable area map, so it will get assembled like a puzzle from pieces dynamically, and you receive only those pieces that you see. Oh wait, you also need the adjacent pieces as well. Oh, you just used quicksilver flask, and now you move far too quickly in order to receive all the puzzles you need to display, so ok, let's send you three puzzles ahead in the direction you are heading to, and now, oh no!, you changed the movement direction quickly while quicksilver is on you, and now you are moving at a black non-rendered area... Ah yeah, here is the puzzle you need to render and bam!, you standing right in the wall or even out of map space entirely! So many problems that you'll get while you also need to actually implement all this new shiny technology. And by the way, having said all that, don't you realize, that GGG surely already though about this and already made a compromise solution - the one that works pretty reliably and still is easy enough to implement - to send all the map at loading time! In order to maphack anything, one has to decipher an encryption GGG is now using (if any at all, as it is still beta), and even after someone succeeds at doing that, changing an encryption algorithm is like a several hours of work instead of several weeks developing and testing all this puzzled map.

I could understand the suggestion about life nodes, because you were appealing to simple logic that was at least understandable by pretty much anyone who played the game for several hours. This suggestion, however, is definitely like suggesting a math problem solution to an academic by a kindergarten guy with a second school year math book in his hands he refers to.

Life node suggestion was at least understandable. Dual-protocol is an ass talking, sorry.

"

Daefecator wrote:

You use UDP for things like internet radio, and TCP pretty much anywhere else. TCP is exactly a phone call, and UDP is exactly an SMS service.

You sure about that?

"

Daefecator wrote:

So basically in order to speak you suggest reading SMS messages from your recipient on top of an already established call? How sane is that? It is not.

This is old-fashioned for the sake of being old-fashioned.

I've been a bit of a one-man crisis hotline in my time; I've had people calling me in cases where shit is going down but they don't want the law involved, or someone's hurt bad but ambulances are expensive. I normally tell them to stay on the line, just hit home on their smart phone (everyone has one it seems) and send me a text with their address. Why? Because fuck pens, that's why. Then I'm talking to them while I'm pulling up navigation.

When I'm not being all responsible, I'm routinely going to Google, Wikipedia, or IMDB during longer phone conversations in the interest of furthering casual conversation.

Yeah, I'm insane, just like all the other whippersnappers with their Androids and such.

I think this is a microcosm for your objection to a dual-socket utilization. Yes, it's more complex, and it requires knowing how to assign particular communications to the appropriate communication method. But that doesn't make it less efficient.

"

Daefecator wrote:

You just suggested it to include the data, that was not generated yet (packet 995) to be sent as a single packet.

No I did not. The 995 packet contains the 990 information, not the other way around.

"

Daefecator wrote:

And please remember, that real underlying UDP (as well as TCP) packets are limited in size. Not that they are very small, but limits should be taken very seriously especially in what you are suggesting - deliver all or nothing in a single packet - if data overflows the packet size, it's split into several UDP packets automatically by underlying socket implementation, but failure to deliver even one packet of such meta-packet will silently cause failure to deliver all the meta-packet wholly (as no retransmissions are queried or made) thus transmission reliability is drastically reduced.

You make "rocket science technologies" and you can't think of ways around this limitation?

I'm suggesting that player stats (life, mana, ES), monster locations, monster stats, projectile locations, and item locations all be sent via UDP. Getting all of those into a single packet MIGHT be a challege. However, you could divide those up; the packet data would start with a category attached to the timestamp. Then if the client already has a monster location packet at 1000 and an item location packet at 990, then get both kinds of packets at 995, it only throws out the monster location packet.

Or, if you have the RAM to spare, you could assign a last_updated variable to pretty much everything client-side, then not even worry about categorization. When it updates information, it uses the packet's timestamp and then checks that timestamp before using that part of the packet data.

I mean seriously this isn't hard logic, it makes me wonder if you've ever actually programmed before.

"

Daefecator wrote:

You also skipped the raised question about Short Allocation. What will you say to people that suddenly have not even a lag spike any more, just sudden UDP packet loss (which can not be predicted or shown via simple latency graph) - and loose valuable drop allocated to them, but they didn't see it at all?

This is actually more likely to happen with TCP, although a desync that persists through an entire allocation is pretty fierce indeed. I guess you have a point that TCP enables you to detect it better, but I'm willing to make desync less detectable in exchange for just having less of it. At least this is a valid concern.

"

Daefecator wrote:

What is the point of having significantly more complex protocol, that will cause at least a) synchronization issues, and b) UDP packet loss (totally unpredictable)?

Ask YouTube why they use it (in conjuction with TCP for control). My answer wouldn't be much different.

"

Daefecator wrote:

As for the whole map being sent from the server to client at load time - well, it's nothing new. It's not like "OMG, the sky is falling" situation as you pretend it to be via use of multiple facepalm memes, and acting like you would have done it at least 10x times better. At most you could suggest using another eight-wheeled cart to solve possible maphacking. Sure, let's fantasize together - basically one will need to implement torrent-like downloadable area map, so it will get assembled like a puzzle from pieces dynamically, and you receive only those pieces that you see. Oh wait, you also need the adjacent pieces as well. Oh, you just used quicksilver flask, and now you move far too quickly in order to receive all the puzzles you need to display, so ok, let's send you three puzzles ahead in the direction you are heading to, and now, oh no!, you changed the movement direction quickly while quicksilver is on you, and now you are moving at a black non-rendered area... Ah yeah, here is the puzzle you need to render and bam!, you standing right in the wall or even out of map space entirely!

When your netcode design assumes a consistently high level of desync and allows trust violations deliberately to accommodate it, things are very bad indeed. I would never lower my standards that much; if that means more non-rendered wall of desync during beta, then at least hardcore players know when to log out, rather than allowing botters a foothold in the economy.

"

Daefecator wrote:

In order to maphack anything, one has to decipher an encryption GGG is now using (if any at all, as it is still beta), and even after someone succeeds at doing that, changing an encryption algorithm is like a several hours of work instead of several weeks developing and testing all this puzzled map.

Encryption does essentially nothing if the hacker is sitting at the client computer; it only protects against man-in-the-middle attacks. Remember that, at some point during the login procedure, the client receives all keys needed to decrypt communications.

"

Daefecator wrote:

And for someone to then point a finger at themselves and say - here, look, it was me who originally suggested using both UDP and TCP (so I am actually the one to blame for the broken game, sorry guyz. I crapped again just like it was with life nodes, when I claimed that was totally my idea, GGG)

You got nothing on me in that you cannot quote me anywhere in this Thread I claimed anything to be MY idea, or that I am the first to say or do anything. Until then, please don't assume things.

Thank You.

"

Daefecator wrote:

In any case, this "proposal" looks like an attempt to treat developers like little kids pretending you know it all better. And for someone to then point a finger at themselves and say - here, look, it was me who originally suggested using both UDP and TCP (so I am actually the one to blame for the broken game, sorry guyz. I crapped again just like it was with life nodes, when I claimed that was totally my idea, GG GGG)

. . .

"

Daefecator wrote:

Do you think Path of Exile developers don't know this already? For me it's like some kindergarten guy advising how to better solve equations to a math professor, and trying to involve as many of his kindergarten friends as possible to strengthen his point of view. The article in question even contains spelling errors, how can it be treated seriously?

It's quite clear GGG doesn't exactly have the answer on how to fix the desync now do they? No. Therefore, regardless of what it sounds like to you or not, until it gets fixed, myself and other people have every right to help spark and refresh the minds in GGG's team to further address the desync issue by making Threads like mine. If GGG were such math professors as you declare, they would have by now at least came close to fixing the desync. They haven't, and therefore, they're just as stumped about it as everyone else right now.

Nowhere (quote me if you can) am I claiming this Thread is 100% accurate OR what other people have said is 100% accurate. The Thread was an attempt to spark interest, talk and possible solutions.

If you're so brilliant and want to take on everyone here vs. what everyone else has said thus far, then perhaps you should join GGG's team and help fix it! Or, send them an E-mail saying you have the answer to the fix. Apply yourself and your so-called experience if you think you can do so better than me and other people have done here so far.

I've applied myself in making this Thread to further help the desync issue along and have said all I am going to say. You on the other side have yet to apply yourself to offer a more solid solution since you know so much more than everyone else here.

Thank You for your feedback, even if I don't agree with all of it.

You are welcome, actually. Please don't act like I am picking You personally, because I actually have nothing on you in the first place. Thanks for starting this topic, and I was just sharing my negative expression to some of ideas, that not You offered.

You definitely have the point my posts were pure critics and held no constructive thoughts. I am sorry, but all my constructive efforts for now have gone to removing MF completely from the game :)

As for applying to GGG, first of all, I don't think I can ever be accepted simply because I don't live where they do, and this is as far as I know a main criterion. And second, I know they have great potential themselves. Hell, they invented and programmed all this wonderful game, and who am I to offer them something? No matter how clever or dumb am I (or anyone else aside from GGG) they know it all better. And they will fix this minor but unbearably annoying issue.

Offering a highly technical solution like this requires tons of intimate internal game engine knowledge that GGG won't expose, and rightly so. Usually you don't fix anything by completely removing that part and composing it from scratch. This would be ideal, but require too much time; so You actually need to know how it works to apply some band-aids to try and see if it helps.

Actually, I think this is still too vague and not worth saying at all, but I think we are really better off not suggesting anything tech-savvy - because of the bold part a bit upper in this post.

If you really want to know my opinion on the subject without having any internal game engine info - I'd probably go with the solution similar to what I saw in World of Warcraft Burning Crusade engine (I have no idea whether it was changed or not after that point) - treat client state a lot more. All game combat and monster positions are calculated exclusively by client, and at the time monster has 1 hp left (effectively being killed) it "freezes" and actually dies only after server registers its death and handles loot drops. This will effectively remove desyncs from solo play, group play definitely needs some further brainstorming, but at the expense of two small details - first, there can be a little delay between monster deaths and loot drops, and second - it opens up possibility to cheat monster kill speed. But as server would actually register each monster death, assuming server is aware of monster health that is about to be killed and average player's sheet (not shit!) dps, any abnormally fast monster killings could easily be logged (and even banned) automatically with no manual efforts.

P.s.: I want to explain, why I even bothered writing in this topic. Earlier in Closed Beta there was a dupe bug with caustic flasks and I cheated that like mad. Then GGG made an update that fixed the dupe bug as they stated. I tried that, but it was still working! And no, this doesn't mean GGG are silly, they just added logging to the game client, and next day the bug was fixed for real. Second - why do I care? Because GGG didn't ban me, like they usually do it in other games, but Chris himself humbly asked me personally not to cheat anymore and tell him about glitches directly instead. This is a very cool move of the game team - these guys just do it right. I have very high hopes for GGG and know they will fix these fucking desyncs that plague us all at the moment. But I would rather give them time and support them silently, as <read bolded part again please>.

P.p.s.: OMG, Scrotie, totally missed your last post. Well, you do have the point. But only up to a point. You state you have high standards, it's awesome, man! But you may have either misunderstood my question or just cleverly quoted the ambiguous part without proper context at which point it makes me think whether you are understanding what you are suggesting yourself or relying on YouTube as end-all-be-all information source. The example situation was

"

Daefecator wrote:

Client gets UDP message with server time 995 stating that some white item drop.
Client gets UDP message with server time 990 stating that Kaom's Heart dropped.
Client gets TCP message with server time 1000 stating that Kaom's Heart was picked up by someone.

The exact question I asked was

"

Daefecator wrote:

What should it [the server] do at time 991? You just suggested it to include the data, that was not generated yet (packet 995) to be sent as a single packet.

I am talking about the server have already sent three packets, and client received them in (995 UDP -> 990 UDP -> 1000 TCP) order. And you are trying to somehow join the independent packets that were already sent in the past? Are you sure you are not talking about implementing something like Amazon Yesterday Shipping?

But nevertheless, don't give up on the efforts, as it will show GGG that we care!