Technical solution to eliminate desync in single-player sessions

How far are we now? Any realistic way to keep the double check system (yes i like it), reduce traffic and still don't open the game to botting/hacking/duping?

thats pretty easy, make doors bigger, tree far away from each other. make game working like quartz flask, and /ooos every 0,1sec autocast

"

SkyCore wrote:

end goals of determinism are far more elusive, costly, and impractical. Fundamentally, the complexities of time itself excludes perfect determinism across any amount of space which is subject to speed limits. Perhaps for PoE 2... i could imagine a much better system than the one we currently have, but of course i do not control GGG. And do not have the resources to pursue such an endeavour as an individual without connections.

Im not talking determinism, Im talking about UDP streaming. UDP streaming doesn't make sense in this game because some mechanics decision mechanics are client side (like hitting versus not hitting, critting versus not critting) and you cant visually stream those choices, and there also isn't any halfway.

Sure streaming can help with things like movement, but "correctness" is about the state in general, and streaming correct positions when your client has critted (and they shouldn't have critted) is just as wrong as monsters being in the correct position, and it will still force a desync

"

deteego wrote:

UDP streaming doesn't make sense in this game because some mechanics decision mechanics are client side (like hitting versus not hitting, critting versus not critting) and you cant visually stream those choices, and there also isn't any halfway.

Sure streaming can help with things like movement, but "correctness" is about the state in general, and streaming correct positions when your client has critted (and they shouldn't have critted) is just as wrong as monsters being in the correct position, and it will still force a desync

Im not suggesting that udp streaming magically removes desynching. All im saying is that it corrects desynch at regular intervals just as /oos does. Its not perfect. But its easy to imp and its an improvement.

"

deteego wrote:

Im not talking determinism, Im talking about UDP streaming.

and my determinism paragraph was not aimed at you.

"

RogueMage wrote:

"

ScrotieMcB wrote:

"

RogueMage wrote:

You need a fallback mechanism to handle cases where the UDP stream is interrupted or the client diverges too far from the server to correct.

No, you wouldn't. Stream interrupted? Restart stream. Problem solved.

This is a minor point, but I want to make sure it's clear that the UDP update stream is not designed or intended to replace any existing mechanism of the current networked open-loop simulator system.

How could it not be? The UDP packets would be sending up-to-date monster position data. All a TCP resync really is is packets of up-to-date monster position data. I guess the intent isn't all that relevant, but in practice the UDP stream makes TCP resync obsolete (assuming we have the bandwidth to stream UDP; TCP's advantage is in selective, rather than continuous, resynchs).

"

RogueMage wrote:

"

ScrotieMcB wrote:

...it isn't desync prevention because the UDP information is always at least a round-trip-ping time late.

"

RogueMage wrote:

No it's not a round-trip

Yes, it is. Remember that the client simulation as a whole is a one-way-ping ahead of the server; thus, whenever the client simulation depends on the server to send it something, it's a round-trip ping behind.

Two misconceptions here:
1. In the current open-loop system, no particular simulator is a "one-way-ping" ahead of the others, they are all proceeding in parallel.

This is either impossible or unacceptable (unsure which). The whole point of the system is to have the client simulation respond immediately to player commands; as such, it must necessarily be a half-ping ahead of the server. (The term "one-way-ping" is kind of a contradiction in terms, so I'm using the term "half-ping" from here on.) The server cannot act on unpredictable information which it does not have; the client's commands are unpredictable. Therefore, what you are proposing here is either impossible (server responding to commands it can't possibly have yet), or deliberately sluggish response to player commands (client waiting until server has data before acting on player commands).

"

RogueMage wrote:

2. In no situation will the client ever be required to depend on the UDP update stream before it can proceed with its simulation.

If it's not needed, why have it? The only reasonable answer to that question is "because it's a backup," which would make it a resync method, not a prediction method.

The real concept that you're not getting is that user commands are both unpredictable and vitally important. Your entire predictive system must be prepared to turn on a dime in response to player commands, ensuring responsive gameplay rather than creating a system where player reactions do not properly alter the flow of events. This is the critical flaw in your system; it attempts to predict gamestates further than a half-ping in advance, which is impossible while giving user input the credit it deserves.

I strongly recommend a partially deterministic predictive system on the client for predicting monster AI/skill use, allowing it to start animations in sync with the client's input rather than waiting on the server. And even though I'm the original guy to suggest a UDP resync stream, talking with the devs has made me realize that continuous resync isn't necessarily the best idea, and that selective TCP resync — in combination with strong client prediction — may be the best route.

"

deteego wrote:

Im not talking determinism, Im talking about UDP streaming. UDP streaming doesn't make sense in this game because some mechanics decision mechanics are client side (like hitting versus not hitting, critting versus not critting) and you cant visually stream those choices, and there also isn't any halfway.

That's not my understanding of how combat resolution works in PoE. The server receives movement and attack events from the clients, and incorporates them into its simulation. The server then resolves all combat and sends the results back to the clients. The clients don't determine how much damage a hit does, whether it crits or not, or whether a target dies. The server makes all these decisions and the clients incorporate them into their simulations.

As a result, there is no need for a UDP update stream to incorporate combat resolution into its positional updates, since these events are already transmitted via the game's TCP communication systems. What the UDP stream provides is a series of server positional updates that the client can incorporate to reduce positional discrepancies between its simulation and the server's. With a client that can more closely track the server's simulation, the player's view of the battlefield will more accurately reflect the tactical situation seen by the server. The player's actions and expectations will then more closely align with the server's actual combat results.

"

ScrotieMcB wrote:

The real concept that you're not getting is that user commands are both unpredictable and vitally important. Your entire predictive system need to be prepared to turn on a dime in response to player commands, ensuring responsive gameplay rather than creating a system where player reactions do not properly alter the flow of events. This is the critical flaw in your system; it attempts to predict gamestates further than a half-ping in advance, which is impossible while giving user input the credit it deserves.

There is no need for a UDP update stream to reinvent any wheels by incorporating these high-level considerations into its low-level operation. The client and server already use all of these predictive and reactive mechanisms in its current open-loop implementation. What you don't seem to accept is the closed-loop engineering concepts that add an additional feedback control loop, not to replace or obsolete any existing gameplay mechanism, but solely to enable the client to more closely track the server.

In practice, the existing open-loop client simulators are predicting far more than "a half-ping in advance", they are attempting to mirror the server's simulation well enough to operate independently for many dozens of seconds between each resync. There's nothing flawed about that - it's inevitable that open-loop simulators will eventually drift apart. That's why closed-loop tracking mechanisms are used in industrial systems to correct accumulated discrepancies rather than relying solely on high-precision, deterministic predictions.

The problem with the current open-loop client simulation is not that it produces inaccurate results - the client's results are just as valid a simulation as the server's. The bottom-line problem is that when discrepancies arise between client and server, the server's version must overrule the client's. Adding a closed-loop UDP positional update stream would change none of this - it would only give the client a means to more closely track the server, and thus reduce the accumulation of discrepancies before the need for a client resync arises.

"

ScrotieMcB wrote:

...talking with the devs has made me realize that continuous resync isn't necessarily the best idea, and that selective TCP resync — in combination with strong client prediction — may be the best route.

I hope the explanation above makes it clear that the UDP update stream is not used as a "continuous resync" system. It is instead used to enable the client to reduce the need for resyncs by more closely tracking the server's reported positional data.

"

ScrotieMcB wrote:

If it's not needed, why have it? The only reasonable answer to that question is "because it's a backup," which would make it a resync method, not a prediction method.

In making this kind of claim, you are arguing from a semantic position rather than an engineering analysis. When making distinctions between "desync fallback mechanisms" and "positional discrepancy reduction" I'm describing specific programming structures and systems rather than classifying the game's functional behavior in terms of how it appears to the user. Things that can appear to be a type of "resync" to a user may in fact operate in a completely different manner under the hood.

"

deteego wrote:

"

gonzaw wrote:

@GGG:

When this game was in Closed and/or Open Beta, have you ever thought about compromising on the mechanic aspects (described by you guys and Chris in the Developer Manifesto), to be able to handle desync?

Basically, did you ever think about removing Stun, or Chance to Hit, etc, from the game because of desync? Or was it always a big NO to do so, even with the flood of complaints from players about desync?

Thanks.

If they were going to do any kind of massive rewrite, it would be far smarter for GGG to just give up the client/server syncing model they have now.

Changing mechanics will break everything, all the way down to items, and there aren't any other practical suggestions being given here (like that UDP streaming change, which won't work simply because binary mechanics, such as hitting versus non hitting, are done client side, and there isn't a nice way to visually revert such changes as explained before)

Do what every other proper game does, which is have a lot of your logic server side, and combine UDP with TCP (or UDP with a reliable layer ontop). Then you would have zero desync at the compromise of slightly more delay (which in most cases is negligible assuming your ping < 50)

Well, yeah I'm asking if they wondered about giving up the client/server syncing model or rewriting it, as well as compromising on the mechanics that prevent them from doing so, while they were in Closed/Open Beta, which would be a "more" ideal time to do such changes. It might not be the ideal time to do so, but would have had the least impact, since well...you are still in Beta and at worst you "could" revert back to the original model before release.

Yes, I presume they would change the syncing model to use UDP and stuff similar to what you guys are stating. But I think they already know about this, and already know what they would have to do if they removed those specific mechanics of the game.

"

RogueMage wrote:

"

deteego wrote:

Im not talking determinism, Im talking about UDP streaming. UDP streaming doesn't make sense in this game because some mechanics decision mechanics are client side (like hitting versus not hitting, critting versus not critting) and you cant visually stream those choices, and there also isn't any halfway.

That's not my understanding of how combat resolution works in PoE. The server receives movement and attack events from the clients, and incorporates them into its simulation. The server then resolves all combat and sends the results back to the clients. The clients don't determine how much damage a hit does, whether it crits or not, or whether a target dies. The server makes all these decisions and the clients incorporate them into their simulations.

This is correct.

"

RogueMage wrote:

As a result, there is no need for a UDP update stream to incorporate combat resolution into its positional updates, since these events are already transmitted via the game's TCP communication systems. What the UDP stream provides is a series of server positional updates that the client can incorporate to reduce positional discrepancies between its simulation and the server's. With a client that can more closely track the server's simulation, the player's view of the battlefield will more accurately reflect the tactical situation seen by the server. The player's actions and expectations will then more closely align with the server's actual combat results.

Incorrect. UDP is best used on ephemeral data, that which changes rapidly; thus, the question is "what is ephemeral?" We can agree that monster movement is ephemeral, but is monster life ephemeral? Are stuns and freezes ephemeral? You could make the argument that current monster life, stuns (with ending-time in terms of initial hit) and freezes (same ending-time designation) should be sent via UDP, because, like movement, a TCP packet that arrives after the stun or freeze is useless and does not require rerouting to destination. Note that, under this implementation, the server wouldn't be sending damage numbers to the client at all, because damage isn't part of the gamestate — however, current life and status effects (if any) of the various actors is part of the gamestate, so it would send that instead. TCP would be reserved for more permanent stats, like picking up items and items stats (once identified).

The reason your statements are incorrect, instead of merely debatable, is that the reasoning behind using UDP is bandwidth (and header) efficiency — the idea is to be more efficient with bandwidth on GGG's end to allow more continuous transmission of data from server to client. This falls apart if the servers are continuously transmitting both UDP data and TCP data — instead of gaining header efficiency, you're losing it by splitting your data up amongst multiple packets where one TCP packet used to do the job instead, using more bandwidth to communicate the same data. Stuns, freezes, and damage calculations are common occurrences, and it would be completely unfeasible to simultaneously rely on both UDP and TCP for common occurrences; you'd have to choose one or the other to gain efficiency.

Thus, in order for a UDP system to be an efficiency advantage, you need to find ways to make UDP essentially the sole server-to-client protocol for ephemeral combat data, to include the effects of damage (adjustments to life totals, stuns, freezes, shocks, etc).

This also serves as an answer to your post directly addressed to me. Yes, you need to reinvent the wheel, and yes, UDP would have to replace TCP for all communication of ephemeral data, otherwise it's simply not worth it. In order for a hybrid UDP/TCP archetecture to work, TCP needs to be used very sparingly to gain efficiency.

Maybe this is what random programmers are talking about when they say that UDP and TCP don't mix. It's not really that they can't mix, is that you need to really optimize your netcode to play to the advantages of each in order to make it worthwhile, and that you're likely to just gum things up if you're haphazard about it.

Can anyone familiar with Wireshark (or your preferred network analyzer) do something for me?

First however:

Background and definitions

Just to be ultra clear, when I say Server I mean the Path of Exile server, and the Client is the local executable.

I explicitly filtered Wireshark to ignore other traffic using this filter:
"ip.dst eq 37.58.0.0/16 or ip.src eq 37.58.0.0/16"

The 0.0/16 configures WS to capture anything from that 37.58 subnet.

I obtained the destination IP by using Process Explorer, viewing the PathOfExile process and then opening the TCP/IP tab; the Subnet did jump around as I left instances however.

I captured 45 minutes of network data and using the above filter gave me 22,715 packets, with an overall packet size of around 1.5Mb.

The data was collected whilst I was playing through the Ship Graveyard level, spawning zombies and looting; basically trying to play as normally as possible, I also performed an /oos at one point.

Findings

When I captured traffic between client and server, I noticed a lot of corrupted packets originating from the server.

To break down the numbers a bit:
Over the period the server sent me 9980 packets, with an average size of 76 bytes.
The client sent 12735 packets with an average of 56bytes length.

Now I noticed that many of the packets showed up in Red and black, indicating an error in the packet, so I added this new filter:
"ip.src eq 37.58.0.0/16 and eth.fcs_bad"

It filtered 6094 our of 9980 total and all because they had an invalid frame check-sum.

So there were only 3886 good packets sent from the server in the period.
In other words, 61% of the packets from the server were bad.

Thoughts

It's notable that everyone of these bad packets was a TCP/IP ACK and I was receiving 3 of these a second, intermingled with a total of 20 packets per second of everything else.

Now as I've mentioned before, I've been pretty lucky with Desync issues, and have rarely had them effect me noticeably. However, could it be that some peoples network hardware, does have a problem and could be choking on all these bad packets ?

There are also a couple of differences, all of the Client ACK' have no byte data and no padding.

The Server' ACK either have a Byte of data (ranging from 01 to FF) and a 5 byte padding;
Or no data, and a 6 byte padding; and there are 764 of those in my sample.

After said padding, each of these packets has a 4-byte Frame Check sequence check-sum.

So back to my original question, I'd be interested to see what results other people are getting. So can someone duplicate my findings? I'm based in the UK so if someone in a different region could test this, it would probably be quite helpful for GGG to know about.

Lastly, If there are any more statistics from this dump that I can give, please let me know and I'll do what I can.