Technical solution to eliminate desync in single-player sessions

"
qwave wrote:
"
Even if all state variables are regenerated deterministically, asynchronous open-loop simulations will eventually drift out of sync with each other.


The snapshots are being streamed to the server in realtime. Please explain how a deterministic simulation can drift out of sync? By calling it deterministic, it cannot drift out of sync. I would love for you to tell me how something deterministic can also be non-deterministic at the same time. Maybe you have discovered some new laws in physics that im not aware of.


No one owes you an explanation. I think GGG has been more than forthcoming explaining things to you, but this forum isn't a subscription to a free course in game design.
"
No one owes you an explanation. I think GGG has been more than forthcoming explaining things to you, but this forum isn't a subscription to a free course in game design.


My reply wasn't directed towards GGG, it was directed towards RogueMage. Trust me, I do not need a course in game design. ;-)



"
No movement in physics is deterministic. All movements are inheretily non-deterministic, but the probabilities can in many cases be so high that we can for all practical purposes treat those cases as if they were deterministic.


In a video game, mob movement based on a deterministic seed is deterministic. That's all that matters for this particular proposal.
Last edited by qwave#5074 on Nov 19, 2013, 11:57:11 PM
So many nerds in this thread.

http://www.youtube.com/watch?v=tNK5Ybr18xI&t=0m7s
"
qwave wrote:
You don't need this much precision for the combat calculations, that's what im saying. Reduce your precision and you shouldn't have this problem.

If you mean "switch to integer-only math everywhere" then yeah, that's what I said in my first post. But there is lots of combat stuff that relies on angles and sin/cos etc. If you don't mean integer-only math, then no, that suggestion will not work, because even "simple" math can create devilish number pairs like the ones I posted. For example: 7 / 3 results in a number not able to be represented exactly by the IEEE floating point format. If that number is later multiplied by 3, the results is then NOT 7, but 7.0001 or 6.9999. And a similar case can be trivially made for numbers falling either side of the .5 boundary.

The issue of death might be fine, then. Changing gear while in combat (as racers do) will feel horrible, though, even if it doesn't cause desync.

I strongly suspect that Roguemage is correct. The system is absolutely dependent on two massive simulations running in parallel that need to be exactly the same. Small, inevitable errors from floating-point calculations will cause desync. I am very skeptical on creating a system that relies so heavily on determinism when floating-point arithmetic is inherently non-deterministic.

And when desync does occur, how do you fix it? The server has to essentially run the simulation into the future, then send that entire world-state to the client. It has to be far enough ahead so that by the time the client receives and reconfigures to it, the following client states are valid for the server. In other words, resync causes several seconds of lag on the client (depends on latency).

The resync delay will also depend on the leniency of the server for late packets. You don't want to force a resync (or disconnect) every time a packet arrives a second late. We do have to support less-than-perfect connections.

Currently, when Path of Exile resyncs something, the client sees something(s) pop to the correct location, but client actions are still being processed (so, you can "fight" the desync by using flasks, casting spells, etc.) Under the proposed system, resync means that last 3-4 seconds of time are "erased", as your character is retroactively forced to do nothing for that time, making you much more likely to die when this happens in combat.

Also, the term "snapshot" is not really accurate as used in this discussion. "State hash" is better.
Code warrior
"
qwave wrote:
Please explain how a deterministic simulation can drift out of sync? By calling it deterministic, it cannot drift out of sync. I would love for you to tell me how something deterministic can also be non-deterministic at the same time.

You appear to presume that a "deterministic simulation" can be fashioned by the wave of a software programmer's hand. In practice, producing precisely deterministic results in a system of open-loop, asynchronous simulators is an extremely difficult challenge to either achieve or verify.

Even in a LAN where data transmission is reliable, communication is not instantaneous and hardware does not run at precisely the same synchronized clock rate. Over time, asynchronous simulators will drift out of sync, even when sharing the same time-stamped input streams. With complex real-time simulations, small timing discrepancies can unpredictably mushroom into perceptibly distinguishable results. When (not "if") that happens, one set of results must be chosen to supercede the others, and GGG's choice was to rely on the server's.
"
inDef wrote:
So many nerds in this thread.

http://www.youtube.com/watch?v=tNK5Ybr18xI&t=0m7s


Matt is a Gym and Christianity nerd, fyi.
This message was delivered by GGG defence force.
"
If you mean "switch to integer-only math everywhere" then yeah, that's what I said in my first post. But there is lots of combat stuff that relies on angles and sin/cos etc. If you don't mean integer-only math, then no, that suggestion will not work, because even "simple" math can create devilish number pairs like the ones I posted. For example: 7 / 3 results in a number not able to be represented exactly by the IEEE floating point format. If that number is later multiplied by 3, the results is then NOT 7, but 7.0001 or 6.9999. And a similar case can be trivially made for numbers falling either side of the .5 boundary.


There are thousands of game development articles on how to handle this. Gas Powered Games and ID Software have both dealt with it, and they are building cross platform games. You don't have to worry about consoles, so your job has been made much easier. This sort of problem is worth solving, and I think you can also take a lot of shortcuts. It's like you need the physics simulations to be deterministic. We're talking about basic combat here, not deterministic cloth and ragdoll physics.



"
Changing gear while in combat (as racers do) will feel horrible, though, even if it doesn't cause desync.


Again, you will only desync if you are allowed to craft equipped gear. Even still, as long as the client sends a 'state hash' to the server during the request to craft/equip/unequip, the server can reply with the current state without a 'desync'.


"
The system is absolutely dependent on two massive simulations running in parallel that need to be exactly the same.


The server simulation would not be 'running' in parallel. Each time it receives a state hash, it would validate the simulation and save it. There is no need for it to run the simulation concurrently because the client is doing that job.

If the server is running the simulation concurrently, then the client's snapshot should always overwrite the server's state as long as the server validates the authenticity. Since the server is continually overwriting its own state, they should not encounter conflicts because they are not running in parallel.



"
And when desync does occur, how do you fix it? The server has to essentially run the simulation into the future, then send that entire world-state to the client.


In the event of desync, the server would begin running the simulation using the last state hash, which should not be older than 1-2 seconds. Actually, according to your understanding, the server would be running the simulation in parallel. If this is the case, then the server would just send the client the current state hash to resync. However, my original proposal had the server doing nothing until it received a state hash, at which point it validated/updated it in storage.


"
The resync delay will also depend on the leniency of the server for late packets. You don't want to force a resync (or disconnect) every time a packet arrives a second late. We do have to support less-than-perfect connections.


If the client latency is exceeding the leniency, then the session should revert to the current synchronization system where the server is fully authoritative.


"
Under the proposed system, resync means that last 3-4 seconds of time are "erased", as your character is retroactively forced to do nothing for that time, making you much more likely to die when this happens in combat.


I would not recommend a 3-4 second leniency. I was thinking more like 1-2 seconds at best. Besides, desync already has the tendency to warp you after 3-4 seconds. My primary goal is to significantly reduce the likelyhood of desync occurring in the first place, particularly with fast internet connections.
Last edited by qwave#5074 on Nov 20, 2013, 12:30:34 AM
"
In practice, producing precisely deterministic results in a system of open-loop, asynchronous simulators is an extremely difficult challenge to either achieve or verify.


Again, this is why the snapshot state is continually streamed to the server. Thousands of games have built network architectures on top of deterministic simulations. Heck, almost every RTS out there has done it due to the sheer number of units/calculations that are taking place. Tell all the major game developers out there that it isn't worth doing. Like ive said before, Gas Powered Games, Microsoft, Id Software, and many other companies are using this sort of networking even cross-platform from PC to consoles!
Actually Rhys, now that I think about it, you could just solve the floating point issue by providing leniency to the client on the precision. Just validate the state hash as acceptable if the math falls within a given range. Problem solved.
Last edited by qwave#5074 on Nov 20, 2013, 12:36:22 AM
No offense qwave, but all of the stuff you are saying is high sky theory which does't work in practice, and in a lot of cases its apparent you don't completely understand what you are saying.

Its practically impossible for the server to be completely deterministic in how the client calculates stuff (and more importantly, how the server perceives the client calculates stuff, things like packet loss may mean even if the client is correct, the server wont care)

And for your information, almost all RTS's are almost completely server side, at most the movement is client side. The amount of games that have 2 simliar states that have to be synced is actually incredibly rare in modern standards, mainly because internet has become so widespread and cheap, that getting low latency is a basic need, not a luxury nowadays.

Report Forum Post

Report Account:

Report Type

Additional Info