Technical solution to eliminate desync in single-player sessions

"

HellGauss wrote:

Loss of precison/overflow is an issue of ALL kind of computer calculation methods. It has nothing to do with determinism. ALU operations are detemrinistic and standard.

Yes, but only locally.

HAL drivers, Thunking, Different CPU architectures, which version of C++ Boost they used, Which OS and patch level it was built on, difference between CPU's that support SSE4 and its extensions and those that don't.
All of these things have an effect on deterministic functions that are based on floating point.

For extra reading google for this term "things computer programmers should know about floating point numbers". A good starting point is Jeff Atwood's 'Why computers suck at math'.

There's also a well studied question on Stack Overflow: here

A math intensive but fascinating read is this PDF.

I hope this is of interest to you!

"

qwave wrote:

I had some great thoughts today that I could use some community feedback on. So my original proposal primarily target combat, right?

What if the proposal JUST was designed to solve pathing/positions of things? Desync truthfully has the most devastating impact on movement. Although some combat skills affect your position, I believe that the game client could use determinism for movement. This should most of alleviate the floating point issues, and also solve the core problems with desync (getting completely out of position). This would also solve all the 'hackz' problems that people have been stating, because it would only keep your movement/mob movement synchronized rather than deterministically handle combat as well.

Also, this solution would be 100x simpler than the proposal which involves the entire combat system. If GGG could just move deterministic pathing to the client-side, that may be all that's needed.

This solution could also be applied to multiplayer in a much more manageable way. I don't know why I didn't think of this 90 pages ago. I also apologize in advance if someone else thought of it first and I didn't notice it being recommended.

Thoughts?

If everyone agrees with this, I will write up a more elaborate code sample that targets this solution.

I really REALLY like this idea! I think we're on to something really good with this!
I was myself thinking of something similar. I think it's a very good idea, well worth seriously looking more deeply into it.

"

qwave wrote:

I had some great thoughts today that I could use some community feedback on.
What if the proposal JUST was designed to solve pathing/positions of things? Desync truthfully has the most devastating impact on movement.

This

95% of the problem is not desynch of the entire combat simulation, rather it's desynch of your characters position. It is generally inconsequential if enemy #39 is out of position or if arrow 6 out of 20 hits its mark. Those small events typically do not change the overall state in the short term. Your position on the other hand, whether you're over here far far away from that pack of enemies or smack in the middle of them matters a great deal.

The two primary things that change that are clicking on an enemy whose position is out of synch or getting stunned. Both can be at least partially ameliorated by forcing syncs of character position far (far) more often. Even if the rest of the state is not updated as frequently.

Beyond that the first can be solved by making the enemy movement routines more predictable by the client. As an example, doors are apparently horribly confusing and the client and server often disagree on whether or not an enemy will walk through a door. Making that a clearer and more easily arrived at answer ie. "I have been aggravated and will chase them even if they go through a door" or "Holy crap that's a door those are scary and I'm not going through that". It doesn't matter which way you move things so long as a more consistent answer is arrived at by the client and server.

My suspicion is that doors essentially introduce a fudge factor to make anything on the other side appear farther away that it really is. This makes it so that giant packs of enemies don't aggro when getting a small glimpse of you through the angle of a doorway. But it also means that when moving away from a door you sometimes trip the code that makes then enemy think you've run very far away, so they return to their idle behavior. Updating the code so that it works as now for enemies in the idle state but doors are nothing more than a pathing concern for them in their active state would fix much of the issue.

The later one, stuns, is thornier and really boils down to a design decision. GGG decided they wanted to make an online ARPG with an authoritative server. Ok cool, no problem there. But they also decided they wanted position to matter and allow enemy actions to commonly and significantly effect that position. Actions that cannot be properly simulated on the client. And that is not so cool as it causes loads of problems. On the one hand I can applaud them trying to make the combat deeper with things like player stun. On the other hand it is largely incompatible with the realities of their technical architecture and inherent unreliability of the internet. It's probably too late to take stun out, but doing so would solve a massive amount of the desynch complaints.

"

Skogenik wrote:

"

HellGauss wrote:

Loss of precison/overflow is an issue of ALL kind of computer calculation methods. It has nothing to do with determinism. ALU operations are detemrinistic and standard.

Yes, but only locally.

HAL drivers, Thunking, Different CPU architectures, which version of C++ Boost they used, Which OS and patch level it was built on, difference between CPU's that support SSE4 and its extensions and those that don't.
All of these things have an effect on deterministic functions that are based on floating point.

For extra reading google for this term "things computer programmers should know about floating point numbers". A good starting point is Jeff Atwood's 'Why computers suck at math'.

There's also a well studied question on Stack Overflow: here

A math intensive but fascinating read is this PDF.

I hope this is of interest to you!

While I'm not saying you're wrong, all your sources discuss floating point calculations or conversion from floating point to integer, none of them discuss fixed point calculations as far as I can see.

"

qwave wrote:

I had some great thoughts today that I could use some community feedback on. So my original proposal primarily target combat, right?

What if the proposal JUST was designed to solve pathing/positions of things? Desync truthfully has the most devastating impact on movement. Although some combat skills affect your position, I believe that the game client could use determinism for movement. This should most of alleviate the floating point issues, and also solve the core problems with desync (getting completely out of position). This would also solve all the 'hackz' problems that people have been stating, because it would only keep your movement/mob movement synchronized rather than deterministically handle combat as well.

Also, this solution would be 100x simpler than the proposal which involves the entire combat system. If GGG could just move deterministic pathing to the client-side, that may be all that's needed. I apologize in advance if someone else thought of it first and I didn't notice it being recommended.

This solution could also be applied to multiplayer in a much more manageable way. I don't know why I didn't think of this 90 pages ago.

Thoughts?

If everyone agrees with this, I will write up a more elaborate code sample that targets this solution.

Scrotie had this idea some time in the thread, however there is a problem. Combat affects movement. If you are stunned on the server due to hight damage, while the client fails to predict this, your client and server movement simulations will diverge. Furthermore, as I said a few pages back, I suspect client already gets the seed for mob movement and AI, will test this a bit later.

"

Scrotie had this idea some time in the thread, however there is a problem. Combat affects movement. If you are stunned on the server due to hight damage, while the client fails to predict this, your client and server movement simulations will diverge. Furthermore, as I said a few pages back, I suspect client already gets the seed for mob movement and AI, will test this a bit later.

Yeah I remember Scrotie's post, though it was based on using UDP for mob movement instead of determinism, which would make the situation far worse.

Anyway, anything that effects movement can use this deterministic system because it doesn't involve nearly as much effort. For example, the client should easily be able to calculate knockback without having to introduce all the other combat algorithms.

If the client gets the seed for mob movement, then how can things get out of sync so badly? I'm trying to think whether mobs are still moving around while you are lagging/disconnected. I believe they do, which does indicate that the client simulates their movement.

I believe if they fix the movement synchronization, 90% of the main desync problems will be gone.

Ok, did the test, nope, client does not get mob AI seed. Mobs only move once confirmed by server.

"

LogoOnPoE wrote:

Regarding "very few rare games" - every game that allows you to view accurate replays, and that's most of RTS games these days, rely on fp determinism.

The fact that you (and HellGauss in another case) would blithely reference the deterministic behavior of turn-based RTS games as if it were relevant to the architecture of an ARPG like PoE is a dead giveaway that you're out of your depth in presuming to speak authoritatively about its engineering. Seriously, do you have any relevant industry experience at all with networked game simulator development? I'd expect even the lowest entry-level QA tech to understand the following basic distinctions:

In a turn-based RTS game, all user input across all simulators is synchronized in lock-step with each discrete turn cycle. Nothing is generated asynchronously, each step of the game is quantized into a single simultaneous turn by the server and reproduced identically on all clients. All network latency is subsumed within each turn cycle, eliminating any possibly of client desynchronization by shackling all simulators to worst-case network delays. As a result, a game session may be preserved and reproduced perfectly, simply by recording the sequence of user input and random number seeds used to generate the session.

With a game like PoE hosted on asynchronous networked simulators, user input occurs asynchronously and is propagated across simulators subject to unpredictable network delays. Each client simulator runs open-loop without supervision by the server, until its reported behavior diverges unacceptably out of sync with the servers'. The client's game state is then corrected to resynchronize it with the server whenever necessary. Due to the asynchronous nature of the simulation, it is not possible to deterministically reproduce a gaming session solely based on recorded user input and random number seeds.

Armchair speculations about the feasibility of retro-fitting a commercial ARPG like PoE into a strictly deterministic simulation are naive. An engineer who has not attempted this type of project before is signing up for a harsh, hands-on education in real-time failure analysis.

I love to read this thread and for the third time now, i want more ideas. Maybe you all come together with some exceptional idea together with @GGG, who knows :P. Or maybe GGG gets some idea and make that idea better and maybe it gets in there. I am really looking forward to where this can go potentially :).

"

RogueMage wrote:

"

LogoOnPoE wrote:

Regarding "very few rare games" - every game that allows you to view accurate replays, and that's most of RTS games these days, rely on fp determinism.

The fact that you (and HellGauss in another case) would blithely reference the deterministic behavior of turn-based RTS games as if it were relevant to the architecture of an ARPG like PoE is a dead giveaway that you're out of your depth in presuming to speak authoritatively about its engineering. Seriously, do you have any relevant industry experience at all with networked game simulator development? I'd expect even the lowest entry-level QA tech to understand the following basic distinctions:

In a turn-based RTS game, all user input across all simulators is synchronized in lock-step with each discrete turn cycle. Nothing is generated asynchronously, each step of the game is quantized into a single simultaneous turn by the server and reproduced identically on all clients. All network latency is subsumed within each turn cycle, eliminating any possibly of client desynchronization by shackling all simulators to worst-case network delays. As a result, a game session may be preserved and reproduced perfectly, simply by recording the sequence of user input and random number seeds used to generate the session.

With a game like PoE hosted on asynchronous networked simulators, user input occurs asynchronously and is propagated across simulators subject to unpredictable network delays. Each client simulator runs open-loop without supervision by the server, until its reported behavior diverges unacceptably out of sync with the servers'. The client's game state is then corrected to resynchronize it with the server whenever necessary. Due to the asynchronous nature of the simulation, it is not possible to deterministically reproduce a gaming session solely based on recorded user input and random number seeds.

Armchair speculations about the feasibility of retro-fitting a commercial ARPG like PoE into a strictly deterministic simulation are naive. An engineer who has not attempted this type of project before is signing up for a harsh, hands-on education in real-time failure analysis.

You still don't understand the proposed solution after my explanation some pages back. Did you read it? If you did, I give up trying. I guess you wont get it.