POE WILL BE THE NEW VANGUARD IF THEY DONT CHANGE SHIT UP

"

RogueMage wrote:

First, you're just speculating that some predictions are so divergent that it "instantly blows up". I haven't seen any cases where the client outright malfunctions and fails catastrophically.

It is very common in PoE. And easily reproducible. If your ping/latency is high enough and not constant, you can

1) Stand in one point, and cast /oos (syncronize)
2) Start to move for a just a second in a straight line
3) Use Leap Slam to jump to a far location

It can happen that the leap slam works on the client but you are still out of range on the server when the leap slam command is received. This happens even if in the map there are no monster and in SP. The server resyncronize the client only after some seconds....

...Or maybe NEVER.... http://www.pathofexile.com/forum/view-thread/786991. (I think the word catastrophically apply in this example).

"

HellGauss wrote:

It is very common in PoE. And easily reproducible. If your ping/latency is high enough and not constant, you can

1) Stand in one point, and cast /oos (syncronize)
2) Start to move for a just a second in a straight line
3) Use Leap Slam to jump to a far location

It can happen that the leap slam works on the client but you are still out of range on the server when the leap slam command is received.

I tried to replicate this report in The Eternal Laboratory, where there are no monsters or players. My ping is under 100ms, and none of the dozens of attempts I made failed. Perhaps this is more of a high-latency issue than desync?

As for it being "catastrophic", rubberbanding is certainly abrupt, but neither the client nor the server are actually malfunctioning, they're both just projecting the action based on the inputs they've received. What you've discovered is that long-range Leap Slams may provoke desync. Whether it's due to excessive latency or some other cause, it's a real hazard, and I'd be interested to know what the safe range is in practice.

"

MrMisterMissedHer wrote:

When I say "blow up", I'm referring to the magnitude of error acceleration, with error roughly and simply being client state minus server state. I've chosen server as the start there because it's authoritative. I'm not implying that the client crashes or something, this is in the context of mechanical behaviour and desync.

There is an enormous amount of evidence for hugely divergent behaviour not caused by floating point non-determinism, where the client, having to rely on the authoritative server for verifiable results, (pseudo-)integrates or predicts and it results in error.

It's only an "error" because you (and the server) retroactively choose to interpret it as an incorrect result by the client. In practice, the client has no reference to compare its results to, so it cannot perform the calculation: "client state minus server state". To do that, the client would need a continuous feedback reference signal from the server that would enable it to work in a closed-loop manner and more closely track the ongoing state of the server.

"

MrMisterMissedHer wrote:

The part that I care about though is this:
Position on client is integrated, or predicted as unconstrained, but is constrained on the server. Error undergoes sudden change in velocity. Highly divergent. Result? Explicit treatment of a player position projection case, GGG add resync on stun.

Here's the other side of the problem: the server has no reference signal from the client either, so in practice, the server is unable to determine exactly when the client diverges too far out of sync. By the time the server finds out, it's too late to do anything but force the client to abruptly resync. That's what makes it a worst-case fallback mechanism rather than a closed-loop feedback control system.

"

RogueMage wrote:

"

MrMisterMissedHer wrote:

When I say "blow up", I'm referring to the magnitude of error acceleration, with error roughly and simply being client state minus server state. I've chosen server as the start there because it's authoritative. I'm not implying that the client crashes or something, this is in the context of mechanical behaviour and desync.

There is an enormous amount of evidence for hugely divergent behaviour not caused by floating point non-determinism, where the client, having to rely on the authoritative server for verifiable results, (pseudo-)integrates or predicts and it results in error.

It's only an "error" because you (and the server) retroactively choose to interpret it as an incorrect result by the client. In practice, the client has no reference to compare its results to, so it cannot perform the calculation: "client state minus server state". To do that, the client would need a continuous feedback reference signal from the server that would enable it to work in a closed-loop manner and more closely track the ongoing state of the server.

Once again, your standard is perfection instead of common-sense. Feedback needn't be continuous to provide some degree of error-checking. For example, instead of sending "player attacks Monster #42," it sends "player attacks Monster #42 at coordinates x,y." If Monster #42 is at a position other than x,y, a /oos is implied.

That doesn't mean unlimited /oos will ever be feasible. For example, a malicious user could pose as a client and flood the server with fake positions. As such, the server needs rules which cause it to respond to some of these requests, but not all of them.

As commonsense as this example is, AFAIK GGG hasn't implemented anything like it. I'm guessing this is due to the security concern mentioned; the /oos text-command is governed by a simple cooldown, an example of ridiculously simple yet rather effective security. Something like this would have to be (at least partially) liberated from cooldown to have a real impact, which would mean taking the time to address security concerns.

But I digress. My point is: you don't need perfect systems, you need effective, intelligent systems which smoothly handle the vast majority of cases. It's still okay to have /oos be used, assuming it's a sufficiently rare occurrence.

"

ScrotieMcB wrote:

Once again, your standard is perfection instead of common-sense. Feedback needn't be continuous to provide some degree of error-checking. For example, instead of sending "player attacks Monster #42," it sends "player attacks Monster #42 at coordinates x,y." If Monster #42 is at a position other than x,y, a /oos is implied.

It's open-loop systems that require precise calibration to remain in sync for any length of time, and that's why they're not very reliable. Closed-loop systems don't require anything approaching perfection, they just need enough loop gain to amplify discrepancies till they're large enough to be detected and corrected. Also, active feedback control is NOT tantamount to a continuous stream of /oos commands, that would be crude and terribly inefficient. An /oos is not a form of feedback; it's a hard reset that overwrites the client state with a copy of the server's.

"

ScrotieMcB wrote:

My point is: you don't need perfect systems, you need effective, intelligent systems which smoothly handle the vast majority of cases. It's still okay to have /oos be used, assuming it's a sufficiently rare occurrence.

Exactly. An /oos resync is a drastic fallback mechanism which should be available only as a last resort (such as high latency or packet loss).

"

RogueMage wrote:

It's open-loop systems that require precise calibration to remain in sync for any length of time, and that's why they're not very reliable. Closed-loop systems don't require anything approaching perfection, they just need enough loop gain to amplify discrepancies till they're large enough to be detected and corrected.

After looking up open-loop systems on Wikipedia, I'd say no one here is suggesting a fully open-loop system. I've never said the suggestions I've presented in this thread would eradicate all desync (although I have claimed the vast majority of it), or the server would never have to resync clients, or we could do away with the /oos command — and all of these are forms of feedback. (Yes, /oos is feedback, although I agree crude and inefficient in terms of routine use.)

The only reasons I can see to go open loop are: going closed loop breaks the budget, or it breaks the laws of nature. In this case, the reason is the latter; this applies to a specific situation, specifically the time period before the client could receive feedback from the server. Data can only travel so fast; however, it does travel fast enough, which is why I'm suggesting only partial implementation of open-loop methods.

Put another way — immediately after the play inputs data, the client only has two choices: either begin displaying results using open-loop techniques, or do nothing but wait. Physics prevents feedback for this brief period of time; closed-loop is temporarily impossible.

In terms of making that choice as developers, this is what GGG has to say:

"

Chris wrote:

The server has to dictate whether things happen or not, but there's a 50-250ms delay before data gets to the server and back. There are three ways that games can solve this:

• Trust the client. This means people can cheat, but the results are instant. We will not do this.
• Wait until data arrives back from the server before doing anything. This is a very common strategy in RTS and MOBA games. If you click to move, the unit will only start moving once the server says so, which is 50-250ms later. If you are close to the server, you'll quickly get used to the lag and everything feels pretty good. If you're far away (New Zealand, for example), it feels like you're playing drunk. Every time you issue an order, nothing happens for quarter of a second. This does not work for Action RPGs.
• Start predicting the result of the action as though the server said yes, immediately. When the server later gets back to you with a result, factor it in. This is what Action RPGs including Path of Exile do. It means that when you click to move, or click to attack, it occurs instantly and feels great. The problem is what happens when the server decides that the action can't have occurred - that's when the game gets badly out of sync.

I've previously attacked the accuracy of saying "this does not work for Action RPGs," but with one rather minor edit — "this does not work as well" — then I'm inclined to agree, with no small amount of zeal. Responsiveness is a huge thing, especially if you intend to design your game with cliffhanger moments requiring mechanical skill to survive, such as Vaal's Slam and Dominus' Touch of God. (These are good things, by the way... assuming desync isn't plaguing your game.)

This all means going open-loop temporarily — by Chris' numbers, for 50-250ms. After that time period, which cannot be reduced by the laws of physics, the feedback starts coming in to correct errors.

If you look at what I'm suggesting, it's not really that radical. I just want the client to "start predicting the result of the" monsters' "action[s] as though the server said yes, immediately." In other words, instead of just the current-player responding immediately, I want the entire client simulation responding immediately (for single-player, anyway). This is only logical, because on the server end of things, player actions and monsters actions do occur simultaneously.

I'm really not advocating the client become "more open-loop." I'm suggesting a saner open-loop for the open-loop we already have.

"

RogueMage wrote:

I tried to replicate this report in The Eternal Laboratory, where there are no monsters or players. My ping is under 100ms, and none of the dozens of attempts I made failed. Perhaps this is more of a high-latency issue than desync?

Technically, it is an issue of variable latency (when there are no monsters). When there are monsters, it is also a desync issue (i.e. it can happens even if you have 0ping). I report that example just as an extreme. Everyone that use LeapSlam on a regular basis can confirm that it is very prone to teleports.

The point is that PoE is full of skills that fail on the server but succed on the client. This is the +1/-1 discontinuous behaviour described by MMMH.

"

ScrotieMcB wrote:

I'm really not advocating the client become "more open-loop." I'm suggesting a saner open-loop for the open-loop we already have.

That's the main problem of your suggestion: although I may agree that things will not get worse, you have not proven how much things will sync better with your proposal. You start with the hypothesis that there is a >0.1sec variable lag between commands on the client and the server, and a full update is possible only once in a while, together with a lot of unknowns related to rng (e.g. combat, which Rhys clearly states that is linked with movement, although in a different context*): this leads to desync as the current system. Giving the AI of monsters to the client is only a small piece of the puzzle.

It requires a massive recoding: if such an effort should be done, it is better to design a stronger system from scratch.

In conclusion, i think that:
1) With current architecture, desync is unavoidable.
2) If (when...) GGG will perform a massive recoding (i.e. PoE 2.0 or 3.0) i strongly suggest a deterministic approach. I really believe that with complex ARPGs it is the only way.
3) In the meanwhile, some heavy bugs like the one i showed in the video can be addressed, as well as maybe some very minor improvements.

*http://www.pathofexile.com/forum/view-thread/626664/filter-account-type/staff/page/2 last Rhys' post, movement and combat spoiler.

"

ScrotieMcB wrote:

"

RogueMage wrote:

It's open-loop systems that require precise calibration to remain in sync for any length of time, and that's why they're not very reliable. Closed-loop systems don't require anything approaching perfection, they just need enough loop gain to amplify discrepancies till they're large enough to be detected and corrected.

After looking up open-loop systems on Wikipedia, I'd say no one here is suggesting a fully open-loop system. I've never said the suggestions I've presented in this thread would eradicate all desync (although I have claimed the vast majority of it), or the server would never have to resync clients, or we could do away with the /oos command — and all of these are forms of feedback. (Yes, /oos is feedback, although I agree crude and inefficient in terms of routine use.)

You're using "feedback" in layman's terms, i.e. any sort of response to an action that is conveyed back to the actor. That's a definition broad enough to include this online forum, as a form of user "feedback" to GGG. That's fine, but it's not what I'm referring by a "closed-loop feedback control system", which is an engineering term that describes a specific type of continuously self-regulating dynamical system.

A stand-alone single-player game simulation like Diablo II is an open-loop dynamical system that responds to user input and internally generated events with actions that the player perceives. When a D2 client is networked to a remote server, it works like PoE does - the client conveys user input to the server and responds to external events generated by the server. The networked client still operates as an open-loop system - it proceeds independently with its simulation based on the inputs it receives. The server likewise proceeds with its own simulation based on the same inputs, and the two simulations turn out very similar, since they're both running on copies of the same game engine.

If client-server communication was instantaneous and the hardware and runtime environments were identical, client and server would produce the same simulated results and the two simulations would run in parallel synchronicity for a long time. In practice, network latencies and runtime differences produce discrepancies that cause client and server simulations to drift apart over time. When this becomes excessive, the server steps in and forces the client to abandon its version of the simulation and mirror the server's version, i.e. a hard resync of client to server.

While the server does monitor the client's behavior, it's not a closed-loop feedback control system. The server has some means of determining when the client drifts too far out of sync, but it doesn't use that information to continuously regulate its own behavior. Instead, the server just imposes its own version of the simulation on the client at sporadic intervals. This is a forced-sync fallback mechanism rather than a feedback control system. The player on the client system perceives it as an abrupt discontinuity in gameplay, i.e. rubberbanding.

A closed-loop client-server system would work in a fundamentally different manner. The client would continue to proceed with its own local simulation, but in addition, it would receive a continuous stream of reference data reflecting the ongoing state of the server's simulation. The client would then compare its results with the server's to generate error correction inputs that it would then use to self-regulate its own behavior. In a well-designed closed-loop system, this error correction occurs continuously rather than sporadically, and enables the client to seamlessly track the server's simulation without abrupt discontinuities in gameplay.

You might suppose that unpredictable network latencies would cause a closed-loop system to malfunction, but time delay compensation is actually a standard part of feedback control systems analysis. There's a rich vein of applied mathematics that describes exactly how such systems function in numerical detail, which design engineers use to stabilize the system's real-time behavior. The fact that server and client simulations are offset in time is explicitly designed into the system. The client uses the delayed reference data from the server to predict the server's current results, and acts to reduce any discrepancies between the client's current state and the server's projected state. Since the prediction extends only as far as the current network latency, errors do not accumulate and the client is able to closely track the server over time. This prevents the client from drifting apart from the server, eliminating the need for sporadic discontinuous resyncs.

Catacombs lost rubbish near chests in places where spiders spawn?

Good!