Technical solution to eliminate desync in single-player sessions

"

qwave wrote:

...I am by no means an expert on this matter...It is possible that I am wrong about the cause of the desync issues...

Qwave man, your passion for helping fix the problem is awesome, but you might want to take a few breaths and step back for a bit.

You had the right mindset going in, but as a software dev yourself you should know full well that if someone on the inside tells you (as someone on the outside) that your basic assumptions about how the software in question functions are flawed, you should know to accept that and go back to the drawing board.

It doesn't mean it was a bad idea, or that the principles you outlined are wrong, but if Mark says they won't work here, you've no choice but to accept that.

It doesn't mean you don't know what you're doing, or that you 'lost' an argument somehow, it's just a reality based on the fact that your analysis of how PoE functions is, by necessity, based on incomplete information.

I'm sure you wouldn't have taken the time to write all of this if you didn't love the game. Mark loves the game too, man. We all do... :)

MeltingPoint:

"

What is in this 'snapshot', what kind of information does it contain? A seed of some sort? A data structure containing the current game state of every object in the area?

The snapshot would contain the player's input since the last snapshot, and a cryptographic hash based on the deterministic seed and current timestamp. Basically it's the same stuff that the client is currently sending to the server.

"

How many of these snapshots will be taken and how often?

They need to be taken every second or so. Currently the game sends data way more frequently, so my proposal would cut down on the data strain a lot and reduce the amount of bandwidth required, due to compression.

"

Will the server be responsible for 'calculating' what happens between each snapshot, or just verify the the snapshot could happen.

The server will verify that the snapshot happened by comparing the deterministic seed to the timestamp/input. This proves the authenticity of the snapshot.

"

Given that fake snapshots could be added at anytime in the process, would each one need to be verified against the one before it, and before it, and before it... Given an hour or more in a single instance, possibly millions of snapshots, would it not be akin to solving every possible chess move after the first move?

No, they could validate each snapshot sequentially without the others in the set.

"

What about storage considerations? Not everyone, dare I say less then a few percent of people finish an area in under a few minutes, I personally enjoy clearing entire areas (bit of a loot junky), would this not affect the amount needed to be stored (I assume on the servers) while I play?

They already store all of this data on the server, so this wouldn't change at all.

Antiquarian: Mark_GGG admitted that he is not an authority/expert on this topic. His answers demonstrated a lack of understanding on deterministic seeds and their role in synchronizing. I would take a step back if he provided a shred of technical evidence on why a deterministic seed is unfeasible.

The only way this is going to get fixed is if the community aggressively pursues a solution with technical proof, white papers, and citations to back it up. I will write the damn code myself if he wants it. I literally write code to solve these sorts of issues all day long. I know there must be other professional developers in this community that can provide technical feedback.


ScrotieMcB: Most software engineers like myself are highschool dropouts. I'm a .NET developer so most of my time is spent in the Microsoft stack (hence my C# snippets), and some of the brightest minds in the industry have no college education. Outside of writing code, PoE is the only other thing that occupies my time. After several desync/hardcore deaths, my programming hours are starting to skyrocket again. I guess quitting PoE is for the better. ;-)

I noticed some of your posts in the past regarding desync. You seem to be one of the only other outspoken members of the community on this matter.

Just fix the bloody desync already. If Qwave's suggestions are unfeasible, fine, find another way to fix it. Just fix it.

"

Mark_GGG wrote:

If you want to know more about how PoE actually communicates between the client and server, I would recommend Chris's manifesto post (link here) as a good starting point.

Link doesn't work. Seems the manifesto posts got hidden when 1.0 was released.

"

Firecrest5 wrote:

"

Mark_GGG wrote:

If you want to know more about how PoE actually communicates between the client and server, I would recommend Chris's manifesto post (link here) as a good starting point.

Link doesn't work. Seems the manifesto posts got hidden when 1.0 was released.

This post on the Nexon forums seems to have the relevant GGG post copy-pasted in regards to their own discussion:

http://dragonnest.nexon.net/community/forum#%2Fshowthread.php%3F1001071-Client-server-Action-Synchronisation

Chris' development manifesto is completely wrong, and does not reflect the modern approach to network architecture in online games.

"

To prevent players cheating, it's important that these calculations are not done on the gamer's computer, because they can easily modify the result of such calculations. Because of this, all calculations that affect someone's progress must be done on servers that we control.

Modern online games use deterministic seeds to allow both the client and server to generate the same results. The server can still authenticate this without sending stateful packets back to the client, which are causing desync.

"

Trust the client. This means people can cheat, but the results are instant. We will not do this.

This does not mean people can cheat. You can allow the client to simulate results WITHOUT fully trusting it. This is why the client should send snapshots. As long as the snapshot is accurate, the server will never need to resync the client.

"

100ms: Your click arrives at the server. The character there starts running towards the monster also. At this stage your local character is already 5% of the way there.

This is a huge flaw, the packet should have a timestamp and hash so that the server's simulation can align with the client's latency.

"

Despite the fact that your information is delayed by 100ms, it arrived before the contact point and the only indication of playing under latency that the client noticed was the fact that it took a tenth of a second for the item drops to arrive.

You shouldn't have this problem in the first place. The client could easily have simulated this with a seed. You don't need to tell the client he hit the creature.

"

Even if no strange lag occurs, the monsters that are nearby are pathfinding on the client to where they think you are - which by definition is different than on the server because of latency.

Big problem here. Latency does NOT need to impact pathfinding in this manner! If the server and client are both running the same simulation, the server can validate the pathing as accurate based on the seed + timestamp.

"

The key thing to understand is that Action RPGs have to use an action prediction system like this. If they wait for confirmation of every action from the server then it feels terrible to control.

The client DOES wait for every action from the server, and it DOES feel terrible to control!

"

The key observation here is that improved resync code involves more rubberbanding than we have at the moment.

Wrong, improved resync code involves no resync code at all and a deterministic simulation.

"

Why some other games appear to not have similar problems

Other games don't appear to have the same problems because they are using the architecture I have described. They do NOT use a 'wait until server responds' method, all RTS games validate the game state in the manner ive described.

"

Unfortunately, we don't want to do any of those things! They each individually ruin part of the hardcore experience

Desync ruins the hardcore experience.

"

There are a lot of changes that we're experimenting with that may individually improve the synchonisation of the combat simulation

You don't need to experiment, the proven solutions have been part of the industry for a decade now.

"

The problem is that by the time the resynced information gets to the client, more actions could have occured.

Pardon my language, but 'no shit'. This is why re-syncing is not a solution, it's a band-aid.

"

qwave wrote:

The PoE client IS a dumb terminal. It does not perform any of the calculations locally, and its only role is to send input and update the game state received from the server.

The client is not a dumb terminal, YOU are the dumb terminal.

A dumb terminal does no calculations locally and updates status. Because of this no-simulate behavior, it CANNOT make decisions that are inconsistent with server state because it is waiting for server state, and as you stated, not performing local calculations.

Computing 101: In case of delay between a dumb terminal and the server, what does the dumb terminal do? It WAITS. It does NOTHING. At most, what you would have seen would be a screen freeze while the client is, according to your definition, UPDATING game state.

I find it amusing how a guy with supposedly years of experience as a developer does not understand the concept of what a dumb terminal is.

Sachiru: Just because a combat animation plays does not mean the client is not a 'dumb terminal'. It does not perform ANY calculations, at all. It is WAITING for a response from the server. Just because you can move around and click things does not imply that the client is performing any calculations.