Sao Paulo Server latency

"

Luxiel wrote:

Instead of bashing, I will try to provide some constructive feedback on this, as me and my friends work in IT and have some infrastructure knowledge that might give it a hint on what is happening.

We have been experiencing lag spikes (as in huge slow-downs, connection related, as the animations keep on playing but the game stops responding to commands) which are followed by what we call "fast-game", where all actions we've sent over during the lag spike are played in a fast-forward manner, as the game client catches up with the game state.

In fact, we have noticed that lag spikes and slow downs happen quite often during mass killing sprees, such as when we are freeing monsters from legion monoliths. Intense effects, like Herald of Ice explosions, Impulsa's, Duelist Blood Explosions, weight harder on this and this get worse when multiple players from the party are exploding things too close to each other and too frequently.

As things go on, we figured this is a problem introduced by bad connection or bad handling between the SP Gateway and the Realm, which is located in the US if I recall it correctly. As monsters dying and the explosion radius calculations for area and damage around the dying mob need to be sent back and forth (and these calculations are, historically, badly optimized, as 2.5k+ hours of playtime made me believe), this get all caught up, queued and jammed. Packages go lost and the mess is done.

On other circumstances, instances freeze for a while and then die, booting us out of the game. When this happens, it is usually accompanied by a Rollback, which lead us to believe the Gateway had a bad time communicating state changes to the realm, which never received them or even rejected them, which in turn makes the instance fail some validation, causing it to be aborted. This make it be as if the instance never happened (apart from the occasional map item being lost, as it was consumed on a healthy hideout instance but end up opening a bad map instance).

Now, I mentioned healthy hideout instances because this is not a constant issue, there are instances that behave normally and while the SP gateway has had quite a common streak of problems in the past leagues, these seem to be kinda specific, so what we figure is happening is this:

The SP Gateway most probably sits on a Datacenter and it is likely compromised by an Array of computers. Most probably more than one Instance Manager (what I am calling the computers that bear instances AND relay their info to and from the Realm) or one Instance Manager and multiple Instance Hosts (what I am calling computers that create and host instances, communicating their state to an Instance Manager for it to relay it to the Realm).

Now, as these problems are not happening 100% of the time, what we figured is that one or maybe a few of these computers are bad, have some hardware fault, bad RAM or dead connection points. Maybe they are fully or partially virtualized and the VMs are running bad or on low resources.

Also, there are healthy Instance Managers (or at least Hosts, if their architecture happens to be like that), and we can get to play as normal from time to time. We have been comparing results of different instances and have seen people playing normally on a Map while other 2 people on the same group and another instance (and each person on different physical locations and ISPs) having the same problems and the very same second. The latter happened to be on a instance created or managed by a bad machine.

To further support this idea, I personally have seem the following message twice on crashes: "Game server lost connection to the realm". This clearly states that the problem isn't between my client and the Gateway, nor with my ISP, but between the Gateway and the Realm, which explain why rollbacks occur.

I hope this helps whoever reads this to track down the bad apples in the SP Gateway, it shouldn't be that hard to pinpoint which ones on the machine array are misbehaving and have them subbed out.

Hopefully, Fitzy or someone else sees this and have time to go about this.

Thanks for the great game, keep up the good work!

"

Freakey_ wrote:

But I already lost all hope GGG will fix the SP gateway. Not now, not in 4.0, not any time in the future.

"

Luxiel wrote:

Instead of bashing, I will try to provide some constructive feedback on this, as me and my friends work in IT and have some infrastructure knowledge that might give it a hint on what is happening.

We have been experiencing lag spikes (as in huge slow-downs, connection related, as the animations keep on playing but the game stops responding to commands) which are followed by what we call "fast-game", where all actions we've sent over during the lag spike are played in a fast-forward manner, as the game client catches up with the game state.

In fact, we have noticed that lag spikes and slow downs happen quite often during mass killing sprees, such as when we are freeing monsters from legion monoliths. Intense effects, like Herald of Ice explosions, Impulsa's, Duelist Blood Explosions, weight harder on this and this get worse when multiple players from the party are exploding things too close to each other and too frequently.

As things go on, we figured this is a problem introduced by bad connection or bad handling between the SP Gateway and the Realm, which is located in the US if I recall it correctly. As monsters dying and the explosion radius calculations for area and damage around the dying mob need to be sent back and forth (and these calculations are, historically, badly optimized, as 2.5k+ hours of playtime made me believe), this get all caught up, queued and jammed. Packages go lost and the mess is done.

On other circumstances, instances freeze for a while and then die, booting us out of the game. When this happens, it is usually accompanied by a Rollback, which lead us to believe the Gateway had a bad time communicating state changes to the realm, which never received them or even rejected them, which in turn makes the instance fail some validation, causing it to be aborted. This make it be as if the instance never happened (apart from the occasional map item being lost, as it was consumed on a healthy hideout instance but end up opening a bad map instance).

Now, I mentioned healthy hideout instances because this is not a constant issue, there are instances that behave normally and while the SP gateway has had quite a common streak of problems in the past leagues, these seem to be kinda specific, so what we figure is happening is this:

The SP Gateway most probably sits on a Datacenter and it is likely compromised by an Array of computers. Most probably more than one Instance Manager (what I am calling the computers that bear instances AND relay their info to and from the Realm) or one Instance Manager and multiple Instance Hosts (what I am calling computers that create and host instances, communicating their state to an Instance Manager for it to relay it to the Realm).

Now, as these problems are not happening 100% of the time, what we figured is that one or maybe a few of these computers are bad, have some hardware fault, bad RAM or dead connection points. Maybe they are fully or partially virtualized and the VMs are running bad or on low resources.

Also, there are healthy Instance Managers (or at least Hosts, if their architecture happens to be like that), and we can get to play as normal from time to time. We have been comparing results of different instances and have seen people playing normally on a Map while other 2 people on the same group and another instance (and each person on different physical locations and ISPs) having the same problems and the very same second. The latter happened to be on a instance created or managed by a bad machine.

To further support this idea, I personally have seem the following message twice on crashes: "Game server lost connection to the realm". This clearly states that the problem isn't between my client and the Gateway, nor with my ISP, but between the Gateway and the Realm, which explain why rollbacks occur.

I hope this helps whoever reads this to track down the bad apples in the SP Gateway, it shouldn't be that hard to pinpoint which ones on the machine array are misbehaving and have them subbed out.

Hopefully, Fitzy or someone else sees this and have time to go about this.

Thanks for the great game, keep up the good work!