[FIXED: 3.18.1.c2] Client deadlocks and very long map load times after 3.18.1c patch [id: 251,24..]

[id: 251,247,940]

Issues to be addressed:
- Map load times are significantly increased (previous: 0-2s, now: 2-7s+ on a high end system)
- Engaging with in-map systems or upon first attacking mobs in a map, results in a client freeze

These symptoms are present with the Vulkan as well as DX12 render engines.

After the latest hotfix, whenever I first start attacking a pack in a map, there is a 1-1.5s hard freeze to rendering and throughput, almost like the network core is getting blocked by the rendering or labeling engine. This is dangerous to the player as it could result in a death or disconnect and thus loss of portal or loot in a map, especially as there is nothing the player can do about it.

I'm able to reproduce the pause/deadlock bug by:
- Have syndicate members ready to interrogate in a map
- Leave the map
- Go back into the map
- Click on a syndicate member to talk to

The game will then pause for a perceivable amount of time, on interacting. In this screenshot, you can see the white server spike which is when I clicked on a betrayal member to interrogate them.


After this spike, the latency doesn't spike unless i leave and re-enter the map again.

Also when entering maps there is now a 2-7 second delay before the map becomes available. This did not exist prior the the latest patch. I have a very beefy, tuned system and well managed/monitored local network (RTX2080TI + i9-9900k + nvme, gigabit cable service).

Here is a video capture with live Client.txt timestamps showing my leaving a map, going to the hideout, then re-entering the same map. The load time is 7s for this near-empty map. Prior to the patch this would have loaded in under 3 seconds. You can see huge spikes in the CPU and server time during the load, again it feels like the network throughput has been choked almost like the concurrency has been reduced or the client is having to re-establish itself for every request that it makes; obviously something in the client has changed here.

I'm guessing that for players without low-latency + high i/o systems, they are experiencing far more disconnects or crashes, especially on area loading, if this "deadlock" condition causes them to hit the client<-->server timeout. Scanning for folks that took the time to file a report here or on the other-official-but-not-official forums there's an arguable increase of people reporting they are crashing between map loads and experiencing in-map freezing/crashing.

If I had to guess, something within this part of the patch is causing this performance degradation:

"

* Improved the texture systems memory usage and performance.

The situation feels like network throughput has been rate limited; almost like if the client opened up 5 concurrent requests at a time that limit has been reduced to 1 request at a time. Or, in prior leagues with an abundance of filtered loot if you bumped the alt key it would lock the client up when applying the filter/labeling (sometimes, causing a disconnect). Looking at the in-game network graphs you can see the disk activity and cpu latency both spike, so maybe this is something to do with how textures are being parsed and/or cached. If I recall there was a bug a few leagues back with regard to traversing and/or recreating a new tree of persisted textures on every texture load, so maybe that's something to look into (regression related to texture cache/load). Whatever changed in the client or at the server, feels like that (a very bad experience). I'm scared that I'm going to get disconnected every time this lock happens; which is every map. As this situation did not exist prior to the latest hotfix. Please revert the latest patch if a solution can't be found by end of business today.

GX Settings:
* W10Pro64bit | NV ver: 512.77 (game has been stable all league with this driver)
* In-Game Graphic Settings

My System:
* I9-9900K
* RTX2080 TI [516.59 as well as v512.77, NO geforce experience]
* Vulkan (Exact same issues exist with DX12)
* Windowed, 1920x1080
* SSD (WD_BLACK SN850 NVMe)
* Multi-threading (Crazy that Single core is even an option anymore)
* Have also cleared the following persistence caches: minimap, GLCache, D3d+Vulkan ShaderCaches, etc.

Thanks.

###

Edit:
* Kieren has acknowledged GGG is looking into the performance degradation (Thanks Kieren!)
* Bex has acknowledged the issue over on "the other place" (Thanks Bex!)

GGG is asking for additional debug information:
* CPU
* GPU (+driver version, if possible)
* Vulkan, DX12, DX11?
* Windowed or Fullscreen?
* SSD/HDD?
* Multi-threading/Single-threading?


Edit 2:
Looks like this is slightly improved, with the latest hotix; the hotfix addresses first-entrance-to-area load times:
* Hotfix 3.18.1.c-hotfix

Edit 3:
Attempted to summarize bottleneck analysis, visually in this comment. This degradation is currently beyond anything a typical player can do to fix.

Edit 4:
It's Monday in Auckland and GGG is back in the office! Nick_ggg posted over on "the other place" that a revert patch is being tested currently.

Edit 5:
Hotfix 3.18.1.c-hf2 posted that reverts the texture performance part of the 3.18.1.c patch.

I came looking about very long load times, have them too.

Another data point: After the patch, I've only noticed long load times in one circumstance. If I die in a map and reappear in my Hideout, then immediately jump in a portal to reenter the map, I get roughly a three-second lockup on the loading screen -- the gears stop turning, and CPU appears to go to 100%. Then the gears start turning again, and the map loads normally.

I'm using DirectX 12.

Also, TwentyFiveEx, have you tried switching from Vulkan to DirectX 12 or DirectX 11? The Vulkan renderer currently seems to be broken in a variety of ways, and I'm wondering if that could be affecting your in-map hesitation issues.

"

KickahaOta wrote:

Also, TwentyFiveEx, have you tried switching from Vulkan to DirectX 12 or DirectX 11? The Vulkan renderer currently seems to be broken in a variety of ways, and I'm wondering if that could be affecting your in-map hesitation issues.

Switched to DX12 to test and I'm able to reproduce every time the same longer map load times as well as the initial pause/freeze that happens when I first engage mobs in a map. With flipping the render engine now I suspect even more that this has to do with the texture loading part of the recent patch.

Additionally I've noticed when moving from a Vaal side area back to the main map, I get the same longer load times as well.

On a positive(?) and unrelated note, DX12-Beta performance looks to be more or less on-par with Vulkan (so far).

Edit: after playing with DX12-Beta for a bit, switched back to Vulkan. Seems unrelated, but DX12 causes some sounds to no longer play; such as the expedition NPC voicelines. I don't understand why the graphics setting would affect the sound settings, but it seems to lower the number of sound channels or something. Also there's tiny bits of input lag that aren't present with Vulkan.

The game is unplayable on high-end CPU. Multi-threading is fucked



multi threading off - butter smooth



multi threading on - lag and stutter

I started to have exact same issues after the patch with high end pc, delirious maps became literally unplayable cause of the stutters and overall i started to have stutters, it was perfectly fine before the patch.

As someone with a low-ish end pc, i too noticed this, though as odd as it sounds my loading times for simuls have not been affected too much, just mapping and some non map areas, i didnt have this issue before the "hotfix".

Been having issues since patch posted in other threads but since this one is more consolidated I'll throw in PC Specs info here

CPU i5 4590
GPU 2070 Super (Driver: 512.59)
Running DX11 currently havent really tried swapping cause Vulkan usually crashes me and DX12 crashes me sometimes on switch
Windowed Fullscreen
SSD - 870 EVO
Running on I believe whatever is default which I think is multithreading

My main issues are increased load times 45s-1m up to 2-3m now. Getting massive spikes when something pops up first time in a map aka influence, Betrayal ( This happens every time I reenter the map not just the very first time aka vaal side zones or portaling out and back in). Side not not sure if it is related but getting random rendering failures like blurred health orbs (rarely) and delve map chart not showing pathways (fixes with I zoom out and then back in, but occurs every time I open map)

Also don't know if this was the same prior to patch. But halfway through map loading, the gears stop I hear the in game sounds from the map gears go back to spinning then I load in after a short period.

Game unplayable after the patch for me. Loading screen takes forever, studders in maps, and Delerium is a big no-no. :/

Somewhat high-end user

Intel i9-10980XE
120GB RAM
GF 3090 RTX