[FIXED: 3.18.1.c2] Client deadlocks and very long map load times after 3.18.1c patch [id: 251,24..]

"

Henry_GGG wrote:

Thank you for the detailed report, we have deployed a fix for this.
If you restart your client you'll download the new patch and this should improve things for you.

https://www.pathofexile.com/forum/view-thread/3287415

The problem persist for me. Made this discovery tho:

I deleted my minimap and vulkan shader cache folders (in appdata). After that the long load upon entering a juiced map goes away, but temporarily. After you port out, the next time entering the map takes 20-30s again.

This isn't a one time thing, I can reproduce it. Map is crimson temple deli orb, and juice sextants/atlas.

Clear cache/minimap folders.Enter map first time, 5s load.

Clear some of the map. Port out of map.Port back in, 10s+ load.

Clear some more. Port out of map.
Port back in, 20s+ load.

Close the game and delete folders again.
Open game and port back to the map, 5s load again.



It seems as you build the shader cache the load times become increasingly longer.

I don't know if they are related but I have severe graphic glitches and more frequent Vulkan crashes since the 30th, and after the 30th hotfix patch it all got significantly worse.

Details in my report: https://www.pathofexile.com/forum/view-thread/3287640

Played yesterday without any problem.
Today I logged in and downloaded the small patch.
Game is crashing every 2-3 minutes.

Nothing have changed from my side. That's why I know the problem comes from your side.

Just fix it!

After minipacth, when I enter to game newly, feels issue gone, but wait, however you starting to run just 1-2 map, there is again exactly same very long loadtime, and in the middle of loadscreen game totally freezes for 2-3 seconds.

"

Henry_GGG wrote:

The main issue was addressing the problem with load screens being too long. In some cases it was quite extreme, that specific problem has been fixed.

However, there are still some other problems we're seeing that need to be addressed.
We'll continue to work on those problem until they have been improved.

Given the number of people having extreme issues with this patch and the obvious before/after, I would urge you to bring to Chris the need to roll this patch back. Unless there is some non-public-knowledge reason this patch needs to stay in place (such as fixing a significant security issue), this patch is absolutely ruining the game performance.

I am creating a repeatable process to analyze bottlenecks between area loads to best demonstrate visually where the problems exist. Unfortunately as I am not a GGG developer, I cannot compare this behavior to the prior-to-3.18.1c production build. So far, it is very much looking like changes introduced post 3.18.1c have created an i/o contention issue even on the highest end NVMe devices. The game state must be absolutely unplayable for persons without high end systems.

I'm ignoring the "first-encounter" pause that happens, as I believe the map load times and the "first-encounter" pauses are directly related. Currently here is where I am at with the analysis.

It is important to note that before the patch, going through my logs, I am able to see that my load times between areas were typically 1s or less.

The scenario:
- Create map
- Enter map instance, clear through boss
- Leave map
- Re-enter map


System operation summary:


While there are some overlaps in behavior, given the map re-entrace post-hotfix now highlights the performance degradation to a stronger degree I am focusing on bottlenecks presented in the re-entrance as I believe they will better enumerate problems for both conditions.

It is measurable the number (in the thousands) of operations that occur reading and writing to the GLCache, and these operations in summary create a very lengthy load time between area changes. However, again, I am not able to compare this behavior to the prior "non-degraded" build and would ask that GGG developers do this comparison to validate the prior behavior.

Also of note: the best way (currently) I have to measure each operation is the time difference (in microseconds) of timestamps between one operation and the next. This does not take into account cases where the operations are performing in parallel. In another analysis, I may attempt to break down the time differential across operational-types (ex: fileop to fileop).

In any case, this is very clearly beyond anything a player of the game is able to workaround and I would strongly urge for a rollback (perhaps of just the texture caching changes?) present in the latest 3.18.1c patch until this problem can be fully reproduced and fixed.

Futher breaking down performance analysis by operation type and total time taken, you can clearly see there is a strong outlier present with registry calls to "HKLM\SOFTWARE\Microsoft\WindowsRuntime\WellKnownContracts".



I took the two repeated operations that seemed to take the most time and broke these down further (lower left and center of image).

I'm not sure what that call to the registry is for, but you can see it happening over and over and in many cases it takes a very long time to respond. The "RegQueryKey" (or subsequent operation) seem to be taking up the bulk of the bottleneck. Again, because I am not a GGG developer, I cannot discern if this is a cause, effect, or wholly unrelated -- but it is certainly worth looking at and understanding. If these are just simple registry key fetches, it probably would make sense for these to get cached rather than repeating hundreds of times per area transfer, maybe?

the game still takes 1-5 minutes to load for me even after the fix karui shores takes 4ever to load.

im really new and hope this gets another fix sent out GREAT GAME having a blast so far
my pc:
intel core 17-8700
DDR4 8G
NVIDAGeForce GTX1060 6GB

I've now added a breakdown of network traffic behavior and there is an observable difference between first entering the map and re-entering the map. Again, I don't have a pre-3.18.1c analysis to compare to; hopefully GGG-devs can perform this task.

On entry, the system operation calls measure at least 2.65 seconds on receiving data from an ip that isn't the instance ip (209.191.158.140). On re-entering the map 1.78 seconds are spent receiving data from the instance ip as well as 1.4 seconds are spent receiving data from this other ip (3.18s in total). This is very strange, for this difference in traffic on entry vs re-entry and likely also needs some digging from GGG development to fully understand. Given that the symptoms are where the client seems to deadlock during area transfer, these values may be the network stack taking an extended pause while blocked by other operations (such as reading+writing the shader/texture files).

Nothing has bean fixed for me takes extreme long to load any map's or any instances.

I used a RAMDisk as a workaround and put %APPDATA%\Path of Exile there. Just install https://sourceforge.net/projects/imdisk-toolkit/ or some other software and put %APPDATA%\Path of Exile on the RAMDisk (you'll have to repeat/automate this for reboots). After clicking a portal, it now takes a mere 3 seconds to load the map. If you have plenty of RAM (~64GB+), you may even put the entire game dir on the RAMDisk.