[Performance] Memory management and Soft Page Faults: Low FPS as a result of...
|
[Performance] Memory management and Soft Page Faults: Low FPS as a result of assets being loaded on the fly from the Commited (private) memory onto the Working Set causing CPU cycles to be wasted and performance to drop as the game can't continue seamlesly without certain assets loaded.
I've been salty about performance this league and decided to take a relatively deep dive for a CS-less consumer pleb like me. In this post i will set up the argument that, under the right conditions, certain game assets are loaded into memory and relegated to unmarked secctions of the private memory, needing to be reloaded into the working set to be accesed, causing slowdowns on game performance. Let's start from the top: What is the memory? Once you start PoE, as with any other program, the application asks the computer "hey where should i put all this data i'm gonna need" and the OS tells the program "put em in RAM, it'll be easier to retrieve it later." So the program does just that, and all the audio, texture and topology data along with all the instructions necesary to use them are loaded into RAM. After it's loaded, the OS says "y'know what? you don't really need ALL of that right now, don't cha? because you're not alone here buddy, so why don't you make a list of all the things you don't need right now and put them at the front of the room so, if any other aplication ever reeeally needs more ram we can evacuate that section and give them some room." Thus, memory is divided into two sections: the Private memory (the whole data loaded into RAM) and the Working Set (whatever the program tought was absolutely necesary). There is then a section of memory that is in ram but not tagged as essential. Sometimes, the whole computer runs out of RAM. "Oh shit, what do i do now?... ohhh that HDD is looking pretty fine over there... lemme just..." And then moves some of the non essential memory to the disk. Where? to the Page file. All the information about what is where in the memory is stored on what is called a "Page Table", a list of assets each with a memory adress assigned to it. Now let's think about the program. Let's say that PoE loaded the texture of Malachai into memory and needed to use it, This would happen every rendered frame. So the program goes to the page table and finds where malachai's texture is stored then moves it to the GPU for processing then does it again and again until the texture is not needed. What happens if RAM ran out at some point and the texture had to be moved to the disk? Well, then the PoE says "Whoops, this shouldn't be here. I'll just move it into memory again and continue." This is what is called a "Hard Page Fault" and it takes ages in terms of computing time. This is why i urge everybody who can do so to drop the Page File Size to the lowest ammount possible, and no Page File would be the best option. However this post is about Soft Page Faults. What happens if malachai's texture was loaded into ram but was deemed "non essential" and de-tagged from the working set? Well, the program simply needs to find a place in the page table, effectively moving it into the working set. This is called a Soft Page Fault. During this process PoE can either show no texture, or show a matte texture while it laods it into memory for processing and displaying. Let me show you some experimentation i did early today. All the following videos were recorded without a pagefile (that doesn't mean there are no hard page faults, but that's normal since the OS will always need a page file idk why tho) First, let me show you what i mean by "Slow texture loading because of Soft Page Faults" ------------------------------------------ TEST 1 ------------------------------------------ --noasync is ACTIVE https://www.youtube.com/watch?v=JqN8DpOpVHw The video shows the loggin process, and what many people will recognize as the "flame flicker", the moment where the flames of the scene dissapear for a moment before reapearing right after clicking on the loggin button. To the left of the video we can see the Performance Monitor showing three lines: In red, processing time (the % of the processor being used), in blue pe thage faults per second (both hard and soft, however i've never seen more than 20 hard faults per second on process monitor) and the IO read bytes per second (how many times the process reads from the disk to load new textures) At 9 seconds we can see that after hitting log in there is a diip in FPS seen as a stutter along with an increase in soft page faults. The peak is read at 30000 page faults per second during no more than a second. It's my duty now to try to correlate this dip in FPS to the increase in Soft Page Faults. I will also argue that, up to a point, processing power is not the primary factor when it comes to stuttering during asset load and remark. ------------------------------------------ TEST 2 ------------------------------------------ --noasync is ACTIVE https://www.youtube.com/watch?v=vUCmPqfOTtk This video shows gameplay of a Tier 4 Phantasmagoria, the reason why will become apparent towards the end of the video. After entering the map, the game loads assets into the working set, namely the map topology and textures. Then, after a period of calm the first pack of monsters in encountered and the game needs to load the textures from non-tagged memory to the working set. This causes a significant ammount of soft page faults peaking at 30000 and mantained for about five seconds as more monsters enter the battle. The gameplay continues in relative normality due to most assets being now loaded into the working set, but as the timer hits 1:55 we reach the entrance to the boss room. As the game has not loaded the texture assets for this section into the working set it must now do so, increaseing the number of Soft Page Faults. However this behaviour is sustained over a longer period of time and contrary to popular belief the Processing time does not go up, but down. We will later see that this behaviour is exclusive to clients running under the --noasync command, however it sheds light to the fact that long asset time is at least not only by cause of low processing power. These Suft Page Faults peak at around 25000 for 19 seconds, and the metamorph Soft Page Faults peak at around 60000 for one second, predecessed and followed by smaller ammounts of Soft Page faults, NOT accompanied by a decrease in processor time (however this could be just because it wasn't a long time) ------------------------------------------ TEST 3 ------------------------------------------ --noasync is ACTIVE https://www.youtube.com/watch?v=F5Z55ECNpvA Phantasmagoria T4 now with double beyond + 4 scarabs that add their own monsters and textures. The map is played under the same Soft Page Fault dips in FPS and the boss entrance FPS drop is reproduced at 4:02, this time it's more apparent that a spike in Soft Page Faults is accompanied by a valley on Processor Time. (Also, RIP my ass). The Metamorph FPS drop is also reproduced at 5:40 with inconclusive dipps to Processor Time. ------------------------------------------ TEST 4 -------------------------------------------- noasync is INACTIVE https://www.youtube.com/watch?v=DXEiMxBJMBQ Big Boi Phantasmagoria T14 with 4 scarabs and double beyond. The FPS dip at boss entrance is reproduced at 7:17 with noticeable improvements to the load time of assets thanks to the exclusion of the --noasync command, however it is still noticeable. The Processor Time is at a snug 100% and the Soft Page Faults peak at 45000 for aprox. 14 seconds. Notice how there is a period of no Soft Page Faults in between the end of asset load and the boss' death. Then, as the loot enters the scene, the Soft Page Faults resume. ------------------------------------------ TEST 5 -------------------------------------------- noasync is INACTIVE https://www.youtube.com/watch?v=aOuBicBALPY Finally, a test showing the interaction between Private memory, the Working set and the gap in between, the Unmarked Memory. Notice how at 2:25 entering the map is accompanied by an increase on both the Working set and Private memory, the I/O reads showing loading memory from disk to ram, and soft page faults. After that, at 2:38 a period of 22 seconds of stuttering accompanied by an increase in Soft Page Faults but very low I/O reads and a slow rise in Memory usage. This behaviour continues on a smaller scale untill my ass RIPs at 4:46 and we see a dip in memory usage, both in Private and in the Working Set. Again, at 7:30 the Boss Entrance FPS dip is reproduced accompanied by a high number of Soft Page faults, low I/O reads and a slow increase in memory usage. The metamorph FPS dip is reproduced shortly after, in this case accompanied by a small peak in I/O reads. Following the Phantasmagoria map i ran a Veritania Citadel because why not. Again, FPS stuttering and dips are accompanied by high numbers of Soft Page Faults. At 10:33 we can see the loading of Veritania's tower, however this was followed by a very small peak in Soft Page Faults and a pretty quick loading time, in comparisson with the Phantasmagoria boss room entrance. (PS: Please don't tear me a new one on my performance against veri, i gon goofed up switching gems LUL) ------------------------------------------ Conclusions ------------------------------------------ Based on these experiments i conclude that Processing Power is, at least, not the only factor associated with texture loading stuttering and that, at the least, the way textures are loaded for use during gameplay cause dips in performance that are non-trivial. How do we fix this?... i don't know, i'm just a salty dude with access to the internet and a whole 2 whooping hours researching this subjet. I'm sure y'all can prove me right and more importantly prove me wrong in most if not all of what i said. I just want to give my 2 cents and hopefully this leads somewhere. EDIT: I misplaced and forgot some of the TEST tags, fixed! EDIT 2: posted in official forums, Tech. Support. Original Reddit thread is https://www.reddit.com/r/pathofexile/comments/eus6u9/performance_memory_management_and_soft_page/ Last bumped on Mar 18, 2020, 1:43:49 AM
|
|
|
This is worth reading!
|
|
|
+1. Bump!
|
|
|
bump - imo this worth dev respond
|
|
|
its not only dynamic loading, its also dynamic shader creation every time
i have 37k files in shader cache folder atm for only 440mb, it kills any file system table
Spoiler
 why just dont pre-made all shaders and put them to ggpk, like most games do? and just load them all to ram when game starts, its only 400mb Last edited by Sonidox#3895 on Jan 28, 2020, 3:04:42 PM
|
|
" With so many files, i think you have a lot of outdated/unused shader combinations in your cache. You should delete those directories, the game will recreate them very fast (you can save them before deleting if you want) I think the correct number is more 4-5k files (i have 3.7k) |
|
|
I was suffering from the very same issue after updating my system from win7 to win10. I got a nice fps increase but started suffering from huge chokes whenever the game pretty much loaded anything. Due to the increase in fps I had trouble pointing the issue to the systemswap but after reading your threat on reddit I did some digging around.
Turns out windows 10 had put the page file onto my HDD while the system is properly installed on my SSD (PoE aswell). If anyone else is running a HDD disk with their SSD and suffers from the same issue you should check where windows keeps your page file. Search for Advanced system settings -> Performance: Settings -> Advanced page, and there you find your Virtual memory settings. There I simply added a page file onto my ssd and disabled it from my hdd. Edit: make sure you have enough space on your SSD for the page file to work properly. This is probably why it was put on the HDD in the first place. Last edited by apeus#2571 on Jan 28, 2020, 4:02:27 PM
|
|
" I was in your situation and did the exact same thing after reading the OP. Solved most of my problems too ! Another thing I found that seemed to help a bit: install in windows key registry that allows you to disable hard-drives power management (I believe win10 set hard drives to idle when not used for a while, this could results in short freezes on sudden pagination or loading). See here: https://www.tenforums.com/tutorials/72971-add-ahci-link-power-management-power-options-windows.html Would be nice if GGG could confirm this ? Cheers Last edited by ValMishra#4225 on Jan 28, 2020, 9:28:36 PM
|
|
" 18 days since folder was created, so only 2 patches long folder :D Last edited by Sonidox#3895 on Jan 29, 2020, 4:19:26 AM
|
|
|
I'll just observation which might or not might be related to the post.
Due to known AMD metamorph crash, I made guide how to use DXVK (directX -> Vulkan translation) workaround. DXVK is known to not work properly with PoE async and requires hack to enable async to some degree (but still stutters). Basically when running game trough DXVK it behaves very similarly to the --noasync option as presented here in the videos (though not as bad). On topic of Hard / Soft page faults, this is managed by operating system memory management and software/game itself has limited control about it. It can flag some memory as nonswappable (so it will not get dumped to page file) but thats about it. Also, following statement is not entirely correct: " To clarify more, the pagefile is used _ONLY_ for data which cannot be written back to disk or don't exists on disk. If you run out of memory, and more memory is needed, files in memory which are same as the one on disks are simply dropped from the memory because they can be read again from original location. However the statement might be true as all files are saved in content.ggpk "archive". I am not familiar with it, but in case it is really some compressed archive, the "Malachai" example would be indeed put in PageCache. Therefore actually disabling PageCache might not be good idea after all, because you would spent more CPU time unpacking the original file than just reading unpacked one from the pagecache. Additional point is that _lot_ of lags comes from shader compilation. Based on my understanding, the game compiles/combines shaders on the fly. Async (asynchronous - default) rendering results that game engine does not wait for shader compilation to finish, but it will give you some placeholder until the shader effect is finished. With async disabled (--noasync) you need to wait, and game will stutter. Over the years the amount of MTX and skills effect went so high, the number of combinations which can occur is staggering and I think using precompiled shaders for every such combination would be extremely costly for HDD space. (much more than you actually see in your Shadercache folder as those are created only for the combinations you've seen so far). In any case, all this points to issue with how PoE handles the rendering part. They are continuously adding new things, without much optimizing, and I guess this is the real reason behind the performance and/or the MM and hard/soft fault we see. |
|
