GGPK Defragmenter
|
I was bored and made a tool to defragment Path of Exile's GGPK files. I shaved 850 MB off my client by using this.
It's a console application which takes two arguments, the filename of the GGPK to copy and a filename for the defragmented output. Support edit: Above exe download link has stopped working. Drakier has provided the original and a slightly modified download for the exe and source code (adds version number), as well as linux friendly code:
Downloads
Usage Instructions: Run it. If that fails, drag and drop the Content.ggpk from your Path of Exile folder on it. Note for Steam Users: Don't use this. Last edited by Michael_GGG on Mar 12, 2014, 11:59:57 PM Last bumped on Dec 8, 2023, 9:02:15 PM
| |
|
What exactly does this do?
| |
" The ggpk file contains many files and folders. It can also contain tons of wasted space. When the patcher adds or updates a file or directory, instead of moving around all the data in the 3+GB pack, it'll just mark that file or directory as free space. Unfortunately, the way it does this tends to create thousands of free blocks that don't get used. The ruby script he linked copies directories and files to a new ggpk, but not the list of free blocks. didn't look at the executable ;| Amusingly, it also incorrectly calls hash key for each directory entry a time stamp. :P |
|
|
We actually have a similar tool for this internally, but our one is an in-place defragment rather than input to output.
Old pack files can get quite fragmented after a lot of game patches. This is especially noticed internally where our developers are patching multiple times per day. I should mention that the the patcher does reuse free space for deleted files, it's just that there are a lot of oddly sized small chunks left over after filling up available free chunks with new data. This is exacerbated by the patcher trying to do atomic operations so that it can be closed at any time without causing corruption or loss of already downloaded assets. This means that if we add a file to a directory, for example, we have to make a new directory entry with the file added, commit that to disk, then swap the parent directory entry pointer over to the new one, then commit that to disk. As files are added or removed it can involve directory entries moving around a lot throughout the pack file, adding to the fragmentation. There are some improvements I'd like to do one day to prevent quite so much fragmentation, especially of the metadata, but in the mean time, feel free to use this tool. EDIT: By the way, that code is very clean. Great job! Path of Exile - Lead Programmer Last edited by Jonathan on Apr 15, 2012, 7:11:18 PM
| |
" It is always a good idea to be suspicious of random executables you download from the internet, so I can't really blame you. In this case though, Zoxc has been kind enough to give us the code for his tool. If you are worried about using the pre-compiled executable version, I advise that you run it from source code directly (which is what I did). You can do this by installing a ruby interpretor and installing the "bindata" package with "gem install bindata". Path of Exile - Lead Programmer
| |
|
I have a couple of questions regarding the format.
How are the hashes in the directory records and entries calculated? What's the purpose of the FREE nodes? They don't really seem to represent free space. What does GGPK stand for? :) | |
|
It seems to work fine for me. ~500 mb saved and I could play no problem!
Thanks a lot mate! :) I appear to be living in "Romance Standard Time". That has to be good! :)
|
|
" My guess is something like Grinding Gears PacKage file. Forum Sheriff
| |
|
The hashes are murmurhash2 on a lowercase'd copy of the entry's name, i think
also jonathan mentioned it tries to use free nodes but they often wind up being the wrong size for new data |
|
