GGPK Defragmenter

"

HanSoloDK wrote:

"

realVIP wrote:

is this still working? after the last patch? i just found this post and figured would do it for the new upcoming league but wanted to ask first if it still works

well you just redownloaded the whole client so there is nothing to defrag.
Also we have to wait for GGG to confirm if this tool works with the new file format.....

official POE discord , people have deduced that it is not ggpk and instead something else that use Oodle

http://www.radgametools.com/oodlecompressors.htm


https://discord.com/channels/174993814845521922/175290321695932416

The GGPK file format is identical except for a version field that most tools have previously thought to be a "child count" field, including this defragmenter. Due to this they are likely to malfunction on the new v3 GGPK files.

The changes needed for this defragmenter on PC are trivial, someone just needs to make them and prepare a new build of it that people might trust.

The standalone Mac client uses a slightly different structure inside and also has a different version tag, so the tool will definitely not do the right thing on macOS.

As for whether you need to defragment anymore - "it depends".

The GGPK pack of the standalone client now has a mix of loose files (mostly dialogue and videos) and new "bundle" files like the Steam version.

This brings the file count down significantly from well over half a million file entries to less than 40 thousand entries. What previously were related loose files have been bundled up into compressed bundles, which are replaced completely when a patch occurs.

This means that there's way less fragmentation potential since the tech patch as the files replaced are larger and fewer, so the need for this tool is greatly lessened.

With the 3.13 patch my content.ggpk grew to 40GB (around 14GB increase from before). My ggpk isn't event that old, I downloaded it a couple of leagues ago. So I deleted it and now the new one is 27GB!
Also deleted the shadercache (8GB). Still waiting to see how large it grows.

This tool definitely needs to be maintained. Saying "Ohh your file grew to 60 GB, just delete and redownload it again" should not be our only option.

According to what ZaoFishbones saying, patch should looks like this:

"

diff --git a/ggpk_compact.rb b/ggpk_compact.rb
index 9e6e1a0..d0a9afe 100755
--- a/ggpk_compact.rb
+++ b/ggpk_compact.rb
@@ -23,8 +23,8 @@ class GGPK
class GrindingGearRecord < BinData::Record
endian :little
record_base :base
- uint32 :child_count, :value => lambda { children.length }
- array :children, :type => :uint64, :initial_length => :child_count
+ uint32 :version, :value => 3
+ array :children, :type => :uint64, :initial_length => 2
end

class DirectoryEntry < BinData::Record

I've tested this with 3.13

"

Defragmenting /home/admin/Games/Path of Exile/Content.ggpk
This will likely take a while...
Elapsed time: 3.46 minute(s)
File reduction: 1210.26 MB(s)
Successfully defragmented the GGPK.

EDIT: Tested, it works!

"

1ofeach wrote:

This tool definitely needs to be maintained. Saying "Ohh your file grew to 60 GB, just delete and redownload it again" should not be our only option.

yes. I am not sure the tool was supposed to be connect to internet or not but the current precompiled version may have some nasty code inserted

"

HALO282 wrote:

small note:
i just ran the pre compiled(dakier version (07.09.2020(local time just past midnight CET))) .exe and it changed 41 gb to 28 gb and the loading times was reduced damatically.(took about an hour on my 10 year old harddrive but the in-game loading time is basically instant again)
however,
at the end, when the console ran only showed "press any button to continue" the exe tried to acces teh internet which i can not see it doing in the source code.(it was blocked by my anti-virus so i could stop it)
so the compiled code i deifnetly not the same as the linked source(im not a ruby programmer so maybe i miss something there)

Ran it and now poe does not launch, probably messed something up in the files, be cautious

"

Aias_o_Telamonios wrote:

With the 3.13 patch my content.ggpk grew to 40GB (around 14GB increase from before). My ggpk isn't event that old, I downloaded it a couple of leagues ago. So I deleted it and now the new one is 27GB!

Can confirm this. The 3.13 patch brought the GGPK from ~24GB to ~39GB, while erasing it and redownloading brought it back down to ~27GB.

So this idea that the new format handles filesizes better is simply wrong.

As redownloading this much data isn't acceptable for low-speed internet, this tool should be correctly updated to work with the new format.
Barring that, maybe GGG could make their own in-house defragmenter available ?

Finally found a working ggpk defragmenter. You can get the zip file from this releases page (or directly from this link).

I cloned the project locally and compiled it myself. The defrag completed successfully. My ggpk went from 40.4 GB to 27.1 GB. I started PoE without issue.

Hi everybody, Bonjour tout le monde !

I just tested the defragmenter from @TheSmellOfNapalm's post and it's working so well !!! Thx for the link :)

So i decided to make a guide, i hope it will help some of you.

Pour les francophones, si vous voulez un peu d'aide vous pouvez me MP ;)

My config :
- windows 10 pro
- 64GB DDR4 - Nvidia RTX 2070 S - I7 10th Gen

Executable downloaded from GitHub --> https://github.com/MuxaJIbI4/libggpk/releases

Get this file : DefragmentGPPK.zip



Unzip the archive -->



Go to your POE directory (program file)

Drag your Content.ggpk file onto DefragmentGPPK.exe (simply open 2 explorer windows, no need to copy anything in POE directory)

You will see a window (the app) like this



At the end of the process, it will be closed.

Check your POE directory, you will find a new file named Content.ggpk.defragmented



As you can see, my original file size was 40Go, 28Go after defrag. Simply delete de big one and rename Content.ggpk.defragmented into Content.ggpk

Launch the game and enjoy ;)