Connection issue that only works with VPN (Cant even play the game)

To start I never had this issue with POE1.

So I have a very odd issue that I cant seem to find anything about online.

I am based in the United States.

My game will connect to the server (Washington D.C) and then when I load into the game either it will never load in and be stuck on the load screen or when I do load in it will freeze after I take my first step in game and then kick me from the server entirely.

I was very frustrated with this at the start but I found early on that for some reason if I turn a VPN (country didn't matter) on I could connect and play perfectly fine. (I started late just last week)

Now I'm having issues with my VPN provider today so my connection with them is unstable which means I cant play at all. At least that's what I'm assuming.

So I'm finally trying to resolve this issue.

I use a direct fiber optic line into my home and I do not share that connection with anyone.

I have a hard wire connection into my router.

I test at 650mbs down and up.

My firewall is not blocking poe2.

I even took my firewall down entirely to make sure that wasn't the issue but it still did the same thing.

I tried creating rules for outgoing connections for the POE2 files and that didn't work.

I'm a bit at a loss on what to check now.

I use Windows 11. Ryzen 3800x, Gigabyte B550 X V2, 32gbs ram, 4070ti, and i run the game off a Samsung evo M.2. IF any of that matters. I also use a Nighthawk router.

Any help would be greatly appreciated.

Edit:
-i have also tried using multiple different gateways
-other people are online and playing just fine as the 4 seconds i get to play sometimes i can see people talking normally in chat
-I thought it could be an ISP issue but if so why would i be able to play everything else normally? (other games, streams, spotify, ect..)
-One thing is for sure is I absolutely cannot and have not been able to play without a VPN.
-I don't think it is the VPN connection as i mentioned previously because again everything else is working but now i am having continuous issues with the VPN

Edit#2:
-i have went through my router settings and the only thing i could think that would effect it is a geo filter which i had enabled which i maxed out to the entire world but still nothing.
-Yes, i have reset my router and that did not work.
-When loading in on my normal connection without a VPN it will freeze my entire computer up and i have to hard shut it off because i cant do anything. No alt tab, no task manager, no alt + f4.

At this point i cant even play the game.

I'm having the exact same issue from sweden with ownit internet provider.

Every load screen will result in "failed to load instance" repetedly or hard lock my pc.

7800x3d
32gb ddr5
rtx 4090.

1000/1000 fiber internet.

Edit: My issue is with PoE2, havent played PoE1 in years.