POE 2 - Ogham Manor - Impossible to access for quest

So basically, I went to the Ogham Manor for the quest to fight the Count. I didn't succeed so I quit to up my character.
Now I want to come back to the manor to fight him but the game says that I cannot obtain access to this area after dying.
I tried by going through the door by passing through the Manor Ramparts area and i tried using the teleport.
Either way I get the same message "you cannot rejoin this area after dying"

Leaving the game didn't change anything. So I'm blocked and cannot move forward

any solution ?