Hit with('X') a loot or an objective (altars in trials) is almost impossible

I declare this a semi-challenge godly bug to solve:
While in combat, clicking a target with 'x' on the ground or an altar, might be a challenge.., more than poe 1.
There's a complicated way:
turning the r3 joystick and hitting 'X' with the index, or:
change the keybind for the: "action", might solve this..
-if it's an option??-...

in a similar topic I suggested that the only possible way to do this maybe to unbind any skill from the X button.

it's terrible design.

gives me flashbacks to trying to interact with the towers in the blight encounters .. some of their systems/mechanics are simply unfit for use in a console game.

edit: I've tested this and it works perfectly. Removing a skill from the interact button (X) allows you to interact with Death Crystals even if enemies are nearby. Though finding a workaround does not excuse the fact it's a bad design choice since we are having to remove one button from use.