(PC) Cannot assign mining workers to different ores using only controller

When pressing Xbox X to assign a worker to a different ore, you cannot move the Ore selection at all on controller, it appears the focus is still tied to the parent worker selection UI and any controller movements will work on that layer and not on the ore selection.

Video showcasing this with an overlay showcasing my inputs:
https://streamable.com/ssgji4

Able to workaround this by using mouse, but its somewhat inconvenient.

Bug is also on the Xbox, and the work around doesn't work due to the mouse not being supported on said platform.