Setting mouse buttons for weapon sets in POE 2

As a sorceress I am running weapon two sets. Set 1 with a fire staff and set 2 with a lightning wand. I want to use my rmb for attacking with either set (like I can in POE1).

When I set my rmb for a fireball attack with set 1 and then change sets to 2 and set the rmb to spark. When I change back to set 1 the rmb stays with spark.

All the various stats under the C an G keys change when I hit X, but the mouse buttons do not.

Am I missing something? Is there a way to have the various buttons (mouse and keyboard) change when switching weapon sets?

Thanks