Monk Not Gaining Power Shards When Downscaled

A few days ago I noticed that when I play with my lower level friends, who have downscaling on, my monk will rarely, if at all, generate a power shard. When my friend portals to town, my power shards start generating normally again, or if they turn off downscaling, my power shards return. I'm level 80, he's level 54. I'm assuming it's downscaling me to to lvl 54-ish. When I went through solo at level 54, I was generating power shards.

Also, there are times when I'm playing solo where I stop generating power shards. If I switch to my weapon 2 and back, it will reset and my power shards come back. Initially, I had no weapon in slot 2, but equipped one as I read a possible fix was the weapon swap. None of my skills are set to weapon 1 or 2 exclusively.

I'm playing on Xbox series X