They need to simply nerf this. How?
Just simply make it so that if you wish to switch weapons, the weapon you first held has to finish "firing."
So basically, the user has to have the firing animation from the wand finish in order for their switch to be available. In this case, the magic orb...