You don't remove the lugs, why remove the sights? To each their own, I suppose.
YMMV, but my preference is to install sights and sight in the rifle while "in the white"; then brown or blue with everything on the barrel. Never had a problem with the finish bleeding somewhere, and no risk of dinging or scratching the brown/blue while drifting or filing sights.