If I’m using wads I always go to a larger than bore wad. I have a set of punches from 1/8” up to 1” and typical use 1/8” thick hard felt from Duro-Felt. For example, 3/8” wad for 32 caliber, 1/2” wad for 45 caliber, 9/16” for 50, 5/8” for 58...... In my experience the slightly larger wad like Walkingeagle suggests seals the bore better and will usually improve accuracy.Try the .54 wads next time and see if things are even more impressive!
From my experience, a lot of truth in your observation. Did some testing years ago and found you could deform the nose of a conical and not lose accuracy, but deform the base and you had flyers.I use an over powder card wad to prevent deformation of flat based conicals. It makes a difference.