Try using a thicker patch, first. You don't indicate the lube you are using. Go to .017 and .020" and see if that doesn't tighten those groups at 50 yards.
Then, try increasing your powder charge in 5 grain increments. Shoot at least 3 shot groups at 50 yds, but preferrably 5 shot groups. I don't know what powder you are using. If its not real Black Powder, you may be simply wasting your time. Some people report good accuracy with the subs; Many do not find them consistent. Is this Pyrodex "P" powder you are referencing? Why would you be shooting a pistol powder in a .54 cal. rifle? Just curious. Try using Goex( Wano, Schuetzen, Graf& sons) FFg powder, or Swiss 1 1/2 powder in that gun. I think you will be much happier with it. Off a rest, and with good eyes, you should be able to put all those shots in one hole.