Well, with iron sights, you should have the front sight in focus and the target and rear sight should be fuzzy. It's impossible for your eye to focus on both at the same time. Some are able to shoot irons with both eyes open. I am only able to do that with shotguns. The theory behind it is the front sight is what gives you control, and you know where the muzzle is pointing. If you focus on the target your sights can wander because you are not actually looking at them. Shotgun shooting is much different. You focus on the target because it is moving, and your brain is "supposed" to automatically calculate where you need to point the gun. That is why having a proper fitting shotgun is so important.
Proper fit for rifles is also important, because it will relieve your body from having to make large muscular adjustments. By making those types of adjustments, you fatigue the muscles and induce shake and unsteadiness. Proper cast, length of pull and drop at heel are all very important. That's why there are so many fancy adjustable stocks for modern firearms.
Also, finding your "natural point of aim" is highly suggested for target shooting. When you aim at the target, close your eyes and then open them again. More than likely you will no longer be on target. Adjust your body position so as to correct and get back on target. You don't want to move the rifle. Scoot your butt left or right, move your off hand or trigger hand elbows forward or rearward or left or right to help bring the rifle back on target. The first time you do this it might take several adjustments. It gets easier with practice. Once you can close your eyes and open them again and the rifle stays on target you have found your natural point of aim, and your shots will be much more consistent. (Edit: That is if you are shooting from a shooting bench. If you are shooting from the prone, all adjustments should be made by moving your hips left/right/forward/rearward.)
From my experience, well built BP's should be able to group much closer than 5" at 50 yds. You might also have to adjust how much powder you use, as well as patch thickness and what lube you are using. Experiment with different patch material and thickness and lube to find what your rifle likes better.
Sometimes thicker patches work, sometimes a .495 ball will work better than a .490 ball. What patch, ball, and lube combo are you using? How much powder?