The patch is too thin. My advice: Go to a .530 ball and a thicker patch.
1. Rifles with rounded grooves need thicker patches. My .54 caliber barrel with rounded grooves was machined by Mr. Hoyt. That rifle is capable of making five bullet holes touch at 50 yards. My patches are made from drill cloth; don't know the thickness, but it works very well.
2. A patch lube that cleans the grooves without shoving crud into the patent breech and flame channel is needed. When i used black powder the best patch lube i found was the original Go Jo hand cleaner without grit, wiped on the patch at the firing range. It's good for swabbing between shots, if needed.
3. Many years ago i stopped using black powder. Black MZ and JSG leave little residue in the bore, but like a tight fitting ball set hard on the powder. Depending on climatic conditions, there may be an accumulation of residue in about the last five inches of the bore that makes the ball hard to start after our or five rounds. When this happens i swab about six inches of the bore near the muzzle.
You missed the OPs cal... it is 50 cal. You are on the right track, for thicker patch, just your suggested ball size will not work.