Your short-barreled .50 caliber gun is just not going to move mountains. Its maximum amount of powder to burn behind a PRB is only 55 grains. Start 10% less, or at 50 grains, and see how that shoots. Then go down to 45, and then up to 55 grains.
This short barrel makes your gun basically a 50-75 yard deer gun. Its fine when you hunt river bottoms, and thick woods and edge areas, which is where deer live. Most whitetails are shot at under 50 yards, year in and year out. Don't believe half the stories you hear guys tell about making long shots. Where i hunt, you rarely can see a deer 50 yards away! At those ranges, the limited powder charge you can put in your gun is more than adequate to kill a deer. But, --- and this is a big but ---- this is no 100 yard deer gun. The distance between the front and rear sight would make it difficult to shoot a good group at those longer ranges. 50 grains should give you about 1300 fps, which is more than enough to drive a 175 grain lead ball clean through a deer broadside.
You can safely load much more powder than this maximum burn rate formula, but all you are doing is making a bigger muzzle flame, as the powder burns outside the barrel. The added weight of the powder adds to the recoil forces you feel, and that in turn will produce a flinch, and distract you from holding your sight picture through the shot to make a good hit. ( lack of follow through is the nice way to say, " you jerked your shot!")
I shot a 25 inch barrel .45 caliber " Kentucky Carbine " made in Spain for a number of years. It was its own challenge, but I learned much on how to make a percussion lock work correctly, and then learned what loads to use in it.
In your .50 you can use FFFg BP, which will give you higher velocity, between 10 and 15% on the average, than using a similar volume of FFg powder. I used FFFg powder in my .45 and eventually found a load combination that worked well enough that I was doing exhibition shooting with the gun.
If you decided to use FFFg power, start at 45 grains, and work your way up. Try to find a load that give you the best accuracy at 50 yards. With a .50 caliber round ball, placement on an animal's body is all important. Between accuracy and a bit more velocity, go with the accurate load every time.
You can do your own penetration tests with your " light load" and your " monster load" the one your buddies are telling you GOTTA USE when you go hunting. The penetration test will tell you which load to choose, and why.