If your rifle is a Whitworth type, with hexagonal bore, it was intended for fairly heavy hexagonal bullets (in the 480 - 500 grain range)shot with moderately heavy charges.
The Whitworth can do fairly well with cylindrical bullets, but they must be cast quite soft, as they need to 'bump-up' to fill the bore to the corners, which measure ca. .490" across - that's a lot of bumping for a cylindrical bullet which can't be much larger than .451". This can be done with heavy bullets, which have considerable inertia, and full powder charges, but is not likely to work with the lighter cylindrical bullets unless charges are increased significantly (if they work at all).
In addition, you will need to use some type of wad or wads under the bullet - my friend who has and shoots a P-H Whitworth says he finds that a fiber wad, similar to a shotgun filler wad, of about 3/8" thickness works best, but, again, only with the heavy cylindrical bullets cast of pure lead and full charges.
If, on the other hand, your rifle has conventional-style rifling, you can use standard cylindrical bullets of lesser weight with reduced charges and still expect good results - my P-H Volunteer does well with a variety of bullet types sized to .451", though it, too, requires a wad under the bullet for best results.
Good luck, and let us know what results you achieve.
mhb - Mike