Cue balls are often larger than the other balls. This is for those pay-to-play tables where the balls are held till you insert more money. The larger cue ball isn't trapped and comes back in case you make a scratch shot.
You have a good chance of it fracturing and coming out in pieces, and oddly shaped pieces of wood at a good velocity do some pretty wild things. Including travelling in a half circle...., personal experience.
Curious... I served with a unit that had 2" Mountain Rifles, and a Mountain Howitzer, but the Mountain Howitzer wasn't rifled, since it was meant to lob shells at the target, not direct fire (unless the infantry was darn close ; then you used grape or canister ) They were lots of fun, since being small they made a huge bang compared to their much larger cousins, as back in those days our powder charge had a uniform max setting for all guns on the field. OH and the rules said that "full sized" cannon got more powder than 1/2 sized replicas, and ours were "full sized" but as "mountain" guns they were smaller than standard field pieces..., use to make the guys with the 1/2 size replicas upset since we'd get more powder than they would, even though at 1/2 scale they were still bigger than us.
If you're trying for accuracy at a distance, a hardwood ball probably doesn't have enough mass while having all the crappy roundball ballistic coefficient that all spheres have. So it's going to slow down waaay fast, or you'll have to be so close that it's kind of a moot point.
The guys down here often find a soda, or energy drink can, fill it with concrete, and shoot those, BUT these guys are shooting smoothbores. That might work but I don't know if the aluminum sides would protect the rifling in your piece.
I'd make a two piece mold from concrete that uses sand as the aggregate material, and use the cue ball as the "pattern". Once hardened, remove the cue ball, and line the interior portion that molds the ball with Flex Seal. Enough of a layer of that stuff, and it will then fit your gun AND release the molded balls easy from the mold.
Then use that mold to cast concrete balls. To protect the rifling I'd patch with chamois cloth or..., use spray-on flex seal or dip the balls in liquid flex seal.
I fired the cannon yesterday with the 2" wood ball patched. It penetrated a 2x4 and broke it in half. Put a sizable dent on the concrete wall behind the 2x4. The wood ball only had powder burns on it and was otherwise not damaged.