Sneezy:
8mm equals .3150 inches. 5/16 equals .3125 inches.
The difference is .0025 which might be a better fit or it might not. It depends on where the thread is in it's allowed tolerance band.
a 1 mm pitch equals 25.400 threads per inch.
For every full turn of the nipple, the metric threads want the nipple to move .0394 inches.
For every turn of the existing 24 threads/inch they want the nipple to move .0417 inches.
The difference being .0023 per revolution.
If the nipple has 6 full threads on it and one multiplies the difference between the metric threads and the inch threads they will find that the difference is .0138 inches (which is about a 1/2 thread difference).
In other words, if the threaded portion of the nipple is 1/4 inch long, screwing it all the way in will create a situation where about 1/2 of a full threads material will have to "give".
This may give the illusion of tightening but in actuality the threads are expending a lot of their strength simply fighting each other.
This is not the best idea for something that is containing over 4000 PSI of breech pressure.
I can't say I would recommend using the metric threaded nipple in your gun.
When you say, "The 5/16 X 24 are slightly undersize..." if they still will tighten down using moderate torque replacing your nipple with the correctly sized 5/16-24 will most likely be stronger than trying to use the metric nipple.
The reason for this is that although the threads are worn a bit and loose, at least ALL of the threads are working together to resist the pressure in the breach.
With the metric nipple stuck in your gun, only one thread will be taking the load while the rest of the threads will be fighting with each other for space.
If the threads are so worn that they won't resist the pressures of firing the gun, this is the only time I will suggest that you have the threads replaced with a helicoil thread insert.
Have this work done by a good gunsmith and remind him that the new insert may have to be modified to allow the flame from the cap to travel unrestricted thru the flame channel (hole) to the main powder charge.