You may run in to issues of the whoosh-bang more frequently if you put lots of priming powder in the pan which creates something of a fuse effect because the TH is buried under the priming powder. A way to deal with that is to file your pan just a little bit deeper. That would let you use the same amount of powder. But, as others have said, if it still works, it may not be worth the aggravation of moving it.
My suspicion is the culprit that made it turn out this way was that the drill bit "walked" on you when you were drilling it. (Out of curiosity, did you use a drill press or a hand drill?). I always start with a very small hole (like 1/32") when I start these with the bit choked up VERY short and close to the chuck which gives it less opportunity to walk on me. Then, one more intermediate size that I can nuance the hole a little bit away from the direction any walking occurred. It won't fix it the whole way, but it can get it fixed some. Then the final size hole, again with a little bit of corrective action (if needed). To have this level of bit / hole control though you really need an x-y axis vice on your drill press though.
The other hole that's something of a "pucker factor" hole is centering the tang bolt. If it's off center slightly it won't affect performance, but it will always bug you since it's right in front of your nose every time you mount the gun.
