Several things come to mind that might help you with your Charles Moore pistol.
Here's a link to your pistol
http://www.davide-pedersoli.com/tipologia-prodotti.asp/l_en/idt_18/pistols-charles-moore.html
Do NOT remove your rear sight.
Your pistols sights sound like they are set up for a "target hold" or view when the gun is properly aimed.
This method of sighting requires aiming the guns sights at the bottom of the bullseye, not at the center.
The proper view with sights like this has the top of the front sight even with the top of the rear sight and centered in the notch. The black area of the target will be floating above the top of the rear sight, just touching it.
While many, including myself prefer the sights to be aimed exactly at the point where the impact is desired many target shooters prefer this "6 o'clock hold".
According to the Pedersoli web site your pistol has a "Single Set" trigger. Are you using it in the set condition?
When used in the set condition it takes very little effort to pull it back to make the gun fire. This helps you avoid moving the gun off target when you fire the gun.
The set trigger has another useful feature.
When it is set and then pulled it will make an audible "click" sound.
Of course you won't hear the "click" when the gun is loaded but you can hear it when the gun is not loaded.
If your pistol has this set trigger you "set" it with the guns hammer either in the fired position or in the "full cock" position by pushing the trigger forward until it latches.
Then by pulling it to the rear you will hear the click and the hammer will fall.
NEVER set and release your guns set trigger when the hammer is at "half cock" position. Doing this can break the locks sear or the tumbler.
Anyway, you can use this set trigger to improve your shooting by first, making absolutly sure your gun is unloaded.
Make sure the hammer is in the "fired" position and set the trigger to the set condition.
Position your body so that a line from shoulder to shoulder would be at about 45 degrees from the target.
Grasp the guns grip with moderate pressure. The gun when it is fired is not going to jump out of your hand. (Yes, I know it is unloaded now but this is important to remember when your actually shooting it.)
Align the sights with the imaginary target on the wall aiming at the bottom of it as I mentioned before.
At this point I must tell you something about precision shooting with a pistol. The sights will weave back and forth across the bottom of the target. This is natural and it is impossible to make it stop.
The thing to try to do is to minimize the movement while watching it.
When the sights approach the centered position lightly squeeze just your trigger finger until you hear the "click". Do NOT squeeze your whole hand when pulling the trigger. Doing so will move the gun off of the point of aim.
As long as the hammer is in the "fired" or down position you can repeatedly set and release your set triggers without harming the gun so practice using this method as often as you can.
It will greatly improve your shooting.