I agree with you.
The spinning ball only has one thing acting on it that will reduce the speed it is spinning at as it flys downrange.
There will be some reduction in the rotational speed because of the drag from the wind reacting against any surface irregularities but if the ball is fairly smooth the effect will be slight.
As for this "stability" thing people keep talking about it really doesn't exist with a spherical object like a ball.
If we were speaking of an elongated bullet then yes, stability is a factor. That's because of something known as "critical speed" which all rotating shafts have and a elongated bullet isn't much more than a shaft.
These critical speeds can cause a shaft to oscillate about their axis to such an extent that the nose of the bullet can end up pointed in any direction.
Getting back to a spherical ball, if we aren't worried about the speed the ball is spinning to provide stability they why does the spinning ball shoot more accurately?
To answer this we must consider what is happening to the ball as it flies downrange.
We need something to overcome one of the main forces acting on the ball.
That force is the force of the wind, acting on the irregularities on the balls surface.
If we shot a ball out of a smoothbore so it wasn't spinning at all, the force of momentum will try to keep the ball flying straight.
The wind blast hitting a irregularity will try to deflect the ball away from its initial line of flight.
For a while, the momentum will win but eventually (about 30+ yards or less) the wind blast will begin to succeed and the ball will start to move off course. It could be up,down, right or left or any combination of these directions.
At first it isn't very noticeable but by the time the ball has traveled 60 yards or more it will be quite obvious the ball is no longer traveling at the point it was aimed at.
Now, if we spin the ball about an axis that is pointed in the direction the ball is traveling we have something else happening. Something good.
The wind will still try to deflect the ball but because it is spinning, the deflection force will constantly be pointed in a different direction.
For those who are having a difficult time imagining this lets say the ball has an imaginary 12 o'clock position marked on it and when the 12 o'clock position is straight up the wind wants to blow the ball to the right.
As soon has the wind started to succeed the ball has rotated so the 12 is now at the 1 o'clock position so now the wind is trying to blow the ball to the right and slightly downward.
Milliseconds later the ball has rotated so the 12 is now at the 3 o'clock position so the wind is now trying to blow the ball down.
This continues with the wind trying to blow the ball first right, then right plus down, then down, then slightly to the left and down, then to the left, then to the left and up, then up and so forth.
The end result of all of this is the ball is not deflected in any direction. It continues to fly along the arc toward the aim point on the target.
It doesn't take a whole lot of rotational speed to accomplish this which is why a roundball shooting gun can use rates of twist like 1 in 6 feet or slower and the ball will still fly true.
Now, getting back to the little .32, typically they are shot from barrels with fairly fast twist rates. They also typically have high velocities (black powder speaking).
Like I said, the wind blast will try to slow down the rotational speed but I think that at ranges out to 100 yards the ball will still be spinning fast enough to overcome the deflection the wind is trying to create.
Now, cross winds is another story.
The little, light weight .32 ball doesn't have a lot of mass and it loses speed rapidly because of the poor aerodynamics of the shape.
This rapidly removes the balls momentum so even a small breeze or gust will easily blow the ball off course.