Yes, it is milliseconds difference but each additional lever movement is added time to the overall firing sequence and time on target is measured in milliseconds as well.
Target shooting teaches this lesson better than any other method I know as when one learns to call their shot, that is take a mental picture in your mind of the sight position in relation to the target at the time of trigger break, and you know if the shot was good or not before ever looking though a spotting scope.
I got so I could score the target very closely without the spotting scope just from remembering how the sights lined up at the shot break.
No one holds perfectly still and has what is known as a cone of dispersion in how well they can stay on target. The trick is to take up trigger pressure and matching up the break at the same time as the sights align.
The pattern one develops in the cone of dispersion give you and indication of when alignment is coming up and the subconscious trigger take up will intersect for a good shot.
Practice is the only remedy.
This is the reason firing sequence time is just as, if not more important, than a super light trigger pull.
I use both simple and set triggers on various match rifles and my best scores have always been shot with light breaking, crisp, simple triggers that are very fast.
On my match .54 Hawkin style rifle I removed the rear set trigger, made a modified lever and moved the firing trigger to the rear of the finger bow and installed a trigger over travel screw. The lock time is faster, trigger pull is very crisp breaking at roughly 1.5 lbs and is my most accurate muzzle loader.