Remember it's not just the lock ignition time that is in play here, but the time from sear trip to projectile exit.
In Pletch's tests a cap lock ball exited the muzzle around .035 seconds after sear trip, and a rock gun exited around .085 seconds. And that was from a 9"-10" pistol. Assuming an exit velocity (from the pistol) of 700 fps, we're going to have some additional barrel dwell time for a projectile from a rifle. So if there is 36" more barrel, and an exit velocity from a rifle of 1700 fps, a simple calculation (just an average of the additional 1000 fps) tells you there will be an additional .006-.008 seconds that will elapse. Not much additional time, but now we're getting up to around .100 seconds, which is getting close. Also, don't forget the gas coming out the touch hole is going to be exerting some lateral force to try and move the breech (and by extension the muzzle) on you too. It won't move it much, but it doesn't have to move it much to throw your shot off a fair amount at some distance down range.
Same thing with barrel recoil. he instant the ball starts moving forward the barrel is going to want to move backwards. The greater the drop at the butt off the bore center, (as well as the greater the charge / pressure / recoil) the greater the tangent force to want to make the muzzle rise. That also applies to the pressure or volume of gas coming out the touch hole (and exerted lateral force).