I swab every shot and have had this problem with my TC too. The patch and jag combo is too thick, so it is pushing fouling down into the patent breech and blocking the flame channel.
Most just pop a cap after swabbing, while pointing the barrel at a leaf or dry patch they've put on the ground. Patch or leaf should move, indicating the flame channel is clear.
Another method was mentioned above, pour in powder charge and seat a wad or similar on it, then swab.
My favorite is to get the jag and cleaning patch fit correct. I read about it on here. Chuck jag into a cordless drill and turn it down by holding it on a flat file. You want the jag/patch combo to go down the bore smoothly, but then the patch bunches when you reverse directions and it pulls the fouling out of the barrel. Works like a charm, I do not even pop a cap after swabbing anymore, I don't need to. I don't have fail-to-fire's anymore and so far I haven't lost a patch of the jag either.
I did buy another .54 cal jag to leave "normal" sized, so I get a tighter fit when doing a full cleaning.
I use a 2" size cleaning swab, my .54 jag had to be turned down until it was about the size of a .52 cal jag. I believe you could just buy a .52 cal jag solely for swabbing and it would work just fine.