I agree with the others about your patch/ball combination being a tad too tight for a dirty barrel. It's typical for a tight fit to be hard to load just as you're describing, tough for the first few inches, then easy to the breech as the lead ball conforms to the shape of the barrel. It gets more noticeable as the barrel gets fouled.
Here's a few things I would do to fix the situation:
First, I would swab between shots to see if it made a difference (I don't think you mentioned anything about how often you ran a damp patch down the barrel). If the problem hasn't existed with a clean barrel, it makes sense to clean the barrel more often. I'm thinking this is your biggest problem.
Second, I would try adding a "sloppier" lube to your patches, such as olive oil or neetsfoot oil. Bore butter is pretty stiff stuff, and doesn't help all that much with a cruddy bore.
Third, try a thinner patch material, but I'm thinking after you try the first two suggestions, you may be happy with the results and stick with the tighter patch. But it doesn't hurt to try different materials anyway, just to see what has the best accuracy and ease of loading. It's all about learning.
Hope this helps, Bill