I also wonder what patch thickness you're using?
I have a .54 Renegade too.
My groups really tightened up when I went from .015" patches to .018" pillow ticking patches. With a .530 RB.
It is also very picky about lube quantity, it likes barely enough on the patch to just coat the entire thing to the edges. Mink oil from Track of the Wolf has given me the best groups to date, although an olive oil / bees wax mix I made works pretty good too. I apply the lube to each patch by hand by just swiping the patch across the lube in a tin and then rubbing it in with my thumbs. Make sure it is clear out to the edges, but use barely enough lube to do so.
Light lube and an .018" patch requires a very firm smack on a short starter to get started, and pushing it down the bore in short 6"-8" strokes on the ram rod, but groups are too much improved to ignore the effort needed.
My Renegade shot patched round ball very poorly with 60 grains of 3f too. Groups really tightened up between 75-80 grains for me.
Mine also loves a barely fouled barrel. I can get two shots without swabbing before groups open up pretty drastically. Knowing that, plus the tight fitting PRB combo, has me just swabbing every shot, one pass down and back with a damp swab patch and then one pass with a dry. If at the range and firing again right away I often don't bother with a dry swab.