I think 100 grains is too much powder, but your rifle will tell you, for sure. Back that off to 75 or 80 grains of FFFg powder, or 90 grains of FFg for a hunting load, and use an OP wad. Use a chronograph to check velocity and Standard Deviation of Velocity for the load. The combination of the .535" and .018 patch sounds tight to me, but then its your gun. Without knowing the land to land diameter, and then the groove diameter, its hard to know what you are using, or equate it to anything I have shot. Lubes make a difference, too. You haven't shared with us what lube you are using.
Yes, cleaning between shots makes a difference at 100 yards and beyond. Ask the target shooters. They want the barrel to be as clean for the last shot as it was for the first. If crud is allowed to build up between shots, and you load the PRB to a mark, the powder chamber ( the area behind the PRB) is shortened, and you have to be seeing higher chamber pressures, and more eratic velocities with subsequent shots in an uncleaned barrel.
The only way to know what is happening with your gun is to test fire it over a chronograph, and on a target. Using a bench rest, to eliminate as much human error as possible, fire 3-5 shot groups using all the same components. In one test group, clean the barrel as you have been doing. In the second, clean it thoroughly. See what difference you have in group sizes at the longer ranges. Do the test more than once, switching around which group is fired first. That will even out issues relating to fatigue, slight changes in loading procedures over the day, lighting and wind conditions. The readings on chronograph will tell you if your loading procedures and techniques are causing wide diffences in Velocity. And, ALWAYS read your spent patches. Collect them after each shot RELIGIOUSLY. The patch tells you as much as the chronograph will about what your powder charge is doing in the barrel.
Best wishes. :thumbsup: