First of all for offhand shooting, you absolutely have to establish your natural point of aim as described above. Because we spend a fair amount of time in the reloading process of muzzle loading, I often check for this while firing a string of shots, before I fire a shot. Also, if we load at a bench or off the firing line, then it is critically important to re-establish your natural point of aim EVERY TIME you go back to the firing line to shoot.
I wonder if you may be "canting" the rifle either on the bench or when you shoot offhand, but not in the other position? "Canting" in this case means you turn the rifle slightly, either to the right or left, so the sights are not straight up and down.
It is also quite possible that you may not be centering the front sight in the center of the rear as well in the offhand position as the bench position and/or that is just the way your eyesight works.
With my astigmatism, I long ago learned I have to really crowd right up close to the rear sight to get the most consistent sight picture/sight alignment. Of course on many rifles and shotguns, that means I get smacked in the mouth on every shot fired. I long ago figured out I would put up with that to shoot the best I can.
I mention this because not everyone can do everything "exactly right" and still shoot the smallest groups or the most accurately they can.
One year at a Division Match, we were using M9 pistols. Now, we were not allowed to file down or weld up the front sights to correct for elevation, but we were allowed to move the rear sights to the right or left. One of the other experienced NM Armorers asked if I could help with a shooter that the NM Armorer could not seem to adjust the windage on the rear sight to suit him. I asked the shooter to bring me his target to see exactly where he was hitting and twice corrected the rear sight, but to no avail.
Finally I told him I wanted to go the line and watch him shoot. Good Grief, he REALLY jerked that trigger like no other shooter I had ever witnessed in years on firing lines, but to my amazement it was THE most controlled and uniform Jerking of the trigger every time he fired, so he fired very small groups.
OK, so that put me/him in a dilemma, because we only had about 6 more practice days before they fired for record in the Match and that was not enough time to correct everything he was not doing right, as others had already tried to work with him.
So I followed him down range after firing one string to paste up his target, which is extremely unusual for an Armorer to do. When he got done pasting up his target, I asked him for the roll of black "pasties" that are used to cover the holes in the target. I looked carefully at how high and to the right his groups were, then put a square of four pasties that much down and to the left of the center of the target. Then I told him to aim at that square of pasties on the next string of fire.
Sure enough his groups were well centered in the bullseye and if he shot that well on Record Day, he was bound to "Medal" in the Match. That is not common for a "First Year Shooter." He was amazed his group had centered so well.
I explained to him the way he was jerking the trigger, caused him to "pull" his shots high and to the right, but there was no way I could mechanically center the sights on his pistol to put him in the center of the bullseye. So what I recommended was to put a square of pasties on each target, before he fired on the target. It worked on record day and he got a very low Bronze place, but he did medal.
I joked with the members of THE Marine Corps Pistol Team that they would have a lot of work to get him to stop Jerking the Heck out of the trigger at the Marine Corps Matches and on the Summer Team if he got that far, but at least he could shoot a very tight group.
Gus