I have all but a 40 cal, so need to get one of those.
I am a club shooter, I think the 40 is an excellent choice. I have several other 40s. 40 is easy to manage and load. Working up loads has been easy for my 40s.
The 32 is a fickle and fiddley thing to work with. Count on more work to find the best load. The 32 is difficult for old hands and cold weather. Precut patches are frustrating. You will have to make a patch cutter. You will have to make a short starter too. I am not aware of either being sold. The balls only weigh 44 grains. My 32 uses about 35 grains for best accuracy.
The 36 is a nice club gun. It shoots accurately. It is thrifty.
If you want to shoot a deer with it get the 45. The 45 is very light weight and balances very nicely off hand.
No matter what you decide you will love it.