In my rather astounding level of experience of nearly completing one rifle I would have to vote that my least favorite thing is cleaning up and polishing cast brass parts. Seems never ending.
As for what gives me the most trouble.... Well, the list is long and distinguished. I seem to spend 1/4 of my time on a particular operation and the other 3/4 fixing my mistakes.
Butt plate comes to mind as mentioned previously...
When the specs say 120-150 hours to complete construction, I have come to understand that they mean "provided you don't bugger up every operation".
for me, getting started. Once I start I can work like an obsessed Madman. Right now, I am sort of overwhelmed at the start of rebuilding my shop. Some of the expensive machines were salvageable or temporarily stored in a shed. All the taps, drill bits, drill press, stock vice, stock carver etc gone and to be replaced. Figure I will get them as I need them to do particular tasks. Two were machines/devices that I made myself. Means making them all over again.
entry pipes on hawken rifles and buttplates. Nothing a whole lot of patience can't fix. I find myself really procrastinating on drilling the holes for the pins for things also....If we didn't love doing it we wouldn't be building them though would we?
Or if the hawken style forend cap is off from the factory it will really mess you up till you figure out that's what it is...... Last 2 or 3 days I spent staring at how the heck did I get the underrib off that much and the barrel is straight with the lines on the cap.....Then it hit me like How did I not see that earlier...