do you mean out for a pee/poo and in again or outside and stop out there? I hope you mean the first and not the second!
I don't think that 2 growls in 5 years is a big deal but if it was me I'd get his collar and lead and put that on him to take him outside, slipping the lead off once he's in the garden. Once he gets a the idea that growling gets him nowhere (don't make a big deal of it, just ignore and put the collar lead on) and that you will wait outside with him till he goes AND that he gets a treat after he has been (NEVER before) then he should give in and behave.