One of my cats is really nervous when travelling and started weeing in his carrier. The Cattery suggested giving him 'rescue remedy' (you can buy it from Boots -its one of the Bach flower remedies) Its meant for humans, but theres a lot of info if you google it for animals too.
When he went to the vets recently I gave him a few drops beforehand and he was calmer, so what Im getting at is that you give him a few drops before any men come into your house and it may help calm him and 'unlearn' his agressive behaviour.
Otherwise, it may be castration for him (pity we cant do it with all agressive males - lol)
Try
http://www.rainbowcrystal.com/bach/floweran.ht ml