I had brachytherapy for prostate cancer after a shortened course of external beam radiotherapy 7 years ago at the age of 52.
The external beam bit was a nuisance (60 mile round trips for half an hour or so in the hospital). I still try not to eat too many fatty or spicy foods, don't overdo the tea and alcohol and always have some Diocalm or something similar handy.
The brachy op was straightforward - in one day, treated the next day, home on the third day (the lead lined room was a bit of overkill, but it was quiet !).
My PSA levels were a bit slower than expected coming down, but other than that the only problems I have had is an occasional trace of blood in my urine - the iodine seeds can move a bit afterwards and cause a bit of minor bleeding - and minor incontinence (the hospital labelled it "urgency"). The incontinence is easily dealt with by having an empty water bottle in the car at all times and making sure I know where the toilets are wherever I am or by keeping an eye open for bushes if we are out in the countryside.
I did sleep in a separate bed from my wife for a few months afterwards, but only because she is 10 years younger than me. I didn't stop giving my daughters hugs when they were needed - that kind of brief contact is OK, and they were both past the stage of lap sitting anyway.
As for the impotence - well things don't work quite as well as they used to but everything does still function, and being sterile we never needed to worry about contraception afterwards.