I'd guess the employer's view would be that if you were too sick to work you'd be too sick to go on holiday. That would be irrespective of the nature of the illness or the destination. Your only argument would be if the trip was part of the treatment or convalescence.
I'd suggest you should have cancelled the holiday on medical grounds and claimed on your holiday insurance.
You should also have informed the employer that you would be unable to take the annual leave because you were sick. You would then be credited with those days when you returned to work.
Perhaps you could explain to your employer that you weren't clear about what you had to do because of the stress of the operation?
Anyway, hope your operation went well and you're on the mend now.