He is trying to guilt-trip you. Had you had asked him to go on the trip with you at the time of booking, he most likely would have declined as hc said. Now he's trying to make you feel guilty for not asking him. You have your own life to live, as does he, and if he wants to be miserable then that's his choice, but he has no right to make your life miserable as well. Just keep visiting him as you do, but as for anything else keep that for you and you partner. You deserve your happiness and it seems that you're not getting that from your dad.