I'm sure they don't ALL hate you being there by any means and he is your father so you are entitled to be there. Fatherhood is not, it may surprise some people to learn, bestowed by genetics, it's who is a father to a child, and he has been yours. I daresay, having a very dysfunctional family myself, that it's merely one or two people wondering why you are there and not ALL of his family by any means. You go where you like with the man you consider to be your father, you are in the right and i'm suer no-one hates you as has clealry been suggested to you, so relax and enjoy your time with your Dad.