If he truly loved you, he wouldn't hesitate to be with you. I don't believe he stays with his wife just for the sake of his daughter, but he's hardly likely to tell his lover all the wonderful, if mundane aspects of life he enjoys and probably cherishes with his wife.
Married life doesn't always provide the excitement that people crave, but it does provide the safe haven that we all need. At the moment he is getting the best of both worlds and I would suggest you take responsibility and walk away and find someone who wants to make you their number one girl.
As for your last comment about hurting him or his wife, I would advise against it. If you upset his wife, you upset his daughter. Is that really the type of person you are?