It is possible for individuals to compartmentalise their lives to such a degree that they can enjoy a happy settled family life, and an outside relationship, with neither impacting on the other.
It's possible - but so is a Lottery win - and the odds on this are worse!
What you have to factor in is that for this to work, you need two people who enter into the relationship on the basis that it will remain at this level of commitment from start to finish, never changing on either side.
And right away, you have problems, because human emotions are involved, and they shift and vary and twist and turn day to day, sometimes minute to minute.
If you can see him for sex - and that means meeting in a hotel room and leaving straight afterwards, no conversation at all, then that is the only way this can work.
If you are emotionally involved with this man - and you clearly are - then there is pain in it somewhere for innocent people, as well as yourself and the man involved.
Be realistic, be adult, you are reaching for something that is simply not there, and it is going to end with people getting hurt.
I utterly understand how the power of attraction can sweep you along, gainsaying everything you thought was the bedrock of your moral compass, but this cannot and will not last - we are simply not programmed that way. as Woody Allen said in 'Play It Again Sam' - "If I'd kept up the level of charm, I'd have had a heart attack!" Humourous, but true.
Some people refer to affairs as 'the best of both worlds' - but as you are finding out, it is often the worst of both worlds.
Give this man up before anyone else gets hurt, or you get hurt any more. This is not love, and the pain you feel will go in time, especially if you put your energies into finding a man who can love you properly, because he devotes his time and energy to you, and doesn't give you the bits left over when his family have had their (rightful) share.
You know this is the best thing, you just need the courage to finish it, and keep it finished.
Ring him today,