Ask him. He's your long-term partner, he should be honest with you. The issue of whether or not either of you need fertility support after two years of trying does now need to be addressed - and it might well be him, he needs to take the test. On the practical front - surely if some girl thought he was the father of her child, she'd have had a DNA test done and would be chasing him for child support now? You need to sit down with him and ask him about this, and if he thought it was his child, why he's not involved with it.
It does seem odd to me that you kept questioning him at the beginning of your relationship about earlier children - what made you think he had?