I think you are making too much of this. It is just friendly banter, and as other people besides yourself heard all this, and the fact that he is not exactly hiding anything from you, as you were there when he bought and took the coffee to her, he is not hiding anything is he?
Take it all in good fun, which seems the best thing to do in this instance, otherwise you could end up sounded like a bitter jealous woman.
then it could be the case of flump 2's answer.
Do you really want that?
:-)