You need to have a serious talk with him - preferably with a Relate counsellor with both of you.
You have issues as a couple - your lack of a sex life, broken communications, mistrust, his use of pornography, and it really would benefit from profesional help.
Your husband is obviously unhappy, and is seeking to bolster his image as an attractive man by fantasising to strangers, which may well be harmful, but why take the risk?
i am pretty sure a counsellor will speak to you both individually, and if they do, be totally honest about everything in your marriage, however embarassing and painful it may feel - remember these are professional people, they will not judge you but they will help you both as a couple.
You are not being petty, but you need to get this sorted out before the two of you drift into permanent relationship decay.