Difficult one. I'd say that as it's your day, missy, you invite who you want. You might not have everyone turn up, for whatever reason, but at least you'll know that you sent out invitations to those you really wanted there.
However, we went to a wedding some time long ago, where the bride-to-be'd insisted on no children attending, as she didn't want them running round and being noisy in the church. Some people were very offended by this, and refused to go. Later, when the bride became a mum herself, she realised with horror, how terrible her request must've sounded. A different case to yours, but unless you explain your reasons to your guests who have young ones, it could cause resentment.