I think reactions have been influenced by thousands of years of gender stereotyping, where the men went out and did the hunting or earned the money and the women stayed at home, tended the fires, did the cooking and looked after the children.
The world has now moved on, yet many peoples' attitudes are still stuck in a timewarp. I honestly don't think it matters which partner stays at home and which partner works. As long as the children have a loving and secure home, that is the only issue which should be influencing you. Running a home and looking after a small child is not an easy option. If you and your wife are happy that your decision will work, other peoples' options are irrelevant. Good luck to you both.