You have to start forming your decision from your own moral standpoint, and work out where you stand there.
If you decide not to serve, and you are happy that your decision sits well with your personal morality, then that is where your decision begins and ends.
You have no control over what people think of you, only what effect, if any, that has on you.
You are not obliged to go through life justifying every important decision you make to every single person you meet.
Discuss it with your loved ones, and make sure they understand the reasons for choice, which ever way you decide to proceed.
After that, you make your decision and live with it, and if you have made the right decision for you, then you will have no trouble doing that for the rest of your life.
If you make any important decisions based on what other people may or may not think of you, you will never know a day's happiness.
Think it over, make your choice, live with it.