It's apparent that you both want different things from this friendship, and depending on your friend's willingness to accept your feelings, your friendship may or may not be saved.
You need to have an honest chat with him. Explain that you care for him as a friend, and that moving the basis of your relationship would mean you loosing that special bond, and that is something you want to keep.
He will interperet the full meaning for himself, without you having to spell out that you don't find him attractive in a sexual way.
Depending on his ability / willingness to accept your position, he will either carry on being your friend, or stop because he is hurt.
Either way, you need to make sure that everyone knows where they stand - if you value his friendship, you will respect him with your honesty. The rest is something you cannont control, but either way, you have done the right thing.