I'm fairly busty so always wear a bra, and find the thought of going braless scary in case the go all saggy when I am old.
However, I do make the most of the fact that they are still pointing in the right direction and will go bra-less if the top I am wearing means a bra doesn't look right (I don't like seeing bra straps unless they are dainty pretty ones, and seeing as my bras look like hammocks that's not usually the case).
As far as health goes, I find that if I wear a supported vest top (without a bra obv) I get awful back ache.