Just one small brandy? No, it can't be the cause of bad health. However, if you've got very high blood pressure your doctor might advise you for a few weeks to cut out all alcohol.
NO WAY can it be bad for you! I have a small brandy every half-hour every evening and some afternoons as well. Sometime it goes on into the next morning and look at me. Just because I am ga-ga, it has nothing to do with the brandy.