You poor thing, and it doesn't help to get no support from your boyfriend either. You should explain to your boyfriend that you were wearing perfectly normal clothes, and that maybe he should be concerned for your welfare here, really than taking the opportunity to have a dig at you. If I were you I would also mention it to the police. You can say that you may be being paranoid, but you are being harassed by a man and it has happened on a number of occasions. They at least will give you advice. You could just ask them what to do. They will have logged it as a complaint so if anything was to happen, you will have done something about it already.
Regarding the man, try being a bit more obvious. Say in a clear, firm voice "I am not interested, please leave me alone." If he persists go back to the police. I think that just by the way you walk or hold yourself you can give off body language to be more assertive. You could also get yourself a mace spray, just incase. Remember, it is your right to walk down the street and feel safe and not harassed. Act APPALLED if he comes up to you again, and do not be afraid to sound like a cross teacher!!x