RE:Airbolt They can dazzle because they are very bright, hence them being effective in heavy fog. If they weren't very bright then they would just be normal tail lights.
Maybe you should stand behind your car with your fog light on and see how long it takes before you get a headache. We are not making assumptions, merely giving you an answer based on the info you have provided.
You say you rarely use it because of the number of flashes you get. This suggests that you have used them several times in the past and want to use it more than you do. I have never had to use mine and judging by the response of other drivers behind you strongly suggests you are not using it appropriately .
Rule 211 of the Highway Code:
211: You MUST NOT use front or rear fog lights unless visibility is seriously reduced (see Rule 201) as they dazzle other road users and can obscure your brake lights. You MUST switch them off when visibility improves.
http://www.highwaycode.gov.uk/21.htm