I used to suffer badly when I was younger but seem to have grown out of it mostly. Still struggle a bit with a lot of cardio and when I have a bad cold but nowhere near as bad as it was.
I know what you mean, I can swim and row but struggle with other things. Can only do a slow breaststroke though, front crawl just turns me into a wheezing mess (tried to learn it for my bronze medallion lifesaving course but ended up having an asthma attack in the water and my own casualty and the lifeguard had to rescue me, needless to say I failed!).
I used to find classes great. I find I can control my breathing a lot better when I can pace myself in things like dancing and step. On step you can control the level by the height of the step and you can up your level of effort depending on how you feel.
I had one rather memorable boxercise class once where, after the whole mistake of not wearing a sportsbra and nearly knocking myself out, I didn't dare leave the changing rooms for about 15 mins after as i was verging on purple.
You could try other dance classes too. I have been to salsa ones in a local club a few times for a lesson then some freestyle (just get on the dancefloor and make it up) and it is definitely good cardio but, again, you can control your effort level.