I have seen a lot of 'fit' people as described by you in my classes over the years. Strangely enough they seem less 'healthy' than others who don't do so much exercise, always off with heavy colds, chest infections. I mean the over enthusiastic types, fitness fanatics, etc.
I have read research that shows that excessive exercise can affect the immune system.
Maybe it's a genetic tendency, some people seem to go from one cold to another. I can't remember having a proper cold for 15 years, just very lucky - but then I can't remember my mother ever having one!!