If it's a combined system Mojay, I can see this happening. Ideally, extractors should have their own sole outlet. I guess your block has them all connected to a single common duct. No problem when the fans are running, but when they're not, residual air in the system (from neighbours) can hang around.
A possibility is that the outlet isn't properly protected from wind and downdraught ........... rather like chimney smoke coming back into a room.
All guesswork I'm afraid, without knowing the layout.
I would guess a simple solution would be to fit a kind of non-return valve in your section. Is yours a simple 110mm pipe with a square grill?