I realise you may already be familiar with synesthesia,
woodelf, but if you're not, I think this Wikipedia article will interest you:
http://en.wikipedia.org/wiki/Synesthesia
Not that one would have to be a synesthete to understand your question, I'm not, and I still see what you're getting at, it just doesn't really evoke anything in me. My mother was a synesthete but nobody knew it until quite late in her life, including her - she thought it was the same for everyone and so never mentioned it until one day she said something casual about Tuesdays being yellow... with a hint of red in the middle... that sort of thing.
I discovered she was really talking about the word Tuesday, the word Wednesday, and so on, not the days themselves. I wrote down her descriptions, then asked her again six months later and guess what, the colours had changed, which I found even more interesting than the phenomenon as such. But I digress.