Hardly - especially as you can obtain chess sets in most of the colours of the rainbow these days. I believe original chess sets were carved from ebony and ivory - and ivory is yellow.
Back in 1984, when he was lead singer of Marillion, Fish used the expression 'Brixton Chess' to describe racial tensions in that borough, but other than that, I'd agree with Andy.
Have you ever tried to play chess with two white or two black sets of pieces? Give it a go.....come to think on it didn't bobby fisher cause a near riot at the world championships one year but trying to insist on playing on a green and yellow chequered board?