1. The demographics of the local community.
For example, I live in a small town of around 5,000 people which has some large employers (including national and international firms) represented on the big trading estate on the edge of town. The bosses of those companies (who might be located hundred or thousands of miles from here, in very different communities) might aim to see 'non-white' employees well-represented in their workforce. However, given that there are only a handful people of Asian or African origin around here (and in neighbouring towns as well), it's almost inevitable that the majority of businesses on the trading estate will have an 'all white' workforce ;
2. Difficulties in monitoring the diversity (or lack of it) within the workplace. An employer can quickly see how many 'non-white' employees he has, simply by walking around the shop floor (and, hopefully, by checking that there's similar representation at management level and in the boardroom). But he can't know how many gay employees he has working for him unless he asks intrusive personal questions.
Chris