Being the good school teacher that he is, EngTeach, given the opportunity would tell you that it's only been within the last decade or two that the racist view of the 'Stars 'n Bars' has become prevalent in our society. Before that, it was seen as what it probably is... an historical symbol of a time past... genteel ladies, frock coated men sipping mint juleps in the shade of the magnolia trees visiting with the returned Major of Cavalry of Virginia's 10th after the Battle of Chickamuga... Somehow, (and i understand parts of the mechanism) the flag and other historical do-dads have come to symbolize something dark and sinister.
Slavery was practised in the Northern States as well and many of the slave owners were black... but that was overcome as was our nation's prediliction for the sad and disastrous practise... following England, I might mention, by several years...