This is one of my more rambling posts, it's a bit more of a train of thought really. Long story short: I don't think that taking statues down is about rewriting history or erasing it.
* * * * * * *
Statues are far more about symbolism than history. If you get rid of the symbol you aren't actually getting rid of the history, you're just saying that you don't especially want to make a public display of it in the same way as before.
As was pointed out elsewhere, most statues at the centre of the current US debate were erected in around the 1920s, and almost certainly to try and hammer home a political message rather than to remember the people in the statues. As was also pointed out, Robert E Lee himself disliked the idea of monuments to the confederacy being erected:
"I think it wiser, moreover, not to keep open the sores of war, but to follow the example of those nations who endeavored to obliterate the marks of civil strife, and to commit to oblivion the feelings it engendered."
But clearly a line has to be drawn somewhere; one aspect of history that I think some people seem determined to ignore one way or another is that nobody can ever be perfect, which leads to past figures being either idolised or demonised depending on what point you are trying to make. As you say, everyone can find something distasteful in a given figure. Churchill is one of my personal heroes, but was more than a little bit racist at times. Gandhi, too, I admire for his approach to trying to bring about Indian Independence, but was also undoubtedly a little racist himself. Presumably, you can (and do!) find people determined to remove statues of both of them, although I am absolutely not one of them.
In the end, the point is that statues are not about history; they are about politics. Here is someone we admire, and we are going to tell the world so! There is no room for subtlety there. If you want that, then you go to the history books, where you can understand the person, and all their faults, properly.
At any rate, it's something that deserves a proper discussion: how best to remember history? I don't like the idea of looking for an excuse to tearing any given statue down, but I don't think you should dismiss the idea entirely. A statue is too powerful a symbol in many cases to be ignorant of what else it could stand for. And, as I say, you ought to look at the context behind when they were erected in the first place.