Those who favor retaining monuments to the Confederacy often say that removing them has the effect of erasing history, and that they honor regional heritage and important figures with virtues as well as faults. Of course, a huge amount of that history and heritage is slavery and racism, and especially in the South statues and monuments dedicated to the Confederacy greatly outnumber those honoring the foes of slavery and the champions of Civil Rights. (See this post from a week ago.)
That said, it's important to remember that by no means all defenders of these memorials are racists. In fact, even the most blatant racists and most enthusiastic defenders of Southern heritage, acknowledge the evil of slavery. Even those who try to paint slavery as relatively benign condemn it as an inherent evil. Indeed, attitudes toward slavery changed so rapidly after the Civil War that by the late 1800s Southern apologists were insisting that slavery wasn't the cause of the Civil War and that it would have gone away in short order in any case (as Lincoln had hoped it might in the Lincoln-Douglas debates).
It's also reasonable to consider soberly what should be done with historical monuments and statuary -- which should be removed immediately, which be retained, and which perhaps modified in some way. The Associated Press recently published an article based on interviews with historians that's worth reading. I won't attempt to summarize it here because it's not that long to start with.
I suspect most of us would agree that there's no single simple answer that applies to all cases. At one extreme is the New Orleans monument to the Battle of Liberty Place, originally erected in 1891 to commemorate the violent overthrow of the elected state government by a paramilitary gang calling itself the "Democratic White League." This was soon reversed by federal troops sent in by President Grant. It's hard to object to getting rid of a monument meant to honor an explicitly racist attempted act of treason (though in fact quite a few did object). Andrew Vanacore wrote good summary of the monument's history that can be read here. An inscription added in 1932 said that the battle had been a victory for white supremacy, but in 1974 a plaque was added saying that the monument no longer reflected modern attitudes, and in 1993 it was removed from its prominent place on Canal Street to a more obscure location next to a parking garage. In April of this year it was taken out of public view entirely. Given the monument's clearly stated purpose, it seems hard to object to that, though of course many did.
On the other hand, almost no one calls for demolishing the Jefferson Memorial. Jefferson was a slave-owner who never freed the majority of his slaves, not even in his will (as had George Washington), but he also wrote against slavery, including in his original draft of the Declaration of Independence, and he proposed what became the Northwest Ordinance that among other things forbade the expansion of slavery into what later became the states of Ohio, Indiana, Illinois, Michigan, Wisconsin, and Minnesota.