Let’s unpack Lee’s so-called distaste for the institution of slavery.
Lee was a “progressive” slaveholder, which doesn’t mean what you might expect. While Lee saw slavery as regrettable, his misgivings revolved around slavery’s inefficiency and messiness rather than out of any benevolence toward “the black race.” He also absolved himself of any responsibility of the institution’s existence, believing that divine providence had ordained slavery as natural. And he never challenged the notion that the white race was superior to the black.
I’ll let him explain: “the relation of master and slave, controlled by human laws, and influenced by Christianity and enlightened public sentiment, [is] the best that can exist between the white and black races.” 
In Lee’s oft-cited 1856 letter, so frequently used to whitewash his legacy because it refers to slavery as a “moral and political evil,” his objection to slavery was that the institution was more injurious to whites than blacks, who were “immeasurably better off here than in Africa.” He never wavered in his belief that the wholesale emancipation of black slaves would be a violation of god’s will. In Lee’s view, the end of slavery would come not at the meddling of abolitionists, but when god deemed it time. As Lee stated, “let us leave the progress as well as the results in the hands of Him who chooses to work by slow influences, and with whom a thousand years are but as a single day.” If Lee had his way, emancipation would have been a long way off. And though Lee did indeed manumit the slaves of his father-in-law, he did so out of legal obligations as the executor of his father-in-law’s will and not out of any recognition of slaves’ humanity or rights. In fact, after the war, he opposed blacks’ enfranchisement and civil rights and refused to condemn lynchings and other forms of violence against black Americans. 
As historian Elizabeth Pryor summed it up, Lee “could embrace the need for justice, but it was a justice defined by unjust principles. His racism and his limited imagination meant that he never admitted the humanity of the slaves with whom he lived. In avoiding that truth, he bound himself to slavery’s inhumanity.” 
This is why, just one month after Lee’s death in 1870, Frederick Douglass fought against his lionization as a Southern saint:
“We can scarcely take up a paper that comes to us from the South that is not filled with nauseating flatteries of the late Robert E. Lee … It would seem from this that the soldier who kills the most men in battle, even in a bad cause, is the greatest Christian… [It is reported that Lee] died being sadly depressed at the condition of the country, that he could stand it no longer. From which we are to infer, that the liberation of four millions of slaves and their elevation to manhood, and to the enjoyment of their civil and political rights, was more than he could stand, and so he died!” 
Granted, Lee was a complex man, one who held notions about slavery seen as “progressive” by white Southern society at the time, and he was a general who could have potentially fought for the North if circumstances had been otherwise. But they weren’t, and Virginia decided to secede from the Union over slavery. As W.E.B DuBois rightfully underscored in 1928, “Robert E. Lee led a bloody war to perpetuate slavery. …Lee followed Virginia. He followed Virginia not because he particularly loved slavery (although he certainly did not hate it), but because he did not have the moral courage to stand against his family and his clan.” 
Confederate statues perpetuate the darkest legacy of Lee, namely the exploitation and intimidation of black people and the elevation of a deeply racist belief in their inherent, divinely-ordained inferiority. Such statues were primarily constructed during two periods–1900 to 1930 and the 1960s–when black empowerment threatened white supremacy, at a time that witnessed first the resurgence and then the burgeoning of hate groups like the KKK. Such statues were built not to remember Lee, but to intimidate black southerners and convey white dominion. 
I’ll end with a recommendation that comes to us from DuBois, who wrote at a time when, four years after the construction of the Lee statue in Charlottesville in 1924, American society faced an eerily similar dilemma: “Today we can best perpetuate [Lee’s] memory and his nobler traits not by falsifying his moral debacle, but by explaining it to the young white south. What Lee did in 1861, other Lees are doing in 1928 [and today in 2017]. They lack the moral courage to stand up for justice to the Negro because of the overwhelming public opinion of their social environment. Their fathers in the past have condoned lynching and mob violence, just as today they acquiesce in the disfranchisement of educated and worthy black citizens, provide wretchedly inadequate public schools for Negro children and endorse a public treatment of sickness, poverty and crime which disgraces civilization.”
No historian would say that we should erase our nation’s history. To the contrary, it is imperative that we remember and learn from our history, to ensure that we don’t perpetuate the inequities and prejudices of our forebears. To that end, I propose we move Confederate statues out of our parks and statehouses and into museums, where historians can meaningfully explain the racism and violence such monuments embody. Not doing so would be to willingly obscure any lessons we might learn from Lee’s mistakes and our nation’s past misfortunes.
 Lee’s letter to his wife on slavery, December 27, 1856
 Lee’s private letters, date 1865, cited in Elizabeth Brown Pryor, Reading the Man: A Portrait of Robert E. Lee Through His Private Letters. (New York: Penguin Books, 2007), 151; Interview with “Eric Foner: White Nationalists, Neo-Confederates, and Donald Trump.” The Nation, August 16, 2017.
 Pryor, Reading the Man, 154.
 Frederick Douglass, “Bombast.” New National Era. November 10, 1870. Accessed on August 8, 2017 at http://chroniclingamerica.loc.gov/lccn/sn84026753/1870-11-10/ed-1/seq-2/
 W.E.B. DuBois, “Robert E. Lee,” Crisis, 35 (March 1928): 97.
 Interview with Eric Foner.
 DuBois, “Robert E. Lee.