Once you go down this road, removing monuments that offend people, few will be off-limits. The folks who have driven this effort are adherents to a religion that weaves a tale of the original sin of light-skinned folks and it can more than ensnare any Euro-American of note over the past 300 years. The folks behind this are devoid of the ability to create nuanced thoughts or place things in historical perspective, with their oversimplification of Lee a prime example. These are also folks accustomed to getting their way. Once you indulge them, they will demand more. Only a fool would believe that they will stop with Confederate symbols.
The ultimate destination here will be statues of themselves, which is, to a large extent, what this is all about.