I read the entire thing. He made it extremely clear he was talking about people who weren't part of the neo-nazis or white supremacists. Here's what he said before the "very fine people" remark:

"But not all of those people were neo-Nazis, believe me. Not all of those people were white supremacists by any stretch. Those people were also there because they wanted to protest the taking down of a statue of Robert E. Lee."

His "very fine people remark" went: "They didn’t put themselves -- and you had some very bad people in that group, but you also had people that were very fine people, on both sides. You had people in that group. Excuse me, excuse me. I saw the same pictures as you did. You had people in that group that were there to protest the taking down of, to them, a very, very important statue and the renaming of a park from Robert E. Lee to another name."

He then went on to say, "I’m not talking about the neo-Nazis and the white nationalists -- because they should be condemned totally. But you had many people in that group other than neo-Nazis and white nationalists. Okay? And the press has treated them absolutely unfairly." A reporter, whose IQ must be about room temperature, asked, "Sir, I just didn’t understand what you were saying. You were saying the press has treated white nationalists unfairly? I just don’t understand what you were saying." Duh...just duh.

You say there were no other people there besides neo-nazis and white supremacists. That's irrelevant to Trump's remarks, because he clearly thought ordinary people not wanting to see the statues removed were there, and he was clearly talking about them.