We did evolve as higher primates living in tribes and they were constantly warring with one another over territory. Maybe there are some irrational aspects to the way that we ALL (no matter what race, or culture or faith) interact with those with appear different.