Xenophobia includes an irrational dislike and fear of those from other nations. It is clearly not rational to dislike immigration by certain people simply because they come from a place you think is undesirable. That’s what he was saying and that’s why it’s xenophobic.