Just to give an extreme example... When one of those people you let in to "sink or swim" inadvertently kills your son or daughter in a robbery gone wrong when they are sinking rather than swimming, you may wish that "we" hadn't left it up to brutal capitalism.
My original point is that as we progress as a nation, regardless of immigration policy, the minimum standard of living should at least mirror in growth the upper ends of society, that hasn't happened recently (last 20 years?) It's obscene to have Billionaires while also having homeless people and malnourished children with shitty schools growing up in a ghetto of gangs, drugs and crime who are more likely to get shot than to graduate high school.
I am okay with an upper limit of prosperity if it means we raise the lower limit of squalor, and I think that it would be better for everyone whether directly or indirectly (like my example above).