That's not what Lou said.

Lou told him that if he was uncertain about the gig, he shouldn't take it, because Notre Dame requires an unwavering commitment.

Davie is the one that told him it was too hard to win there and that it was a "different" place than when he was coaching there on his staff.