The notion that he "wanted nothing to do" with Willingham isn't supported by facts.
Willingham was the 1st candidate White really approached (allegedly after Gruden was nixed by Malloy and there is little indication that Gruden was a White initiative anyway), both from the fact that he thought he was a good fit and he knew him from the PAC 10( White had tried to hire Ernie Kent, another PAC 10 connection,for bball before hiring Brey ). Willingham (according to Willingham himself) was ambivalent and more or less passed. White then went to O'Leary. After O'Leary's firing, White went back to Willingham who regretted that he had passed the first time and had been persuaded by outside forces to be open to the job.
White had his strengths -- he was the first truly professional AD at ND -- but he was in over his head re: big time college athletics and it showed when he tried to hire Meyer(Meyer badly played him) and how he he was manipulated by Weis -- that contract is all on White, nobody else.
Willingham was fired because he was an inept recruiter as well as other obvious failings as a football coach and it was learned that he was secretly negotiating with Washington. White understandably didn't like it more because it usurped his position more than anything else. He wasn't close to Willingham.