From what I understand, he was actually given an option to return in 2005, if he would just fire the inept Bill Diedrich, but refused to do so.
We would have been in a much worse position for that class of 2006 had Willingham done his usual recruiting job.