Tyrone Willingham was someone who took very few active steps to even maintain the program, and certainly didn't take any steps to improve it.
Look at 2004 and 2005's recruiting classes, which were the two absolutely worst classes of all time. If it weren't for Charlie Weis pulling off a last second snag of a few more recruits, 2005's class would have been as bad, or worse as 2004.
In that 2004 class, I can count maybe 6 players who made at least a meaningful contribution. That's it. Normally, you would expect at least 10+ to make some sort of contribution.
Then look at how terrible the offensive line recruiting was under Willingham during those last two years... It was no surprise when 2007 rolled around, that we had virtually zero developed junior or senior talent ready to play. You simply can't even hang in there with teams at their prime.
Charlie Weis at least helped take the right steps to right the ship that was sinking, leaving Brian Kelly with a much better stocked cupboard than what Willingham left him.