Dumpster-fire prior 3 HCs. No explanation needed on how bad the prior coaches were, but it's rare to have a team alternating losing seasons all of a sudden become elite. Turnarounds usually happen at schools with mediocre competition. ND developed a reputation as a way overrated program, and that stink takes time to lift in recruiting.
Scheduling. The schedule for ND is the schedule of an elite team. Even when the competition is down, ND is the only school besides Hawaii that regularly leaves their home time zone. That makes return to glory that much tougher. And as we can see this year, and the past few years as a half-ACC member, it is much easier to win the games you're supposed to win when it's verses conference competition. Familiarity is good for top programs.
Laughable facilities. Barely Power-5 level, much less elite. This was not fixed immediately. Couple that with real academic standards (I won't call them tough - you can be pretty mediocre academically and get into ND as a football player, but compared to other revenue teams at P5 schools, there is a huge talent pool that is simply off-limits to ND), and you again have off-limits top recruits.
Lots of bad fans (see Nebraska, Georgia games, although red is admittedly a color that stands out). ND stadium should be one of the toughest to play in the country. Until rather recently, with changes that many old farts on this board hate (d), it was like playing at a golf course. There still is a large "down in front" contingent, but the atmosphere is so much better now.
I reject your thesis he "took so long to get it right." 2012 was an undefeated season. The NC game was an embarrassment, but still. Teams were in the hunt until the disastrous 2016 season, if always a cut below the elites. Brian has steadily built the program, and evolved his approach. Is he less rah-rah and more of a politician than say a Lou Holtz? Sure. But I don't believe a 2010's Lou equivalent would have taken the 2010 ND job, and would not have lasted this long. That politician approach helped save his job when things weren't going well, which helped save the program. Go back to the message boards in 2015/2016 and look at the names being thrown around to take over. What have they done since then, and what has Brian done?