If you consider each conference as its own bracket, then only the champion should go. So what if on paper you might be the second-best team in the country? Plenty of pro teams in stronger conferences/divisions get left out every year when they're better than a team from a weaker conference/division. Yes, this isn't pro sports, but the principle is the same.