you primarily face 1 style of offense all season vs. 2 types of option, smash mouth, spread and air-raid (or a version of it). That's really hard to try to defend 5 different styles of offense alternating week in and out throughout the season.
Not to mention, the 7 or 8 other teams in a conference are much weaker than what we play.
Granted in a regular season we'd also add back some of the games we had to give up but it's still way easier in a conference.