Losing late in the season is almost always penalized big by the polls. Especially for a Group of Five team. The committee has also said repeatedly that they consider how good they think a team is at the current moment. It is not at all unusual for a team to be ranked ahead of another team who they lost to. It happens all the time.
Do I think it likely? No. Do I believe it's possible? Yes.