He claimed, and replays showed, he was making a greek omega symbol, which was common at the time. No one knows if the official would have called a penalty for that gesture, but officials were pretty quick to call those kind of celebrations back then.
Couple side notes; The unsportsmanlike conduct call was on the 2 point conversion which caused the kickoff to be moved back 15 yards. On Michigan's following drive ND also received a late hit penalty. Both of those big penalties aided Michigan in driving for the winning score. And Bob Davie unnecessarily burned a timeout before the 2 pt conversion, which would have been crucial on that last drive. You just absolutely don't use a timeout on a 2 pt conversion in that circumstance.