The problems are myriad.
First, almost all of the advantages of this superpower enable unsavory behavior. With regard to women, there is absolutely no good that can come from being invisible (unless you are married and trying to avoid work). At worst, well, I think the temptation would be strong to do something completely evil. With regard to business, the temptation would be to break the law in so many ways to gain advantages. I suppose you could become a CIA agent or something, and use the invisibility to spy on enemies, but that's not really that fun.
Second, what about clothes? Unless you walk around naked, people are going to notice a pair of jeans and a polo shirt walking around.
Third, assuming you are naked, you will freeze your cajones off, particularly if you are undertaking the CIA stuff and presumably spending a decent amount of time in colder climes.
Fourth. Anyone that knew you had the power wouldn't like you. Because they'd think you were creeping around all of the time.