We let younger people play up. Maybe the women could swim with the men if they want, but the men shouldn't swim with the women?
I don't think drugs should factor into it. If the reproductive organs are present, then the person is in sport what they are in biology. If the person has a true disorder (e.g., born with extra chromosomes), then perhaps a ruling by a medical body can be made as to the true sex. But, that is so rare, it will likely not present a problem.
That way, grown adults don't have to walk around pretending that someone else's fantasy is their own reality. We indulge children when they don't perceive reality correctly, but we don't reinforce them, and we don't require adults to indulge other adults whose mental perception does not match reality (in this case, biological reality)...at least, we didn't used to do that.