To answer your question, I am rarely in situation in which I feel downright uncomfortable. I had a bum come after my wife one time--he physically grabbed her--that was about a year ago. I've had some kids (18 or so) terrorizing other pedestrians for kicks...that was an evening in the city, a few months ago. (They stayed away from me and the wife, though, and went after easier prey.) It happens, but only when I go "into the city." I live in a safe area, which is why I don't carry. (I have the training; I just never got my permit.)
I don't think I have a right to be comfortable. I don't look at it as an imposition if I have to stay alert in certain places or certain crowds. I just do. I figure it is my duty to be alert, especially if my family is with me. I do try to be in places where the risk is low for conflict, though.