I would say that there are two main reasons why women tend to wear identifying clerical garb more than men (and yes, it's very much a reality here too). One is what you've identified. And I would say it goes further; I have noticed that when I wear a collar I am less likely to be ignored, dismissed, belittled or patronised inside the church, even when the fact of my ordination is well known to all present. It's as if it's a visual reminder to people that "Oh yeah, she's not just a kid, she's the vicar." Maybe this will change with age; I'll let you know in twenty years or so...
The other reason is the sexual abuse crisis. Many of my male colleagues won't wear a collar to walk down the street because they're wary of being harrassed or assaulted by people who equate "man in clerical dress" with paedophile, which is distressingly common. Women, by and large, don't have that problem to the same degree. (I have had people assume that I would cover up abuse or be an abuse apologist because I'm clergy, but I have not yet had people assume that I am myself an abuser because I'm clergy; and I have not suffered confrontation in public spaces to anything like the same degree).
I find, outside the church, wearing the collar is a big benefit to my ministry. It makes me identifiable, and therefore available. I have had people approach me on the street and ask me to pray with them, approach me in the supermarket and ask me for material aid, see me on the train and strike up conversations about God. The collar is a great way for me to open that door because it isn't pushy, people can recognise my role and choose the level at which they want to interact. But by making myself visible people do choose to approach me, and I've had many really precious interactions because of that.
I have never had the kind of freebie Jack mentions, though. I think that might have been more true here a generation ago.
Some of my colleagues avoid the collar more because they think it sends unfortunate messages, reinforces clericalism, etc. We each have to find our own personal style on this stuff, but I think a lot of it comes down to who you are and how you interact with people, much more than just what you wear. And like everything, there's a time and a place. There's a joke about clergy who are so unhealthily invested in their role that they have clerical collars in their pyjamas (not an actual thing, by the way - at least as far as I know. Suddenly I'm afraid to google!)
Oh, and for the record, clerical collars aren't medieval. They were invented by a Presbyterian in the 19th century. Prior to that, the primary form of distinctive clerical dress was the cassock.