If it's actually about comfort level, then does it really matter? It's pretty clear when dealing with most modern 'gendered' issues, perception is reality. This goes for all involved, or potentially involved. The toll for using the women's bathroom if you're biologically a man is dealing with women's sadly often justified fear that biological men in women's restrooms set off some alarm bells because men rape women. None of this was ever a worry for women when biologically born men stuck to men's bathrooms, was it? So perhaps these are the birth pangs of society's increasing acceptance of transgender issues and people, in a similar way to how a lot of people (myself included) can't help but notice how different life is now re: homosexuality versus how it was just a few years ago, when DOMA was still in affect and nobody would dream that they could get into legal trouble for not issuing marriage licenses to homosexual couples. That quite simply wasn't a thing at the time. And today, transgender people are in that transition from being marginalized to gaining mainstream acceptance. That's bound to make people uncomfortable, particularly since the biologically-rooted male/female sex binary is...well, rooted in biology. So I imagine it would be hard to convince many people that a lack of increase in rapes in gender neutral bathrooms would mean that they're safe using one if they don't feel safe using one.