When I went to university, for about a year there was a (horrible) street musician on the only sensible pedestrian route from the train station to the campus. Every single day, this guy decided he had the right to annoy me and hundreds/thousands of others with his tunes and his begging, ruining fine mornings with mangled renditions of whatever song could coax a few cents from the dumb minority who appreciate accordion music at 8 AM. And that just involved horrible music. Someone telling me that I'm going to hell every morning is much more than "just being an unwanted listener for a few seconds". It ruins the mood, it ruins the atmosphere, it makes the society a crappy place to live for everyone except the one guy who really feels a need to shout at strangers.
I really like the dutch word for society, "samenleving", or literally something like "together-living". It indicates that the idea is to live together, that the thing only works if you keep each other's needs, wants, and desires in mind. Actions that are only beneficial to one person and detrimental to everyone else (and no, some madman raving on the streets is no way beneficial towards establishing "thoughtful dialogue" or anything of that kind..) are not beneficial to the society, and thus don't belong in public.