I think that people aren't aware of the ramifications of what would have happened if Joseph did have relations with Mary. People think in such simple terms: if Joseph married Mary, then they had sex! Hur hur! Let's assume the Jews accepted Jesus wholeheartedly as savior:
Jews: Wow! Jesus, you really are the Son of God, the promised Messiah!
Jesus: Yep.
Jews: Your parents Mary and Joseph must be very proud!
Jesus: Well, just as a point of theology, I was conceived by the Holy Spirit as was foretold in the prophets. My mother was a virgin when I was born. Joseph isn't my biological father.
Jews: WOW! Praise our Lord! A miracle and a fulfillment of the prophets! Your mother was a virgin when you were conceived? Wow!
Jesus: Yep.
Jews: But...wait a sec. Your mother, Mary, had other children, right?
Jesus: Yep.
Jews: So...you're saying that Joseph had sex with Mary?
Jesus: Well, yeah.
Jews: So...you're saying that Joseph had sex with the womb that carried the Messiah to earth, the womb which carried our Lord, the very same virgin womb that was indwelt by the Holy Spirit itself?
Jesus: Yeah, that's what I'm saying. Oh, there's Joseph now. Maybe you should go talk to him.
Jews: *riot*
I find it very hard to believe that such a devout Jewish man who was visited by an angel and also had enough honor to marry Mary even though she conceived by the Holy Spirit would then turn around and - not to be crass - have "sloppy seconds" with the woman who carried the Messiah. No Jew who understood the Messianic prophesies would act in such a way.