To be fair - While the bible never explicitely says Mary stayed a virgin... IT also never explicitely says she had other children. Like it or not Chris - the words used could be interpreted any number of ways. There's no cut and dry answer here except for those who trust their tradition. For the rest of us it's only speculation.
Well, I can tell you a theory:
Joseph had some sons before to marry Mary: two of them are well know: James 'the brother' of the Lord and a Jude.
After the death/ascension of Jesus these two groups came in fight:
- the group of Jesus relatives, headed by James 'the brother of the Lord', that was more tied to the Jewish tradition and had the supremacy in Jerusalem.
- the other group, the 11 apostels headed by Peter and Paul, that was loosing power in Jerusalem and had to spread the gospel out from such a town.
We had same traces of that in the attacks of the Gospels to the 'brothers of Jesus' (Lk 8:21), or by the fact that Peter in Act 12 do not go personally to James before to escape in Antioch or by even the fear of Paul to go to Jerusalem
On the other hand, we know that at the death of James 'the brother of the Lord', the (Jew)-Christians of Jerusalem elected bishop a very relative of Jesus, a son of a brother of Joseph.
Of course Luke in the Acts try to shown the eatly christians as united, so whichver was the truth, we will never know it.