Laying aside all of the he said/she said quotes from people who were born long after the fact, lets stick strictly with the scriptures that was written by those who were eye witnesses, who lived "on site" with Mary, Joseph and Jesus.
That said, correct me if I'm wrong, but it seems that everything boils down to the interpretation of the Greek word "Adelphos", which is translated "brothers" our English Bibles.
According to Thayer's Greek Lexicon Adelphos means:
a brother (whether born of the same two parents, or only of the same father or the same mother):
Matthew 1:2;
Matthew 4:18, and often. That 'the brethren of Jesus,'
Matthew 12:46, 47 (but WH only in marginal reading); f;
Mark 6:3 (in the last two passages also sisters);
Luke 8:19;
John 2:12;
John 7:3;
Acts 1:14;
Galatians 1:19;
1 Corinthians 9:5, are neither sons of Joseph by a wife married before Mary (which is the account in the Apocryphal Gospels (cf. Thilo, Cod. Apocr. N. T. i. 362f)), nor cousins, the children of Alphaeus or Cleophas (i. e. Clopas) and Mary a sister of the mother of Jesus (the current opinion among the doctors of the church since Jerome and Augustine (cf. Lightfoot's Commentary on Galatians, diss. ii.)), according to that use of language by which ἀδελφός like the Hebrew אָח denotes any blood-relation or kinsman (
Genesis 14:16;
1 Samuel 20:29;
2 Kings 10:13;
1 Chronicles 23:2, etc.), but own brothers, born after Jesus, is clear principally from
Matthew 1:25 (only in R G);
Luke 2:7 where, had Mary borne no other children after Jesus, instead of υἱόν πρωτότοκον, the expression υἱόν μονογενῆ would have been used, as well as from
Acts 1:14, cf.
John 7:5, where the Lord's brethren are distinguished from the apostles. See further on this point under Ἰάκωβος, 3. (Cf. B. D. under the word ; Andrews, Life of our Lord, pp. 104-116; Bib. Sacr. for 1864, pp. 855-869; for 1869, pp. 745-758; Laurent, N. T. Studien, pp. 153-193; McClellan, note on
Matthew 13:55.)
Thayer clearly makes a convincing argument that James, Jude...etc...etc are indeed the sons of Mary. Adelphos meaning either full blood brothers, or step-brothers there can be no other conclusion.
The argument that Joseph was married to someone else [too] has its own problems.
1. This is demeaning to Mary. Ask any women if being the only wife is more of an honor, or being one of many wives is more of an honor.
2. If these brothers of Jesus was indeed Joseph's sons, and not Mary's, then Joseph must have been the blood Father of Jesus, which we know is not the case. Jesus' Father, biologically is God. Therefore, "adelphos", would not have been used.
3. There is no scriptural evidence of this even in the slightest.
Bottom line: If there is anyone who thinks that "adelphos" can be clearly, and unequivocally applied to mean that Mary had no other children, then please show me some scriptures, using "adelphos" is this manner, where it does not imply blood relation from at least one bio parent. As you know, the only way Jesus can be blood related to anyone is through Mary. as Joseph wasn't his bio father, and therefore "adelphos" would not be used as Thayer stated.
This is not to mention the scriptures that clearly state that Joseph "knew" Mary after Jesus was born. Any scholar that denies that this means sex is in fact in denial himself. All throughout scripture when a man "knew" and woman (especially his wife) it means sex.
Anything else, in my opinion, is denial.