There isn't much biblical support for Mary to have given birth to any other child BUT Jesus. The support is more for the fact she only had one son: JESUS! And remained a virgin afterwards.
Under Mosaic Law, the term "first-born" was given to the first child regardless of if there were other siblings afterwards. It is the child who would be ritually cleansed (cf. Exodus 34:20). The "first-born" child is the one who opens the womb (cf. Exodus 13:2; Numbers 3:12). It is a title regardless of if there are other children or not, and thus, not exactly the most overwhelming argument "proving" Mary had other children after Jesus.
Interestingly enough, Sacred Tradition is one reason to believe in the perpetual virginity (certainly a grace bestowed by God, no? -- cf. 1 Cor. 7; Matt. 19:12), as well as mere LOGIC. It is essential to remember that the belief that Mary was ever-virgin in no way exalts her beyond any human status, in that it is a grace from Christ.
On February 2, 1546, Martin Luther wrote that Mary was "a virgin before the conception and birth, [and] she remained a virgin also at the birth and after it."
Zwigli wrote in January 1528 to "the holy Church in Zurich" that in his sermons and writings, "I recognize Mary as ever-virgin and holy."
Calvin held the same belief. In his "Commentary on Matthew" he called "pig-headed and stupid" anyone who believed or implied the Virgin Mary gave birth to other children, citing that "we have already said in another place that according to the custom of the Hebrews all relatives were called 'brethren.'"
These three men (who upheld their personal interpretations of the Bible, and theologies which contradicted each others', due to "Sola Scriptura") agreed from a BIBLICAL perspective that Mary was ever-virgin.
There is nothing in the Biblical texts to say that Mary had other children. For we already know ("Protoevangelium of James" aside -- that is HARDLY the basis of the belief in her perpetual virginity, mind you!) that nowhere is it said that the children who are brothers of the Lord are the children of Mary! Just "brothers of the Lord."
You already know that the term "brother" ("adelphoi") can be translated to mean "kinsmen" just as "brethren" in English has a broader definition than "brother." We don't have to elaborate on that speculation (cf. Luke 1:36; Luke 22:32; Acts 1:12-15, 7:26; 11:1; 13:15,38; 15:3,23,32; 28:17,21; Rom. 9:3; Gen. 11:26-28("anepsios") / Gen. 13:8; 14:14,16; Gen. 29:15; Deut. 23:7; 1 Chron. 15:5-18; Jer. 34:9; Neh. 5:7; 2 Sam. 1:26; 1 Kings 9:13, 20:32; 2 Kings 10:13-14; 1 Chron. 23:21-22; Neh. 4:14; 5:1,5,8,10,14; Tobit 5:11; Amos 1:9)
Hebrew and Aramaic have no word for "cousin" and this is carried over into the way the Greek is expressed.
What's more, opponents of the perpetual virginity of Mary also point to Matt. 1:25 - this verse says Joseph knew her "not until ("heos", in Greek)" she bore a son. Some Protestants argue that this proves Joseph had relations with Mary after she bore a son. This is an erroneous reading of the text because "not until" does not mean "did not...until after." "Heos" references the past, never the future. Instead, "not until" she bore a son means "not up to the point that" she bore a son. This confirms that Mary was a virgin when she bore Jesus (see also: Matt. 28:29; Luke 1:80; Luke 2:37, 20:43; 1 Cor. 15:25; 1 Tim. 4:13 ; Gen. 8:7, 28:15 ; Deut. 34:6 ; 2 Sam. 6:23; 1 Macc. 5:54; Revelation 2:24-25).
Since the ossuary of "James, the brother of Jesus, son of Joseph" has seemingly been found, a lot of people speculate it proves Mary had other children besides Jesus. Actually, if the relic is authentic (and really, there have been lots of fake relics out there - why are fundamentalists so quick to accept this one? After all, Jesus/Joshua, James and Joseph were very common names in the first century, and the only noteworthy thing about this relic is that it denotes a brother's name on it, which was not common.) it does not prove that Mary had other children, for as you yourself wrote, James could have been Joseph's child from a previous marriage (from which he was widowed).
Now, Church historian Eusebius wrote about "James, the Brother of the Lord":
"Then James, whom the ancients surnamed the Just on account of the excellence of his virtue, is recorded to have been the first to be made bishop of the church of Jerusalem. This James was called the brother of the Lord because HE WAS KNOWN AS A SON OF JOSEPH, and Joseph was supposed to be the father of Christ, because the Virgin, being betrothed to him, 'was found with child by the Holy Ghost before they came together,' as the account of the holy Gospels shows.
"But Clement in the sixth book of his Hypotyposes writes thus: 'For they say that Peter and James and John after the ascension of our Saviour, as if also preferred by our Lord, strove not after honor, but chose James the Just bishop of Jerusalem.'
"But the same writer, in the seventh book of the same work, relates also the following things concerning him: "The Lord after his resurrection imparted knowledge to James the Just and to John and Peter, and they imparted it to the rest of the apostles, and the rest of the apostles to the seventy, of whom Barnabas was one. But there were two Jameses: one called the Just, who was thrown from the pinnacle of the temple and was beaten to death with a club by a fuller, and another who was beheaded." Paul also makes mention of the same James the Just, where he writes, 'Other of the apostles saw I none, save James the Lord's brother.'"
(Eusebius of Caesarea - Church History, Book II)
According to the same chapter, Eusebius writes: "But Hegesippus, who lived immediately after the apostles, gives the most accurate account in the fifth book of his Memoirs. He writes as follows: 'James, the brother of the Lord, succeeded to the government of the Church in conjunction with the apostles. He has been called the Just by all from the time of our Saviour to the present day; for there were many that bore the name of James. He was holy from his mother's womb; and he drank no wine nor strong drink, nor did he eat flesh. No razor came upon his head; he did not anoint himself with oil, and he did not use the bath. He alone was permitted to enter into the holy place ; for he wore not woolen but linen garments. And he was in the habit of entering alone into the temple, and was frequently found upon his knees begging forgiveness for the people, so that his knees became hard like those of a camel, in consequence of his constantly bending them in his worship of God, and asking forgiveness for the people. Because of his exceeding great justice he was called the Just, and Oblias, which signifies in Greek, Bulwark of the people' and 'Justice,' in accordance with what the prophets declare concerning him. "
Notice we also see "he was holy FROM HIS MOTHER'S WOMB" - no mention it is the same womb in which Jesus took flesh!
Matthew 1:25 tells us that Joseph did not have relations with Mary before the birth of her Son. Why not? There was no ban on intercourse during pregnancy under the Jewish Law under normal circumstances, and if he was going to have intercourse with her after the pregnancy why not before?
It was because in Jewish culture this would have meant that Joseph acknowledged biological fatherhood over Jesus. Because he refused to do this he was saying in essence that he did not accept Jesus as his biological son though he would accept him as his legal son. By refusing to consummate his marriage with Mary he was in conformity with the Jewish Law. By not having intercourse with Mary before the birth of Jesus he is acknowledging in principle that he was also forbidden to have intercourse with her thereafter. A woman found to be with child that is not her husband’s is forbidden forever more to him and to the man who impregnated her. (In fact, a woman caught in adultery is likewise penalized.) Under Jewish law, because Mary was found to be with child before she had consummated her marriage to Joseph she was forever forbidden to him. He could keep her as his wife but he was not allowed intimate relations with her. Had Mary born any children after Jesus she would have been stoned to death under the Law. Had Joseph claimed those children to be biologically his, he would have been stoned to death also.
When Joseph found out that Mary was pregnant, he was going to give her a 'get' or paper of divorce from their betrothal. When he elected not to do so, she remained LEGALLY his betrothed. After they lived together she was considered his wife. But under Jewish law, since Jesus was not his biological son, Joseph was forbidden to have relations with Mary.
Here are some sites that speak to this further:
http://www.catholic.net/rcc/Periodicals/Homiletic/2001-10/orchard.html
http://www.udayton.edu/mary/resources/documents/RC.html
http://www.geocities.com/~eingedi/joseph.html
http://www.geocities.com/~eingedi/maryandjoseph.html
Theologian and Old Testament scholar Br. Anthony O****o wrote in his essay "The Perpetual Virginity of Mary":
"We also have to take into consideration that when Mary was told by the archangel Gabriel 'Behold, you shall conceive in your womb, and bring forth a Son, and you shall call His name Jesus' (Lk 1:31), he also added that this was to come about because 'The Holy Spirit will come upon you, and the power of the Most High will overshadow you; therefore the Holy one to be born shall be called the Son of God' (Lk 1:35).
"By stating it in those terms the archangel declared to Mary that God would enter into a marital relationship with her, causing her to conceive His Son in her womb, For 'to lay one's power (reshuth) over a woman' (Targum to Dt 21:4) was a euphemism for 'to have a marital relationship with her.'
"Likewise 'to overshadow' (Lk 1:35) by spreading the 'wing' or 'cloak' over a woman was another euphemism for marital relations. Thus, the rabbis commented (Midrash Genesis Rabbah 39.7; Midrash Ruth Rabbah 3.9) that Ruth was chaste in her wording when she asked Boaz to have marital relations with her by saying to him 'I am Ruth you handmaid, spread therefore your cloak ( literally, "wing": kanaph) over your handmaid for you are my next-of-kin' (Ruth 3:9).
"Tallith, another Aramaic-Hebrew word for cloak, is derived from tellal = shadow. Thus, 'to spread one's cloak (tallith) over a woman' means to cohabit with her (Kiddushin 18b, see also Mekhilta on Exodus 21:8). Did not the Lord say to His bride Israel: 'I am married to you' (Jr 3:14) and 'your Maker is your husband'? (Is 54-5:5; Jr 31:32)? And what is more intimate than what the Lord said to His bride: 'You developed, you grew, you came to full womanhood; your breasts became firm and your hair grew... you were naked... and I saw that you were now old enough for love so I spread my cloak over you... I gave you My oath, I entered into a covenant with you and you became Mine, says the Lord God' (Ezk 16:7, 8).
"Having been enlightened by an angel in a dream regarding her pregnancy, and perhaps further by Mary concerning the words of the archangel Gabriel to her at the Annunciation, Joseph knew that God had conducted himself as a husband in regard to Mary. She was now prohibited to him for all time, and for the sake of the Child and Mary he could only live with her in an absolutely chaste relationship."
But if you also look at the "contractual" agreement which betrothal was, Joseph and Mary were more than likely never "lovers" -- given the age difference (this is not based on "Sacred Tradition" but rather secular historians maintain that Mary must have been around 12-13 when betrothed, and the man was usually much older; it was a "business deal" between him and the father of the Jewish girl being betrothed). Therefore, there is no evidence that he and she would have had romantic, passionate sexual attraction period. Especially when you know that the woman you are supposed to marry is carrying the Messiah in her womb, that in itself changes your whole outlook. Your entire focus would shift to God, worshiping Him constantly, as you (Joseph) are guardian of the New Ark of the Covenant (who is Mary) and the Lord Incarnate, her Divine Son.