Some people believe just the opposite -- same father but different mothers -- that James was Joseph's son from an earlier marriage.
People who believe in perpetual virginity of Mary see it this way.
ADELPHOI in Koine Greek can mean 'close relative' and is not limited to actual brother.
You are mistaken. "Adelphos", the Greek word translated "brother," does not mean a close relative. Here is the complete definition of "adelphos" from Bauer, Arndt, Gingrich, Danker one of, if not, the most highly accredited Greek lexicons available. Note, "cousin" or "close relative" is not one of the meanings.
[size=6=ἀδελφός, οῦ, ὁ [/size](Hom. [ἀδελφεός]+; accord. to B-D-F §13; Schwyzer I 555; Mlt-H. II 58; PKatz, TLZ 83, ’58, 315f vocative ἄδελφε should be accented on the antepenult in Ac 9:17; 21:20 contrary to the practice of the editions; also GPt 2:5.)
① a male from the same womb as the reference pers., brother, Mt 1:2, 11; 4:18, 21 al.; τὸν ἀ. τ. ἴδιον J 1:41 (s. Jos., Ant. 11, 300). Of Jesus’ brothers (passages like Gen 13:8; 14:14; 24:48; 29:12; Lev 10:4; 1 Ch 9:6 do not establish the mng. ‘cousin’ for ἀ.; they only show that in rendering the Hebr. אָח ἀ. is used loosely in isolated cases to designate masc. relatives of various degrees. The case of ἀδελφή [q.v. 1] is similar Gen 24:59f; Tob 8:4, 7 [cp. 7:15]; Jos., Ant. 1, 211 [ἀδελφή = ἀδελφοῦ παῖς]. Sim. M. Ant., who [1, 14, 1] uses ἀ. for his brother-in-law Severus; the same use is found occas. in the pap: JCollins, TS 5, ’44, 484–94; s. VTscherikover HTR ’42, 25–44) Mt 12:46f; 13:55; Mk 3:31f; J 2:12; 7:3, 5; Ac 1:14; 1 Cor 9:5. James ὁ ἀδελφὸς τοῦ κυρίου Gal 1:19. The pl. can also mean brothers and sisters (Eur., El. 536; Andoc. 1, 47 ἡ μήτηρ ἡ ἐκείνου κ. ὁ πατὴρ ὁ ἐμὸς ἀδελφοί; Anton. Diog. 3 [Erot. Gr. I 233, 23; 26 Hercher]; POxy 713, 21f [97 A.D.] ἀδελφοῖς μου Διοδώρῳ κ. Θαΐδι; schol. on Nicander, Ther. 11 [p. 5, 9] δύο ἐγένοντο ἀδελφοί, Φάλαγξ μὲν ἄρσην, θήλεια δὲ Ἀράχνη τοὔνομα. The θεοὶ Ἀδελφοί, a married couple consisting of brother and sister on the throne of the Ptolemies: OGI 50, 2 [III B.C.] and pap [Mitt-Wilck. I/1, 99; I/2, 103–7, III B.C.]). In all these cases only one brother and one sister are involved. Yet there are also passages in which ἀδελφοί means brothers and sisters, and in whatever sequence the writer chooses (Polyb. 10, 18, 15 ποιήσεσθαι πρόνοιαν ὡς ἰδίων ἀδελφῶν καὶ τέκνων; Epict. 1, 12, 20 ἀδ. beside γονεῖς, τέκνα, γείτονες; 1, 22, 10; 4, 1, 111; Artem. 3, 31; Ptolem., Apotel. 3, 6; Diog. L. 7, 108; 120; 10, 18. In PMich 214, 12 [296 A.D.] οἱ ἀδελφοί σου seems to be even more general=‘your relatives’). Hence there is no doubt that in Lk 21:16 ἀδελφοί=brothers and sisters, but there is some room for uncertainty in the case of the ἀδελφοί of Jesus in Mt 12:46f; Mk 3:31; J 2:12; 7:3, 5; Ac 1:14.
② a pers. viewed as a brother in terms of a close affinity, brother, fellow member, member, associate fig. ext. of 1.
ⓐ one who shares beliefs (for an associated duality, s. Did., Gen. 127, 6 ἀ. ἐστι τοῦ φαινομένου ἔξω ἀνθρώπου ὁ κρυπτὸς καὶ ἐν διανοίᾳ ἄνθρωπος=brother to the man as he appears from without is the man who is hidden in thought): Jesus calls everyone who is devoted to him brother Mt 12:50; Mk 3:35, esp. his disciples Mt 28:10; J 20:17. Hence gener. for those in such spiritual communion Mt 25:40; Hb 2:12 (Ps 21:23), 17 al. Of a relationship w. a woman other than that of husband Hs 9, 11, 3 al.; 2 Cl 12:5.—Of the members of a relig. community (PParis 20 [II B.C.] al. of the hermits at the Serapeum in Memphis; UPZ 162 I, 20 [117 B.C.] ἀδελφοὶ οἱ τὰς λειτουργίας ἐν ταῖς νεκρίαις παρεχόμενοι; IG XIV, 956 B, 11f. ἀ.=member of the ἱερὰ ξυστικὴ σύνοδος; IPontEux II, 449f εἰσποιητοὶ ἀ. σεβόμενοι θεὸν Ὕψιστον [Ltzm. ZWT 55, 1913, 121]. Mystery pap [III A.D.]: APF 13, ’39, 212. Essenes in Jos., Bell. 2, 122. Vett. Val. 172, 31; Cleopatra ln. 94. See GMilligan 1908 on 1 Th 1:4; Ltzm. Hdb. on Ro 1:13 [lit.]; Dssm. B 82f, 140 [BS 87f, 142]; Nägeli 38; Cumont3 276). Hence used by Christians in their relations w. each other Ro 8:29, 1 Cor 5:11; Eph 6:23; 1 Ti 6:2; Ac 6:3; 9:30; 10:23; Rv 1:9; 12:10; IEph 10:3; ISm 12:1 al. So esp. w. proper names (for ἀδ. in a figurative sense used with a name, cp. the address of a letter PMich 162 verso [II A.D.] ἀπὸ Ἀπλωναρίου ἀδελφοῦ) to indicate membership in the Christian community Ro 16:23; 1 Cor 1:1; 16:12; 2 Cor 1:1; Phil 2:25; Col 1:1; 4:7, 9; 1 Th 3:2; Phlm 1; 1 Pt 5:12; 2 Pt 3:15; AcPl Ha 1, 30 al. Completely ἀδελφὸς ἐν κυρίῳ Phil 1:14. Oft. in direct address 1 Cl 1:1 (cod. A); 4:7; 13:1; 33:1; 2 Cl 20:2 al.; B 2:10; 3:6 al.; IRo 6:2; Hv 2, 4, 1; 3, 1, 1; 4; AcPl Ha 7, 4; 8, 21; AcPlCor 1:16. ἀδελφοί μου B 4:14; 5:5; 6:15; IEph 16:1; ἄνδρες ἀ. Ac 1:16 (rabb. par. in EStauffer, TLZ 77, ’52, 202); 15:7, 13; 1 Cl 14:1; 37:1; 43:4; 62:1. To interpret ἀ. in Ac 15:23 as ‘colleague’ (e.g. PGaechter, Petrus u. seine Zeit, ’58, 141f) is speculative; and the interpretation of ἀ. in 3J 5 and 10 as itinerant preachers (AKragerud, D. Lieblingsjünger im Johannesevangelium, ’59, 105) is based entirely on the context.
ⓑ a compatriot (cp. Pla., Menex. 239a ἡμεῖς δὲ καὶ οἱ ἡμέτεροι, μιᾶς μητρὸς πάντες ἀδελφοὶ φύντες; Lev 10:4; Dt 15:3, 12; 17:15 al.; Philo, Spec. Leg. 2, 79f ‘ἀ.’ τὸν ὁμόφυλον εἶπεν he termed a compatriot ‘brother’; Jos., Ant. 10, 201; 7, 371 after 1 Ch 28:2) Ac 2:29; 3:17, 22 (Dt 18:15); 7:2, 23 (Ex 2:11), 25f al.; Ro 9:3.
ⓒ without ref. to a common nationality or faith neighbor (of an intimate friend X., An. 7, 2, 25; 38. Specif. in the sense ‘neighbor’ Gen 9:5; Lev 19:17 al.) Mt 5:22ff; 7:3ff; 18:15, 21, 35; Lk 6:41f; 17:3; B 19:4; Hm 2:2 al.
ⓓ Form of address used by a king to persons in very high position (OGI 138, 3; 168, 26; 36 [both II B.C.]; Jos., Ant. 13, 45; 126) Herod says ἀδελφὲ Πιλᾶτε GPt 2:5.—JO’Callaghan, El vocativo sing. de ἀδελφός, Biblica 52, ’71, 217–25.—B. 107. DELG. M-M. EDNT. TW. Sv.
Arndt, W., Danker, F. W., Bauer, W., & Gingrich, F. W. (2000). A Greek-English lexicon of the New Testament and other early Christian literature (3rd ed., pp. 18–19). Chicago: University of Chicago Press.