Rather than rumor or word of mouth, Quizzler, Mary's history, or rather the history of Joseph's family is found written down. (we need to flag down Wolseley to tell us exactly where)
But, you can also find "clues" to this tradition in Scripture. When Jesus was dying on the Cross, one thing He did was to summon both His mother and the Disciple John. In very explicit words, He turned His mother over to John. Now, in Catholic teaching, the non-specific wording of Jesus: "Woman, behold your son," we have the Tradition that at that moment Mary became the mother of us all.
But aside from that, the action of Jesus was quite remarkable. The Man was close to death, and yet He was concerned over the care of His mother. Don't you think that if Jesus had younger brothers and sisters, He would not have been so worried about who would watch over Mary? The Jews are very diligent with familly responsibility. Had there been siblings, Jesus would NOT have severed their familly ties to their own mother in order to relinquish her care to a non-relative.
By that single action, Scripture is telling us a few things: Joseph is dead. Were he still alive, Mary would be under the care of her husband. And Mary has no other children.
Typically, the Protestant belief is that James is the brother of Jesus. James is nowhere to be found, though. At the foot of the Cross, Jesus turns His mother's care over to John.
Something to think about.
Peace be with you,
~VOW
....unwadding my underwear.... 
