The Protoevangelium of James isn't considered Scripture, even by the RCC. It's pseudopigraphical, and Origin called it dubious and of recent appearance. It speaks of temple virgins in the Temple in Jerusalem - a practice there isn't the slightest bit of evidence for, and which doesn't make an ounce of sense.
Its similarity with the Vestal Virgins in the Roman temples, though, is striking.
Hey, are you going to say now that Pseudo-Matthew is also Scripture? Full of dragons, and panthers, and newborn Jesus telling the palm trees to bow to give Mary shade?
Or is that somehow conveniently different?
There isn't a single shred of evidence to suggest that Mary and Joseph alone, of all the couples in Israel, didn't have sex. The concept is ridiculous.