While I know better than to directly question the authenticity of it as it has been received by EO people on their own board, I do think there is some legitimate reason to question whether or not it was meant specifically for the monks of St. Catherine's, rather than any given monastic community that might fall under the rule of Muslims (i.e., written to apply in any such case, including that one).
Not only are there similar letters that it can be shown to related to in its form (i.e., the letter of Muhammad to the Christians of Najran, as preserved in the
Chronicle of Seert; the Nestorians behind the
Chronicle would say similarly that the letter to the Christians of Najran is authentic, even though it's obviously not attempted to be presented there as the autograph), there is also the question how it could be meant to apply specifically to a monastery in the Sinai when Muhammad was not even alive when the Arab conquests began in that area. The conquest of Egypt occurred under the subsequent Rashidun (the first four caliphs after Muhammad's death), with 'Amr Ibn al 'As not even setting out for Egypt until December of 639, whereas Muhammad died in 632. There is absolutely zero evidence that Muhammad ever set foot anywhere in the Sinai Peninsula, and even the conquest of the wider Levant didn't begin in earnest until after Muhammad died, as in the immediate aftermath of his death the Rashidun were bogged down by the Ridda wars, which wouldn't end until the following year. If I recall correctly, Muhammad himself never made it any further north in battle than perhaps into a tiny sliver of Jordan, and even then that's not clear, because the battle there took place
while he was alive (the Battle of Mutah, 629 AD), but he apparently didn't take part in it (so the question is if they would've carried him around for stuff he wasn't involved in). He seems to drop off the military radar at some point a bit before his death due to illness (the last battle he took place in being the Battle of Tabouk in October-December of 630), leaving Usama Bin Zayd to attempt to invade Palestine under the tutelage of Abu Bakr in May of 632. This "expedition of Usama Bin Zayd" is listed as
chronologically last on the list of expeditions of Muhammad, and seeing as how Muhammad died on June 8, 632, we can assume that he was too ill by that point to have done anything more than perhaps inspire the planning of it at the very outset. We might do well to remember/realize that according to Islamic sources Muhammad's eventual death was
brought on by poisoning following the Islamic conquest of Khaybar, which is an oasis north of Medina (still well within the borders of modern KSA), which took place several years before in 628. (Not the most effective poisoning in the world, I suppose, but it couldn't have happened to a nicer fellah.)
Given all this, I'd be willing to bet it's authentic in the sense of representing a legitimate tradition of some sort, but that probably somewhere along the way it got conflated with some other stuff that was closer to Muhammad's own time period, probably to the monks' and monastery's benefit.
Unless, again, Muhammad dictated this to 'Ali to be given to monastic communities that they presumed would later fall under Islamic control after Muhammad's own death, of which St. Catherine's would've obviously been one. This seems very plausible to me, as there are parts of the Qur'an itself that are very praiseworthy towards Christians on account of their monasticism in particular (e.g.,
5:82), and the forms of Christianity taken up by the Arabs tended to be very much inclined towards hermeticism, probably in keeping with the Arabs' own nomadic lifestyles. (Those who were more sedentary or became so over time tended to be linguistically/culturally 'Syrianized' -- i.e., to adopt Syriac at least as a language of worship if not always a first language; see here Trimingham's
Christianity Among the Arabs in Pre-Islamic Times. This might also help explain why we don't have any written evidence of the Bible existing in Arabic prior to the rise of Islam, as discussed in Griffith's more recent book
The Bible in Arabic: The Scriptures of the "People of the Book" in the Language of Islam.)
So...yes? No? Sort of? Maybe? I wouldn't blame anyone for just taking the local Bedouins' word for it, and to the extent that it
can be traced back in written form to at least some point in the Middle Ages, it's 'authentic' so far as that goes. Generations of Muslim leaders have seen it as their duty to abide by it, and I'd care a lot more about that (since it directly involves the treatment of Christians in lands occupied by Islam) than whether or not every detail is historically accurate. Maybe I'm crazy, but as a Christian it wouldn't change a thing to me if Muhammad came back from the dead today to either certify it or deny it, because who is Muhammad anyway? But if it compels Muslims to not treat Christians anywhere like subhuman garbage, then good.