Wait, what? What ethical and moral reasons do you need to "sure about things"? Despite your assertion about how you are not doing science, but some art of medicine, the use of p values necessarily includes uncertainty - and mostly , a quite arbitrary level of uncertainty. Do you think being sure is useful for projecting a your personal level of confidence, for convincing your patients to undergo a treatment, or merely to protect yourself from lawsuits? Please choose some other option.
Let me give you an example: Cricoid Pressure.
This is a technique to try and limit passive regurgitation during a rapid sequence induction of Anaesthesia, by pressing on the cricoid cartilage and thus hopefully obstructing the Oesophagus. Now, the studies upon which this practice is based were done on Obstetric patients back in the 60s, which showed a marked reduction of Aspiration. Subsequently it was universally adopted and extrapolated into other patient groups that have a similar risk for aspiration - due to raised intra-abdominal pressure or delayed gastric emptying, say. However, it has a downside: It potentially distorts the airway, making intubation more difficult, the anaesthetic assistant occupies one of their hands to give it, with active wretching there is marked risk for oesophageal tears, etc.
To top this off, we can never repeat these studies to test them - those pesky Moral and Ethical reasons. Once it was shown beneficial, we cannot morally not provide it to at risk patients, so no study with controls not given cricoid pressure can ethically be repeated in Obstetric GAs. From the principle of Beneficence, we cannot withold it. Even if patients agreed, it would be Medical personnel vs layman, so our reassurances or such, would be from a position of implicit trust and thus not 'freely chosen', but an assumed safety by the patient.
So these initial studies are it. We have to provide care that has been shown beneficial. We cannot withhold it, we cannot falsify these studies via repeating them. By doing so, we would not have those patients' interest at heart, and are thus in contravention of medical ethical principles of non-maleficence and Beneficence.
This holds true for anything that shows strong benefit (or conversely strong negative effect) - such as giving anti-hypertensives, providing emergency blood, discontinueing drugs that give dangerous effects, etc. We cannot repeat such studies, so Repeatability becomes an issue in Scientific Method, and why it cannot be properly applied. So we must be sure of ourselves in these things.
That is where EBM comes in so brilliantly, as it treats the data as non-falsifiable information. By utilising p values and the like, we can minimise the practices that are thus elevated to sacrosant Best Practice - the type that would have you scrapped from the medical rolls, or taken to court for malpractice, if you did not do them; and by doing them, potentially reduces morbidity and mortality. Everyone expects the best possible care for their own health, and barring lapses in Medical ethics like the Holocaust doctors, we are stuck here.
Of course, we can never be hundred percent sure. That is one of those epistemologic pitfalls of life. You can always doubt a result, back down to the axiomatic if need be. Mathematics has opted for those Fisherian variables of 5% statistical significance - as you said, arbitrarily (to facilitate calculations with a slide-rule if I recall), but all the safeguards and plotting and graphing that Math developed, was built around this mark. So do you have a better alternative? Not that I am aware of, nor that SBM supporters have preposed - So with p values and the ilk, we are just dealing with whining that the world isn't perfect, akin to why can't I eat as much carbs as I want and not get fat.
So it is theoretically imperfect, but by experience, the sensitivity is more than adequate. Retrospectively we can confirm these values, by seeing what had happened in practice (so again, not denigrating lower tier evidence, which can help to be confirmatory). At times, we adopt things even with imperfect p values, such as the widespread use of Anti-depressants, so when something does pass muster in this regard, we of course take notice.
In the invidious comparisons category for $200, I'd like to suggest that this resembles creationist misunderstandings of science. There are no proofs in science, and that really extends to medicine too.
Really? These are Mathematically proven values by statistical analysis. Of course, you can always argue Math hasn't been proven itself, which is true seeing that Russel failed to prove its axiomatic grounding and the Incompleteness theorum, but that is really just being abtuse.
So according to our best indicator for 'proof', cold hard numbers, these are valid findings. Again EBM is deductive - meaning its findings are necessarily valid and sound based on the supplied data; which differs from Induction systems where we cannot affirm it necessarily valid. Ever heard of Hume's Swans?
So yes, proven by the best means currently available to humanity, backed up by extensive statistical analysis. While not 'proven' in an absolute epistemologic sense, frankly nothing is; and nor did I ever make that claim. You are just erecting Strawmen here once more.
True, these deductive values only apply internally, so we must show external validity of a study in a population, but again we must show this mathematically usually, by investigating the variables. This is where Bias becomes so important. Recently they did a mock-serious study of testing Parachutes when jumping from planes, to help illustrate this concept; where the study found parachutes offered no benefit, but could of course only be conducted on a stationary plane on the ground, so external validity to flying planes is lacking. With extensively studied things like Cardiovascular risk, we can usually achieve external validity though. In this OP's findings, a prospective study would not be possible for instance, as any enrolled subjects would need to have their mental illness treated ethically, thus biasing results and ruining any validity it might have had.
Frankly, I don't know what you are trying to achieve here. You clearly are out of your depth where Medical Studies are concerned, as you even try and conflate EBM with SBM or 'science' (whatever you mean by that) in general. You keep casting aspersions on p-values, but what would you have us replace it with? Nor have you shown in what way the q-value in this study was flawed, as the study itself addressed the low number of cases it investigated, and supplies justification why it remains valid regardless.
So, um, what are you trying to achieve? What is your point? For at the moment we have Evidence, EBM, Math, on the one side; and solely your opinion on the other. As I said before, I feel akin to arguing with an Antivaxxer - which is not meant as an insult, but a juxtaposition to maybe help you see the silliness thereof. In my opinion, Evidence of something that can be supported by ancillary means, always trumps mere anecdote and personal opinions on it. You are free to disagree with the study's findings, but you are really not talking from any position of strength.
