Intercessory prayer study analysis

[ifriit]

That's why you have statistical analysis. You have only 1 chance in 20 at p =0.05 that it is a sampling (type I) error.

Sloan's statement was not about any individual study, but an evaluation of the collection of studies, so I'm not sure how this would apply.

1. I used it as an example where, if the system is running at maximum, addition of a little more won't give you a big effect.

True. I guess the problem I have is mostly in the analogy; we can test the human body, determine the effects of endorphins and morphine, so we can determine that the system is running at maximum. However, with prayer, we can in no way quantify it, and even qualifying it is dubious; how are we to determine that there is a system, much less that it is "running at maximum?"

2. When morphine was first tested, no one bothered about the endogenous levels of endorphins. What mattered was the difference between groups -- no morphine and morphine. So you are agreeing that having uncontrolled IP in both groups won't make an effect appear where there isn't any.

Definitely.

Prayer refers to the act of APPEALING TO A DEITY. It doesn't mean that the deity is responding. Or even that there is a deity TO respond. IP is simply a description of the act, not a declaration that the deity appealed to is doing anything. We could call it by something else, but the terminology is already in place to describe the act. We would have to redefine "wishing" or "magic" to be the exact same as IP.

Actually, some forms of magic are IP. Magic is a bit of a catchall for abnormally caused effects. However, point taken.

Because I had already been skeptical of their methodology and they passed. I am just now looking at the Mayo study and you are getting a glimpse of the same process I went thru with Byrd and Harris et al. Except in this one I found a flaw very quickly. Or, more accurately, I found a flaw in Williams' description of their methodology. To be quite fair, I need to go over the original Materials and Methods section to see if Williams reported their methodology accurately.

Yeah, sorry about that. Should have screened that article more carefully.

 

[ifriit]

No, I don't think I made myself clear on that issue, sorry. The peer-reviewers see exactly what everyone else is going to see. However they can make suggestions as to additional information they'd like to see published before accepting it. The raw data is reviewed by Byrd and colleagues prior to publication, but we are given only a handful of possible conclusions here: 1) Byrd faked the data. 2)The data was misanalysed. 3)The conclusions drawn from the data are correct. Reporting science is as much based on an honor system as anything you'll find. If Byrd's data is false, then another identically performed study will easily show as much. However it's not Byrd's responsibility to conduct such a study, and if the reviewers have accepted the paper for publication, then Byrd's study has passed the rigors of the scientific process.

Okay. Has anyone tried repeating Byrd's study?

 

[MartinM]

14. Byrd, RC, Positive theraputic effects of intercessory prayer in a coronary care population. Southern Med Jour 1988 81(7):826-29. http://www.godandscience.org/apologetics/smj1.html
15. WS Harris, M Gowda, JW Kolb, CP Strychacz, JL Vacek, PG Jones, A Forker, JH O'Keefe, BD McCallister, A randomized, controlled trial of the effects of remote, intercessory prayer on outcomes in patients admitted to the coronary care unit. Arch Intern Med. 1999;159:2273-2278 http://archinte.ama-assn.org/issues/v159n19/rfull/ioi90043.html

Thanks. The second link is dead: the abstract of the Harris paper can now be found here. I can't access the full article from my current location.

I've had time to take a quick look at the Byrd study. The univariate analyses are underwhelming. Small number of significant results in a fairly large pool - classic data dredge, from the looks of things.

The problems with the multivariate analysis are much more interesting. Since the predictors appear to be strongly interdependent, collinearity and multicollinearity are serious issues. In such cases, random errors can seriously affect the results. Also, the sample size seems on the small side given the number of predictors. Not only a problem in its own right, small sample size can greatly exacerbate the already considerable issue of collinearity.

 

[MartinM]

That's why you have statistical analysis. You have only 1 chance in 20 at p =0.05 that it is a sampling (type I) error

That depends on the circumstances. Naive interpretation of p-values as simple probabilities doesn't always hold up. For example, it would be wrong to say that since there are 29 variables under inspection in the Byrd study that a confidence level of 95% implies an expectation of only one or two statistically significant results by random chance alone. The interdependence of the variables torpedos that interpretation.

 

[MartinM]

The ONLY reason Tessman would ask for the raw data was to check to see if Byrd had falsified the data. An unwarranted ad hominem attack and a direct insult, absent any evidence, to his character

I can't agree with that.

Again, the data in Tables 1, 2, and 3 is a summary from patient charts. The raw data would be the charts themselves. In order for the raw data to be different than what is presented in Tables 1 and 2 is for Byrd to have copied the data from chart to table wrong

But the raw data would contain more information than the charts - a totally distinct issue from the accuracy of the charts themselves. For example, the raw data would allow us to check how many patients had any two of the conditions in question - information simply not contained in the charts. It would also allow us to, say, re-randomise and re-analyse the data. That would be a very powerful test of the methodology.

 

[lucaspa]

I've had time to take a quick look at the Byrd study. The univariate analyses are underwhelming. Small number of significant results in a fairly large pool - classic data dredge, from the looks of things.

When Byrd did the study, IP was a hypothesis. So Byrd did what we all do: he used a big bore shotgun to look at everything on the first pass because he had no idea what effect he was looking for.

And the small number of significant results don't tell the whole story.
In 14 other parameters, the prayer group was less than the controls, in one they were the same, and were worse in only 5 areas. Also, the chi-square test is not that sensitive. For instance, for diuretics 5 patients in the prayer group needed them compared to 15 in the controls. That is statistically different. But for major surgery before discharge 5 patients in the prayer group and 14 in the control group had it. That difference of 1 patient means it was not quite statistically different. Or in congestive heart failure 8 patients in the prayer group vs 20 in the control was statistically significant. In the category of receiving ionotropic agents (drugs to help the heart beat) 8 patients in the prayer group compared to 16 in the control was not statistically different. For antianginal agents 21 patients in the prayer group needed them compared to 19 in the controls.

Thus there is a trend. Where the prayer group is better, in 14 categories it is a lot better by 30-50% fewer patients. In the 5 categories where prayer is "worse" it is only by 1 or 2 patients. Example, 2 patients in the prayer group needed a permanent pacemaker compared to 1 in the control.

The problems with the multivariate analysis are much more interesting. Since the predictors appear to be strongly interdependent, collinearity and multicollinearity are serious issues. In such cases, random errors can seriously affect the results. Also, the sample size seems on the small side given the number of predictors. Not only a problem in its own right, small sample size can greatly exacerbate the already considerable issue of collinearity.

I've heard this argument, but in looking at the parameters most are actually independent. For instance, many people think that pneumonia and antibiotic use are linked. But they aren't. Most pneumonias are viral and don't receive antibiotics.

 

[lucaspa]

I can't agree with that.

But there is no other reason whether you agree or not.

But the raw data would contain more information than the charts - a totally distinct issue from the accuracy of the charts themselves. For example, the raw data would allow us to check how many patients had any two of the conditions in question - information simply not contained in the charts. It would also allow us to, say, re-randomise and re-analyse the data. That would be a very powerful test of the methodology.

A patient with two conditions is in the charts. That's how he got the data. And, since he is looking for complications, he may not even have taken down that they occurred in the same patient. Just went thru and checked which of the 26 categories were "yes" and which "no". I wouldn't have necessarily marked them down.

Now, TODAY there would probably be a spreadsheet with a identifier for each individual in the spreadsheet that couldn't be linked back to the patient. However, there is still no real reason to ask for the spreadsheet and no stigma if it is refused.

You can't "re-randomize" the data. The code is broken and you know which patients were in which group. It doesn't test the methodology at all. You get the same result Byrd did!

Now, Cochrane's Review did reanlyze Table 3 by putting Intermediates with Good and then putting Intermediates with Poor to see if that changed the results. It didn't.

 

[ifriit]

Sure, he COULD have simply omitted the giving of antibiotic to an prayer patient or said a non-prayer patient got antibiotic. However, as I think about, the ONLY way to check that would be to go thru the patient charts of all the patients. Privacy laws won't permit that. They certainly won't permit the files to be SENT to Dr. Tessman. Even if Tessman showed up at San Fran General, he couldn't get access to them. It's a red herring.

Hm, good point.

The ONLY reason Tessman would ask for the raw data was to check to see if Byrd had falsified the data. An unwarranted ad hominem attack and a direct insult, absent any evidence, to his character.

I think ad hominem might be used a little too loosely here; I'd be more interested in seeing Tessman's stated justification for examining the data. Unfortunately, I can't get a copy of his analysis online, but it appears Tessman's analysis appeared in the May/June 2000 issue of Skeptical Inquirer (Efficacy of Prayer: A Critical Examination of Claims). It also appears from a summary that this article was the source of Williamson's discussion on the Harris study; it may provide detail that Williamson did not. The Straight Dope's summary of the complaints about Byrd's study are:

"Specifically, while the test was supposed to be double-blind and the article describing it claimed it was, a number of investigators have found that this was not true. Byrd himself determined who did better, those who were prayed for or those who were not, and he determined it after he knew who was in which group. Furthermore, the coordinator of the study was not blinded. Double-blind studies are done so those running the experiments don't accidentally contaminate the results with their own viewpoints. The failure to properly blind this study calls the results into serious question."
(Straight Dope Staff Report: Have studies proven that prayer can help heal the sick?)

Also, Harris and Tessman apparently had a debate; a transcript can be found at
http://www.csicop.org/articles/20010810-prayer/. (Note: don't put a URL in a url block, confuses the heck out of the BB software.)

I also note that Harris's study is alleged to be a repeat of Byrd's; this was not obvious to me before.

 

[lucaspa]

That depends on the circumstances. Naive interpretation of p-values as simple probabilities doesn't always hold up. For example, it would be wrong to say that since there are 29 variables under inspection in the Byrd study that a confidence level of 95% implies an expectation of only one or two statistically significant results by random chance alone. The interdependence of the variables torpedos that interpretation.

That is not the context I was speaking in. It was addressed to the question whether the results could be due to sampling error. That's why you do a statistical analysis: to test that the results are not due to accidentally putting all the people who do well into the prayer group and all those who do badly into the non-prayer group.

Since you make the claim and know about statistics, why don't you show us the equations that would determine how many groups would give a p value of less than 0.05 by chance alone?

BTW, a p value is not a 95% confidence interval. A 95% confidence interval is the distance from the mean that you expect to find 95% of the values within a group, not the difference between two groups.

 

[nyj]

Specifically, while the test was supposed to be double-blind and the article describing it claimed it was, a number of investigators have found that this was not true.

Who were these investigators? Accusations of academic fraud are very very serious.

Furthermore, the coordinator of the study was not blinded.

Someone has to not be blinded. Someone has to coordinate the study and place the patients into their respective groups. So this isn't only a non-valid criticism, it's pure silliness.

 

[ifriit]

Also, something I noticed, regarding something way back from post 6:

1) SOMEONE always has to know the treatment groups and can't be blinded. The person delivering the drug and placebo has to know which group the patient belongs to. In every study. If this is a valid criticism, then it applies to EVERY such study and they all have to be thrown out.

The Straight Dope's analysis directly contradicts this, saying about double-blind studies, "neither the subjects nor the test administrators know who's getting the treatment and who's getting a placebo."

Which is it?

 

[lucaspa]

The Straight Dope's summary of the complaints about Byrd's study are:

"Specifically, while the test was supposed to be double-blind and the article describing it claimed it was, a number of investigators have found that this was not true. Byrd himself determined who did better, those who were prayed for or those who were not, and he determined it after he knew who was in which group. Furthermore, the coordinator of the study was not blinded. Double-blind studies are done so those running the experiments don't accidentally contaminate the results with their own viewpoints. The failure to properly blind this study calls the results into serious question."
(Straight Dope Staff Report: Have studies proven that prayer can help heal the sick?)

WHEN Byrd did Table 3, he was apparently unblinded. However, Tables 1 and 2 were done before the blinding was broken. As I said, Cochrane's Review addressed this. It is possible that, for Table 3 alone, Byrd could have been biased. The way to test that is to first put all the Intermediates in the Good and then to put all the Intermediates into the Poor groups. So now you have only 2 groups, Good and Poor. When this was done, the results were the same. Prayer did better. Now, IF Byrd had been shading the results, in one of those retests, the statistical test should have been given different results.

Of COURSE the coordinator wasn't blinded. This is a specious cmplaint that I already addressed above. Are you not reading my posts? That is annoying.

To go over it again, SOMEONE has to be unblinded in order to make sure that the right people get the placebo or the drug. For instance, you do a study to see if morphine relieves post-op pain. You have shots of saline and shots with morphine. At the least, the pharmacist preparing the shot has to know which person is getting saline and which is getting morphine.

Here, the coordinator with the intercessors was the pharmacist. She had to be unblinded so she could tell the intercessors who to pray for.

This criticism shows just how far out of science the criticisers are. They are telling you that methods that anyone can figure out are necessary and unimportant for outcome are a flaw. As I said, if you accept this criticism as valid, don't EVER take another medicine. Not even an aspirin. Because they all were done in studies where someone associated with the study was unblinded but had to be.

The point is that
1. The patients didn't know which group they were in.
2. Byrd until the end of the study didn't know.
3. The nurses and other hospital staff didn't know; therefore they couldn't give favored treatement to one group or the other.

So, the only way for Byrd to profit from the unblinding in Table 1 was to change the data. Right?

I also note that Harris's study is alleged to be a repeat of Byrd's; this was not obvious to me before.

What Harris et al did was to send the raw data to the statistician (Byrd did his own stats) and that statistician if you look at the authors lived in California. So they had no idea how the study turned out until the statistician told them.

Even so, I've seen charges that Harris et al chose their scoring system because it was the only one that worked. How ridiculous!

 

[lucaspa]

Also, something I noticed, regarding something way back from post 6:


The Straight Dope's analysis directly contradicts this, saying about double-blind studies, "neither the subjects nor the test administrators know who's getting the treatment and who's getting a placebo."

Which is it?

Again, THINK about it! Did Byrd know? Was the coordinator of the intercessors "administering the study"? NO!

When I do the articular cartilage study, I have polymer felts soaked in saline and polymer felts with adult stem cells to put in the defects. Now, I know which is which but the surgeons don't. Neither do the animal care people. And neither do those who grade the result. I HAVE to know otherwise we have no idea which animals got the cells (which we are testing) and which didn't.

Again, if you do a study where the patient takes a real pill or the sugar pill, the pharmacist at least has to know which patient is which otherwise he can't give the correct pill to the correct patient!

 

[ifriit]

Who were these investigators? Accusations of academic fraud are very very serious.

All the information I've gathered so far traces back to Dr. Tessman's article, which I though I did not have access to. Just found it: Efficacy of Prayer.

Byrd's claimed violation of proper procedure is cited as "personal communication."

Someone has to not be blinded. Someone has to coordinate the study and place the patients into their respective groups. So this isn't only a non-valid criticism, it's pure silliness.

The issue is deeper than I stated. The coordinator regularly interacted with the patients as well as the prayer groups, going so far as to even inform the praying individuals the status of the individuals being prayed for. The coordinator was too heavily involved.

 

[lucaspa]

Also, something I noticed, regarding something way back from post 6:


The Straight Dope's analysis directly contradicts this,

Are you getting the idea yet that the "Straight Dope" isn't? Or that he/she is a real dope? Misinformation is not limited to creationist websites.

 

[lucaspa]

Someone has to not be blinded. Someone has to coordinate the study and place the patients into their respective groups. So this isn't only a non-valid criticism, it's pure silliness.

Thank you!

What we have here is a group of people just as devoted to this website as those devoted to AiG or Hovind. They simply aren't being skeptical and critical of the information being received from the website.

I find this particularly ironic since
1) The website portrays themselves as critical thinkers (but then, doesn't AiG?)

2) the posters think they are critical thinkers, but aren't using their skepticism about the website.

 

[ifriit]

Again, THINK about it! Did Byrd know? Was the coordinator of the intercessors "administering the study"? NO!

Actually, said coordinator divided the patients into their groups, collected all the data and entered it into the database, and informed the prayer groups of patient status, according to Byrd.

When I do the articular cartilage study, I have polymer felts soaked in saline and polymer felts with adult stem cells to put in the defects. Now, I know which is which but the surgeons don't. Neither do the animal care people. And neither do those who grade the result. I HAVE to know otherwise we have no idea which animals got the cells (which we are testing) and which didn't.

However, are you the person collecting the data?

 

[nyj]

Actually, said coordinator divided the patients into their groups, collected all the data and entered it into the database, and informed the prayer groups of patient status, according to Byrd.

I'm trying to figure out how this would skew the results. From my vantage point, I can't see how it would. Now, if she were telling the patients that someone really was praying for them, then I could see a problem. Unless of course you think that the intercessors could have given off "extra good vibes" knowing that their patients were doing better or worse than expected. Of course, if part of the point of prayer is to pray for something accurately, then the intercessors would need to know exactly what they were praying for, correct?

 

[lucaspa]

All the information I've gathered so far traces back to Dr. Tessman's article, which I though I did not have access to. Just found it: Efficacy of Prayer.

Byrd's claimed violation of proper procedure is cited as "personal communication."

"Byrd's Table 3, which might best have been constructed by a panel of "blinded" doctors, was constructed by Byrd alone. But it was done in response to criticism of an earlier version of his manuscript, the writing of which had already required that the code be broken. Thus Byrd was no longer blinded when he determined the answer to the key question of which did better, the intercessory prayer group or the control group (Byrd, personal communication)."

This is what I have addressed THREE TIMES. Ifriit, if you don't start reading the answers FIRST, I'm going to get angry. The statistical re-analysis done by Cochrane's Review showed that Byrd did not influence the results at all. The results are "robust", which they would not be if Byrd had screwed up.

The issue is deeper than I stated. The coordinator regularly interacted with the patients as well as the prayer groups, going so far as to even inform the praying individuals the status of the individuals being prayed for. The coordinator was too heavily involved.

This is what the articles stated "In his acknowledgments, Byrd thanks "Mrs. Janet Greene for her dedication to this study," but without any elaboration of her role. In a later publication (Byrd with Sherrill 1995) we leam that Janet Greene was hired "... to be our coordinator.... Janet entered names of all the volunteer patients into a computer that randomly divided them into two groups. ... half of the patients--only Janet knew who they were--were prayed for daily by our intercessors.... She kept derailed records of all patients in both groups." Thus the very coordinator of the study was completely unblinded. Once patients were assigned to one of the two groups, Greene should have had no further contact with the hospital.[3]"

She kept the detailed records of which groups they were in, but not their clinical evaluations. Tessman has overstated Greene's role. I have seen where Byrd discussed Greene's role, and it did NOT include outcomes.

 

[lucaspa]

Actually, said coordinator divided the patients into their groups, collected all the data and entered it into the database, and informed the prayer groups of patient status, according to Byrd.

However, are you the person collecting the data?

I'm like Byrd and doing the stats.

I just addressed this. Tessman overstated Greene's role. Notice the use of ellipses ( ... ) in how Tessman quotes Byrd. You know right there that there is some out-of-context going on.