Study: half of studies in psychology journals wrong

[Colter]

https://reason.com/blog/2015/08/28/study-finds-studies-are-wrong



Study Finds: Studies Are Wrong
A major project to reproduce study results from psychology journals found that more than half could not be replicated.
Peter Suderman|Aug. 28, 2015 10:59 am



"One of the bedrock assumptions of science is that for a study's results to be valid, other researchers should be able to reproduce the study and reach the same conclusions. The ability to successfully reproduce a study and find the same results is, as much as anything, how we know that its findings are true, rather than a one-off result.

This seems obvious, but in practice, a lot more work goes into original studies designed to create interesting conclusions than into the rather less interesting work of reproducing studies that have already been done to see whether their results hold up.

That's why efforts like the Reproducibility Project, which attempted to retest findings from 100 studies in three top-tier psychology journals, are so important. As it turns out, findings from the majority of the studies the project attempted to redo could not be reproduced. The New York Times"...

 

[MehGuy]

Not very surprising..

 

[ChristsSoldier115]

What if that study is wrong about the other studies?

 

[cow451]

The most difficult area to study is human behavior. There are sooooooooooooooooooooooooooooooooooooooooooooo many variables. And a great many studies involve college students. Oddly enough, college students don't act like normal people.

 

[Colter]

What if that study is wrong about the other studies?

We will have to do a study.....by a committee.....a blue ribbon panel.....

 

[keith99]

The most difficult area to study is human behavior. There are sooooooooooooooooooooooooooooooooooooooooooooo many variables. And a great many studies involve college students. Oddly enough, college students don't act like normal people.

You reminded me of a joke my college psychology prof made. It was something like 'We don't know much about people in general, but we know everything about college sophomores'.

He made another one that goes to the heart of this thread. He said he has used the same for 10 years. exact same questions, but the answers kept changing.

 

[rambot]

What if that study is wrong about the other studies?

Beautiful this

 

[TerranceL]

The most difficult area to study is human behavior. There are sooooooooooooooooooooooooooooooooooooooooooooo many variables. And a great many studies involve college students. Oddly enough, college students don't act like normal people.

Hahahaha.

 

[Audacious]

A lot of this is apparently due to small representative sample sizes, which is... exactly what you'd expect. A lot of studies, especially in social psychology, have 5 or 10 people in them and are then generalized to a population, which really, really doesn't work.

In psych classes you hear a lot about interesting results, but the methedology is often just terrible. Either the US needs more funding for psych studies, or psych studies are being done by completely incompetent individuals, or both.

 

[Archaeopteryx]

The authors' conclusion (Science 2015;349(6251):aac4716):

After this intensive effort to reproduce a sample of published psychological findings, how many of the effects have we established are true? Zero. And how many of the effects have we established are false? Zero. Is this a limitation of the project design? No. It is the reality of doing science, even if it is not appreciated in daily practice. Humans desire certainty, and science infrequently provides it. As much as we might wish it to be otherwise, a single study almost never provides definitive resolution for or against an effect and its explanation. The original studies examined here offered tentative evidence; the replications we conducted offered additional, confirmatory evidence. In some cases, the replications increase confidence in the reliability of the original results; in other cases, the replications suggest that more investigation is needed to establish the validity of the original findings. Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims.

The present results suggest that there is room to improve reproducibility in psychology. Any temptation to interpret these results as a defeat for psychology, or science more generally, must contend with the fact that this project demonstrates science behaving as it should. Hypotheses abound that the present culture in science may be negatively affecting the reproducibility of findings. An ideological response would discount the arguments, discredit the sources, and proceed merrily along. The scientific process is not ideological. Science does not always provide comfort for what we wish to be; it confronts us with what is. Moreover, as illustrated by the Transparency and Openness Promotion (TOP) Guidelines (http://cos.io/top) (37), the research community is taking action already to improve the quality and credibility of the scientific literature.

We conducted this project because we care deeply about the health of our discipline and believe in its promise for accumulating knowledge about human behavior that can advance the quality of the human condition. Reproducibility is central to that aim. Accumulating evidence is the scientific community’s method of self-correction and is the best available option for achieving that ultimate goal: truth.

 

[Archaeopteryx]

Interesting perspectives on the reproducibility project from Russ Poldrack and Dorothy Bishop.

 

[Archaeopteryx]

Reproducibility concerns everyone in the scientific community. This issue doesn't just apply to psychology, but to all fields. From the Boston Globe:

Evidence of a quiet crisis in science is mounting. A growing chorus of researchers worry that far too many findings in the top research journals can’t be replicated. “There’s a whole groundswell of awareness that a lot of biomedical research is not as strongly predictive as you think it would be,” said Dr. Kevin Staley, an epilepsy researcher at Massachusetts General Hospital. “People eventually become aware because there’s a wake of silence after a false positive result,” he added. The same is true in every field of science, from neuroscience to stem cells.

Ideally, science builds on and corrects itself. In practice, the incentives facing scientists can hamper the process. It’s more exciting and advantageous to publish a new therapeutic approach for a disease than to revisit a past discovery. Yet unless researchers point out the limitations of one another’s work, the scientific literature can end up cluttered with results that are partially or, in some cases, not at all true.

Recently, researchers and the US government alike have sought to assess how much research is irreproducible — and why — and are looking for systematic ways to retest experiments that make headlines but yield no further progress.

The article is well worth reading in its entirety.

 

[SuperCloud]

What if that study is wrong about the other studies?

In theory it can't be, based upon the principal of repeatability in science. That's why scientific papers are published, so that other peers in science can replicate their exact tests using the same methods and materials. If the same results are repeated confidence is gained in the conclusions of the first paper. If the results differ then its likely the conclusions of the first paper will no longer be accepted and will be challenged.

Repeatability is not a minor thing in science. It is one of the major things that differentiates science from theology and non-sciences.

When I took a philosophy of science course (the early philosophers of science held multiple doctoral degrees in both philosophy and one of the natural--often physical--sciences) my class learned that most philosophers of science don't regard psychology as a real science.

I met a pretty smart atheist in the VA hospital that similarly dismissed psychology as a science. This was a dude that liked to read a lot of science.

Philosophers of science accuse psychologists in the field of psychology as of often committing ad hoc hypotheses. You can google what that is up on Wikipedia.

As far as I'm concerned psychology is not a real science. Albeit, as a field of inquiry (just as theology is as well) it makes use of experiments and statistical mathematics. So, it's closer to a science than the academic fields of English and history. Nonetheless, it's less one than biology. And I would argue biology--whilst a science--is less so than its siblings in the physical sciences (e.g., chemistry, physics, astronomy).

 

[Archaeopteryx]

In theory it can't be, based upon the principal of repeatability in science. That's why scientific papers are published, so that other peers in science can replicate their exact tests using the same methods and materials. If the same results are repeated confidence is gained in the conclusions of the first paper. If the results differ then its likely the conclusions of the first paper will no longer be accepted and will be challenged.

Repeatability is not a minor thing in science. It is one of the major things that differentiates science from theology and non-sciences.

When I took a philosophy of science course (the early philosophers of science held multiple doctoral degrees in both philosophy and one of the natural--often physical--sciences) my class learned that most philosophers of science don't regard psychology as a real science.

I met a pretty smart atheist in the VA hospital that similarly dismissed psychology as a science. This was a dude that liked to read a lot of science.

Philosophers of science accuse psychologists in the field of psychology as of often committing ad hoc hypotheses. You can google what that is up on Wikipedia.

As far as I'm concerned psychology is not a real science. Albeit, as a field of inquiry (just as theology is as well) it makes use of experiments and statistical mathematics. So, it's closer to a science than the academic fields of English and history. Nonetheless, it's less one than biology. And I would argue biology--whilst a science--is less so than its siblings in the physical sciences (e.g., chemistry, physics, astronomy).

Why would psychology and biology be "less" science than chemistry and physics?

 

[SuperCloud]

Why would psychology and biology be "less" science than chemistry and physics?

Putting psychology aside, biology itself suffers from a statistical error of 1.

A sample size of 1 always constitutes in statistical mathematics an error. Earth is that sample size of 1. Or that is to say life on earth. Chemistry in physics holds true--per their laws--throughout the entire universe. The only place known to have life is planet earth and therefore the only place life can be studied is on earth.

That in itself constitutes a problem. This is not my original idea either. I learned this in either a philosophy of biology class or a philosophy of science class. I think it was the former. Bearing in mind this kind of critique comes from philosophers with doctoral degrees in physical sciences like chemistry or physics.

There are only a few laws in biology relative to the many in chemistry and physics. Aside from the fact that what drives the physiological functions of all biological organisms are the laws of chemistry and physics. So few laws in biological laws original to biology, specific to biology, in my mind indicates biology is less a science than those other two siblings in powers of predictability.

Repeatability and predictability are not minor things in science. There is a lot more storytelling in biology than there is in chemistry and physics. You know... in chemistry, a certain amount of A chemical interacting with B chemical will always result in C outcome. Always. It's predictive. And it's repeatable.

A lot of the modern day life sciences (which biology is one of) rely less on repeatability, predictability, or even observation. Almost all of its weight is placed upon inferential science. We infer from the fossil records mankind descended from earlier primates of a different species than we are. We infer from brain scans that grown men like Bruce Jenner were born with "female brains." We infer from statistical data on X sample size that a gay gene(s) existed and caused two men to want to give each oral sex. Albeit, this last prevailing view has been falling to the new epigenetic views.

Chemistry and physics place a lot less weight on inferences (although they do exist) and a lot more weight on powers of prediction. Physics I believe even more so than chemistry.

 

[Archaeopteryx]

Putting psychology aside, biology itself suffers from a statistical error of 1.

A sample size of 1 always constitutes in statistical mathematics an error. Earth is that sample size of 1. Or that is to say life on earth. Chemistry in physics holds true--per their laws--throughout the entire universe. The only place known to have life is planet earth and therefore the only place life can be studied is on earth.

That in itself constitutes a problem. This is not my original idea either. I learned this in either a philosophy of biology class or a philosophy of science class. I think it was the former. Bearing in mind this kind of critique comes from philosophers with doctoral degrees in physical sciences like chemistry or physics.

There are only a few laws in biology relative to the many in chemistry and physics. Aside from the fact that what drives the physiological functions of all biological organisms are the laws of chemistry and physics. So few laws in biological laws original to biology, specific to biology, in my mind indicates biology is less a science than those other two siblings in powers of predictability.

Repeatability and predictability are not minor things in science. There is a lot more storytelling in biology than there is in chemistry and physics. You know... in chemistry, a certain amount of A chemical interacting with B chemical will always result in C outcome. Always. It's predictive. And it's repeatable.

A lot of the modern day life sciences (which biology is one of) rely less on repeatability, predictability, or even observation. Almost all of its weight is placed upon inferential science. We infer from the fossil records mankind descended from earlier primates of a different species than we are. We infer from brain scans that grown men like Bruce Jenner were born with "female brains." We infer from statistical data on X sample size that a gay gene(s) existed and caused two men to want to give each oral sex. Albeit, this last prevailing view has been falling to the new epigenetic views.

Chemistry and physics place a lot less weight on inferences (although they do exist) and a lot more weight on powers of prediction. Physics I believe even more so than chemistry.

I don't think you've fully explained why this makes biology and psychology "less of" a science. All science is built on inferences. Laws are merely descriptions of regular patterns. The aim of science isn't merely to develop a catalogue of laws, but to provide good explanations for the phenomena under study; that is, to develop theories.

 

[Archaeopteryx]

Putting psychology aside, biology itself suffers from a statistical error of 1.

A sample size of 1 always constitutes in statistical mathematics an error. Earth is that sample size of 1. Or that is to say life on earth. Chemistry in physics holds true--per their laws--throughout the entire universe. The only place known to have life is planet earth and therefore the only place life can be studied is on earth.

Why would this entail that life cannot be studied scientifically?

 

[bhsmte]

The most difficult area to study is human behavior. There are sooooooooooooooooooooooooooooooooooooooooooooo many variables. And a great many studies involve college students. Oddly enough, college students don't act like normal people.

This!!!

The variables involved in studying human behavior are countless and also can be very subtle.

 

[SuperCloud]

I don't think you've fully explained why this makes biology and psychology "less of" a science. All science is built on inferences.

Yeah, but chemistry and physics less so. One of the aims of science is to develop powers of predictions. We can use the predictions in the physical sciences to make 1st, 2nd, 3rd, and 4th generation cellphones with higher resolution pixel counts.

What can biology or physics predict? Not much. Biology only a very little relative to the great many things the actual physical sciences can predict.

Physics is the queen of all sciences. Everything under it envies its for its predicative powers. And biology overtime emerged into a science, not a physical science but a natural science, and envying physics it hopes for predictive power just as psychology does. That's why they want a deterministic answer to why Bruce Jenner wants to wear dresses and be called Caityln, because deterministic causes consequently provide predictive powers. You can like the people that program and build 4th and 5th generation cellphones say to a mother with a fetus developing in her, "Your child will come out to be a cross dressing Nazi who will in fact have a female brain encapsulated in a male body."

Laws are merely descriptions of regular patterns. The aim of science isn't merely to develop a catalogue of laws, but to provide good explanations for the phenomena under study; that is, to develop theories.

According to the early philosophers of science--just about all of whom had Ph.D.'s in the physical sciences--one of the prevailing aims of science is to develop predictive powers. Yes, you are correct, explanatory power as well. But without predictive powers how "explanatory" is a scientific explanation from a non-scientific one?

I'll accept scientific laws as being explained as descriptions of regular patterns. Albeit, often times laws can be expressed mathematically. But math is applicable to anything with patterns.

But a scientific law would be distinguished from biologist's punnet square. The former has predictive powers that can predict things "always" shall we say, with accuracy and precision. The latter has predicative powers but less... it can only give odds that this will possibly happen x out of y times.

 

[Archaeopteryx]

Yeah, but chemistry and physics less so.

How so? And what would that even mean?

One of the aims of science is to develop powers of predictions. We can use the predictions in the physical sciences to make 1st, 2nd, 3rd, and 4th generation cellphones with higher resolution pixel counts.

Yes, and? Discoveries in other scientific disciplines also have practical application.

What can biology or physics predict? Not much. Biology only a very little relative to the great many things the actual physical sciences can predict.

I get the impression that you haven't studied much biology or psychology.

Physics is the queen of all sciences.

Science has no rulers.

But a scientific law would be distinguished from biologist's punnet square. The former has predictive powers that can predict things "always" shall we say, with accuracy and precision. The latter has predicative powers but less... it can only give odds that this will possibly happen x out of y times.

Why would that make it less of a science?