Peer Review – The Good, the Bad, and the Ugly

Status
Not open for further replies.

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
In some recent postings, there has been some major confusion about the peer review process. In this thread, let’s talk about peer review and gain some understanding. I’ll start. ;)

Disclaimer: I work managing the web presence for a non-profit company that publishes a number of academic technical peer-reviewed journals. I interact daily with the editors and volunteers. However, I am not a part of the review process for any of the journals. The journals published by my employer have nothing to do with biology, paleontology, geology, religion, creationism, or evolution. My opinions are my own as an individual, not as a representative of my employer (who I am deliberately leaving nameless).

Here is the preview of the upcoming posts:

  • How Peer Review Works
  • Reputation and Peer Review
  • The Good – the Advantages of Peer Review
  • The Bad - the Flaws of Peer Review
  • The Ugly – the Results of the Flaws
  • One Alternative
  • Supporting links
There are no “rules” for peer review, only conventions. This means that there is a lot of variation in how peer review is employed. For example, sometimes the identity of the author is kept from the reviewers, while at other times it is not.

Peer review is a very valuable process. However, while it can aid quality, it is no guarantee that such research is completely accurate. Peer review also has problems in that it tends to enforce and extend a particular orthodoxy over innovation.
 

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
There is a fine article about this on Wikipedia: http://en.wikipedia.org/wiki/Peer_review
I especially recommend the external links at the end. If I use any wording similar to the article, it is unintentional, but I did just peruse it, so I want to be sure to give it credit.

Peer review is a process used to critique, improve, and assure the quality of published research material, or research grants.

Here is a reasonably typical flow for a technical journal:

First – the pre-review period (not really included in the Wikipedia):

The editor collects various manuscripts. These can be from authors who have submitted them blindly, who have submitted them in response to a call for papers (typical), or from particularly noteworthy authors who have been specifically invited to submit.

This pre-review period is a major time of weeding out a lot of manuscripts that are considered by the editor (or their staff, or often by a subset of reviewers) to be either so off topic, or so unscholarly, or so poorly written that it is not worth going forward. Because the review process is so time-consuming, the editorial staff does not want to waste the time of the reviewers on articles that are not expected to pass the process, so this step typically errs on the side of rejecting manuscripts that might have made it. The reviewers are almost always anonymous, and not paid, and seen as experts in the field. This means that busy, important people are volunteering their time. This time is to be respected.

The author may or may not suggest reviewers for the manuscript. The editor is not bound to use suggested reviewers, but often the author knows who the experts are in their field better than the editor. The more specialized the field, the more likely the editor is to use or at least consult suggested reviewers.

The editor selects the reviewers who will be reviewing the article and makes the appropriate arrangements.

Second – the article is sent out to the reviewers. The author virtually never knows who will be reviewing their article. The reviewers typically remain anonymous, with some exceptions. The reviewers may or may not know who the author is. In more specialized fields, it becomes less and less important because the reviewers are likely to be able to determine the author (or the teacher of the author) based on the area of investigation which has been pursued.

Third, the reviewers return the article with suggestions and recommendations. The editor works with the author on the suggestions, if the journal moves forward. It is rare that an article is rejected at this point, and common that an article is modified and improved.

Fourth, depending on the scope of the changes, the reviewers may be asked to review the changes.

Finally, the article is published. By this time, the article has been polished both in terms of writing and content. While it is not required for the reviewers to agree with publishing the article (the decision is in the editors hands), in most cases an article will not be published against the opinion of any of the reviewers. The reviewers may not agree with everything in the article, but typically they will agree that it is worth publishing.

While this takes a good deal of time, it can greatly aid quality by catching errors in presentation, methodology, logic, etc., and fixing them before publication.
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
One of the keys to understanding how peer review works in the real world is to understanding the role reputation plays in the peer review process.

The technical journal (or conference) desires to enhance its reputation by publishing quality material that will attract a large audience within its field.

The editor desires to enhance their reputation by selecting papers on topics which support the journal (or conference).

The author(s) wish to enhance their reputation by being published in a prestigious publication. For college professors, this can have direct bearing on their advancement, such as attaining tenure.

The reviewers enjoy the pleasure of having a reputation as subject matter experts worthy of being sought for input.

With the concept of reputation also comes risk. Being adverse to risk in order to preserve their reputation is a major force causing the peer review process to be very conservative. This does not mean that unique and radical ideas never make it through the screening process, but they are the exception, not the rule. Incremental advancements are much more likely to be accepted than radical deviations.
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
Because of the conservative nature of the peer review process, it can tend to reinforce the status quo. It is much better suited for incremental developments within a settled field or worldview than it is for radical changes. For example, some of the key papers from Stephen Hawking were initially rejected because they did not conform to the current orthodoxy.

[FONT=&quot]Once a viewpoint has been labeled, both papers supporting that viewpoint and the practitioners of that viewpoint have severe problems with access to the technical journals. For example, both creationism and intelligent design scientists have had significant problems being published, even scientists who have had a long history of previously published papers.[/FONT]
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
One alternative when you have such widely varying viewpoints or interpretational models is to establish parallel systems of peer review. This allows each model to benefit from review, while allowing quality research to be shared openly. As the models develop in degree and complexity, they are better able to be fairly compared. This does not mean that it is not healthy and desired to have cross model interaction – just that the peer review process is not the best forum for such interaction.

This alternative also avoids rehashing old ground over and over. For example, if the research is supporting a fine point in the YEC understanding of fossil formation, it is better when reviewed by others that at least accept the possibility of such a viewpoint being accurate.

There have been some actions toward this method. YEC organizations have begun to use peer review in greater amounts. There are some serious financial problems with this, because of the limited audience, but there are signs of good progress.

Of course, journals are not the only source of peer-reviewed research. Proceedings from conferences also include peer-reviewed papers. Again, due to financial realities and the limited number of openly creationist scientists, there have not been many conferences. Perhaps the best known conference is the International Conference on Creationism. The sixth such conference will be held in 2008 in San Diego, California. http://icr.edu/icc2008/

Some may complain about the conference restricting itself to young earth creationism. This is not a valid complaint however, in that every conference always restricts the areas in which it is willing to invest the time and energy to explore. It is not as though there are not lots of other conferences available to non-YEC scientists.
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
(if you only go to one link, read the first article!)

Refereed Journals: Do They Insure Quality or Enforce Orthodoxy?
Abstract / link to full article at:
http://www.iscid.org/boards/ubb-get_topic-f-10-t-000059.html

Letters showing Michael Behe’s interaction with peer-reviewed journals:
http://www.discovery.org/scripts/viewDB/index.php?command=view&id=450

Peer review as scholarly conformity
http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/ss5.html

Bigotry in Science
http://icr.org/articles/print/840/
Examples of problems creationists have with being published
From the article:
”Creationists who publish scientific research in mainstream journals have found that they can publish articles with data having creationist implications, but will not get articles with openly creationist conclusions published. When they attempt to do this, their articles are usually rejected. Those who are well-known to evolutionists as creationists have more difficulty even with articles which do not have obvious creationist implications.
In the summer of 1985 Humphreys wrote to the journal Science pointing out that openly creationist articles are suppressed by most journals. He asked if Science had ‘a hidden policy of suppressing creationist letters.’ Christine Gilbert, the letters editor, replied and admitted, ‘It is true that we are not likely to publish creationist letters.’ This admission is particularly significant since Science’s official letters policy is that they represent ‘the range of opinions received.’ e.g., letters must be representative of part of the spectrum of opinions. Yet of all the opinions they receive, Science does not print the creationist ones.”

Do Creationists Publish in Notable Refereed Journals?
http://www.answersingenesis.org/docs/538.asp
Authors also detail multiple people who have published non-creationist articles, yet have problem with anything creationist

Quarterly peer-reviewed creationist journal
http://www.creationresearch.org/crsq.html
 
Upvote 0

random_guy

Senior Veteran
Jan 30, 2005
2,528
148
✟3,457.00
Faith
Christian
I really don't think there's a conspiracy of scientific journals actively suppressing scientific articles from Creationists.

"Creationists who publish scientific research in mainstream journals have found that they can publish articles with data having creationist implications, but will not get articles with openly creationist conclusions published. When they attempt to do this, their articles are usually rejected. Those who are well-known to evolutionists as creationists have more difficulty even with articles which do not have obvious creationist implications."

I definitely think the last 2 sentences are incorrect. Again, a professor on my thesis chair is a well known IDer. However, he publishes more papers in a single year than I have in my entire academic career. Views don't matter if the science is good. Second, every time a paper is rejected from a journal, they will list a reason why it was not accepted and what needs to be done to fix it. Can you show me which Creationists papers have been submitted and what the reject letters say? If there are very few, then it's clear that Creationists aren't active in the scientific peer review process.

Finally, conferences on just YECism and only on views that support YECism is not scientific. They should accept all papers that deal with the topic as long as they follow the scientific method. I'm sure when the idea of epigenetics first came up, it faced a lot of controversy. The idea that traits could be inheritted from the environment is a lot like LaMarkianism (which was disproved). However, it's now widely accepted. If you don't allow competing views that go through the scientific method, you're suppressing science.

EDIT: One more thing, yes scientific method is slow, but it helps maintain accuracy. That's why cold fusion was rejected (it also didn't follow the correct procedures). However, evolution has been around for 150 years, and been tested and tried for 150 years and it still stands today.
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
I never said conspiracy -- that implies collusion. Its just that there is an inherent bias in the system. It is more of a statistical bias than a direct one. In certain cases of editors/reviewers it can be more direct. Being statistical, you will find exceptions.

Privately, over a cup of cofee, off the record - ask your prof if he thinks there is bias in the system.

Read the links - and you'll see why I can understand and support parallel peer review.
 
Upvote 0
This site stays free and accessible to all because of donations from people like you.
Consider making a one-time or monthly donation. We appreciate your support!
- Dan Doughty and Team Christian Forums

random_guy

Senior Veteran
Jan 30, 2005
2,528
148
✟3,457.00
Faith
Christian
laptoppop said:
I never said conspiracy -- that implies collusion. Its just that there is an inherent bias in the system. It is more of a statistical bias than a direct one. In certain cases of editors/reviewers it can be more direct. Being statistical, you will find exceptions.

Privately, over a cup of cofee, off the record - ask your prof if he thinks there is bias in the system.

Read the links - and you'll see why I can understand and support parallel peer review.

But if Creationists never submit to scientific journals and/or show the reject letter, how do we know collusion exists? As for the inherit bias, it think the inherit bias is following the scientific method. If papers don't follow it, it won't get published. This includes not using the supernatural as an explanation. So yes, there is a bias, in that scientific journals will only accept scientific explanations and articles.

Also, this link might also be of interest. Nature, one of the leading scientific journals, is having a debate about peer review.

http://www.nature.com/nature/peerreview/index.html

It's very interesting and it talks about how it may be updated and changed. I think this discussion would be more fruitful if we keep the idea of, "Scientists are keeping the Creationists down" out. I would like to see a real discussion that doesn't involve conspiracies (yes, it would be a conspiracy if scientists tried to supress certain research).
 
Upvote 0

LewisWildermuth

Senior Veteran
May 17, 2002
2,526
128
51
Bloomington, Illinois
✟11,875.00
Faith
Christian
Marital Status
Single
Could you supply us with some rejection letters so we can know that these creationists even tried to publish?

I have seen the rejection letters of friends that are not creationists but I have yet to see a creationists that cliamed to have been rejected even show a rejection letter.

I just would like to know if this is a real problem before we go changing a system that seems to work well or if this is just a bunch of lies to cover up for the real reason why creationists never publish in real journals.
 
Upvote 0

notto

Legend
May 31, 2002
11,130
664
54
Visit site
✟22,369.00
Faith
United Ch. of Christ
laptoppop said:
Letters showing Michael Behe’s interaction with peer-reviewed journals:
http://www.discovery.org/scripts/viewDB/index.php?command=view&id=450

This points out the problem with Behe. He wanted to publish a response to critics of his books in peer reviewed journals, but he didn't submit the content of his book to peer review in the first place.

Using this to suggest that Behe is trying to enage in peer review with his actual work would be to mistate the circumstances.

Behe has not submitted his data, research, or conclusions to peer review. Behe hasn't sumbitted any original research in a way that would make it accessible to repetion or correction or dialog on his methods. He openly admitted this under oath.

Q. [Rothschild] Now you have never argued for intelligent design in a peer reviewed scientific journal, correct?
A. [Behe] No, I argued for it in my book.
Q. Not in a peer reviewed scientific journal?
A. That’s correct.
Q. And, in fact, there are no peer reviewed articles by anyone advocating for intelligent design supported by pertinent experiments or calculations which provide detailed rigorous accounts of how intelligent design of any biological system occurred, is that correct?
A. That is correct, yes.
 
Upvote 0

XTE

Well-Known Member
Jun 27, 2006
2,796
113
Houston, Tx
✟3,642.00
Faith
Atheist
Politics
US-Others
laptoppop said:
Bigotry in Science
http://icr.org/articles/print/840/
Examples of problems creationists have with being published
From the article:
”Creationists who publish scientific research in mainstream journals have found that they can publish articles with data having creationist implications, but will not get articles with openly creationist conclusions published. When they attempt to do this, their articles are usually rejected. Those who are well-known to evolutionists as creationists have more difficulty even with articles which do not have obvious creationist implications.
In the summer of 1985 Humphreys wrote to the journal Science pointing out that openly creationist articles are suppressed by most journals. He asked if Science had ‘a hidden policy of suppressing creationist letters.’ Christine Gilbert, the letters editor, replied and admitted, ‘It is true that we are not likely to publish creationist letters.’ This admission is particularly significant since Science’s official letters policy is that they represent ‘the range of opinions received.’ e.g., letters must be representative of part of the spectrum of opinions. Yet of all the opinions they receive, Science does not print the creationist ones.”

http://www.creationresearch.org/crsq.html

I knew I would see this one day.

So far from the truth it is unreal. Another conspiracy story?

If this really were the case, you would see YECs do what they always do, CREATE THEIR OWN PEER REVIEW JOURNAL.

In their never ending attempt at legitimacy, they would do that in a heart-beat!

The reason those papers are not out their by YECs is easy, they do not have a leg to stand on as far as data goes and even if they felt it might be something good, NO ONE WANTS TO PUT THEIR NAME ON THAT VAGUE, BUTCHERED peice of "evidence" for all eternity. Imagine writing a paper you knew people would laugh at in 10 years. Hate to say it, but you might feel like a YEC "Creation Scientist" right now if you did.

I will keep praying for you guys though. :)
 
Upvote 0
This site stays free and accessible to all because of donations from people like you.
Consider making a one-time or monthly donation. We appreciate your support!
- Dan Doughty and Team Christian Forums

Mallon

Senior Veteran
Mar 6, 2006
6,109
296
✟15,392.00
Faith
Lutheran
Marital Status
Private
laptoppop said:
Peer review can take a long time to accomplish.
Meh. If the product that gets spit out the other end of the peer review process is of higher quality than when it went in, the extra time it takes to groom a research paper is worth it.
Peer review is conducted by people.
Who else would you have conduct peer review? Giraffes?
Bigotry in Science
http://icr.org/articles/print/840/

...

Quarterly peer-reviewed creationist journal
http://www.creationresearch.org/crsq.html
Your first link complains that the respected journal Science does not often publish letters from creationists. Your second link points to a "quality" journal to which non-creationists aren't even allowed to subscribe, let alone submit papers or letters to. There seems to be an obvious conflict in logic here. At least Science doesn't limit its subscribers to being athiestic evolutionists.
Bigotry in Science
http://icr.org/articles/print/840/
Examples of problems creationists have with being published.

...

Do Creationists Publish in Notable Refereed Journals?
http://www.answersingenesis.org/docs/538.asp
Authors also detail multiple people who have published non-creationist articles, yet have problem with anything creationist
Another fairly obvious self-contradiction. In one breath, you complain that creationists are unfairly prevented from publishing in peer-reviewed journals, and in another breath you give a list of creationists who have published in such journals. I think it's important to point out that while creationists obviously CAN publish in non-religious, peer-reviewed journals, what don't get published are papers which draw conclusions about the supernatural based on the natural methodology of science. Such conclusions are by definition non-scientific, which is why they never see the light of day.
Submitting a paper on creationism to a science journal is like going to a vegetarian restaurant and ordering a steak.
 
Upvote 0

laptoppop

Servant of the living God
May 19, 2006
2,219
189
Southern California
✟16,420.00
Faith
Non-Denom
Marital Status
Married
Politics
US-Republican
Just a note - in the links section, I am not "presenting" anything - I'm trying to share a collection of links to inform the discussion. I've tried to be reasonably balanced, so it doesn't surprise me in the least that the material in the links would not completely agree with each other.
 
Upvote 0

random_guy

Senior Veteran
Jan 30, 2005
2,528
148
✟3,457.00
Faith
Christian
laptoppop said:
Just a note - in the links section, I am not "presenting" anything - I'm trying to share a collection of links to inform the discussion. I've tried to be reasonably balanced, so it doesn't surprise me in the least that the material in the links would not completely agree with each other.

I really suggest you check out Nature's link on the debate of peer review. Most people won't take your links seriously since they aren't from academic sources (except maybe Brian Martin's link, but I'm still reading it). Nature is one of the most reputable scientific magazine and you'll be able to read a lot of good discussion on the merits of peer review, both pro and con. Not only that, they are currently experimenting with open public peer review (and while papers submitted there won't be featured in Nature) it seems like an interesting idea. Allow the entire world to see your paper and comment on it before you submit it to peer review. When if you get rejected see if the rejection slip matches the comments from other scientists and try to determine if there is a bias.
 
Upvote 0
This site stays free and accessible to all because of donations from people like you.
Consider making a one-time or monthly donation. We appreciate your support!
- Dan Doughty and Team Christian Forums

random_guy

Senior Veteran
Jan 30, 2005
2,528
148
✟3,457.00
Faith
Christian
laptoppop said:
Yes, I've started looking around at the Nature's link. Good stuff. Personally, I like the concept of open review -- its much more like the web -- or even how we form reputations here on the forums!

I like it. However, I think if we combine both of them, we get a lot better system. I envision something like putting the paper on the open web so that you can receive lots of feedback and make corrections, if needed. It may also help disseminate information a lot quicker. After you receive enough feedback, submit it to the peer review group where it goes through the traditional peer review. Traditional peer review is a lot more rigorous so it may catch errors and mistakes that the open process did not. If you get rejected, you can discuss on the open forums whether or not the rejection was for valid reasons.

The only problem I view this is some scientists may be competing with each other to get their paper published first. But I think a two fold system would be a pretty neat idea, especially with the internet being as widespread as it is.
 
Upvote 0
Status
Not open for further replies.