Back of the Envelope

Joe Carter at the Evangelical Outpost is asking whether the entire Peer Review process is irrelevant. Being a science type myself (I'm technically an Electrical Engineer who does Physics, so I have papers both in Physical Review and IEEE Transactions), I feel that I should be defending peer review. I'll at least put together a list of what peer review does (or tries to do), and let you decide whether it's useful or not:

• It raises the threshold for what papers people submit. Without the controls of peer review, a lot more papers would be published than currently are. A lot of papers are never submitted because the authors decide that they probably aren't good enough. This screens out a lot of junk, perhaps more effectively than peer review itself does.


• It establishes a hierarchy of papers. What journal a paper is published in depends not only on its subject matter, but how good it is. Papers which are groundbreaking (in my field) may be published in Science or Nature. Important work that doesn't rise to the level of Science or Nature are submitted to Physical Review Letters. Solid research of less import goes to Physical Review B. Papers which have trouble getting into Physical Review may need to shop around the less prestigious journals. This allows my colleagues to pick up one of these journals with some expectation of what quality to expect when they read through it. Since there are so many papers produced each week, they need some way to decide which articles are worth reading. There are all sorts of caveats to this, though. A lot of important papers fall in the lower ranks simply because they are too long--the top ranks of journals prefer shorter articles. Science and Nature both publish a wide variety of work, but it's not all inclusive, and many categories of research just don't fit into the eclectic mix covered by these journals. And, of course, the sociological factors come into play--well respected authors have a better chance of publishing in the higher journals, and you sometimes end up with a bad reviewer or editor who doesn't appreciate the work.


• It provides feedback before publication. A good reviewer not only reads the article carefully to provide a yay or nay to the publishing question, he also finds mistakes and poor explanations and provides that feedback to the author. This allows the author to correct his mistakes and explain his data and analysis more clearly, so that when the paper is finally published, it is of a higher caliber than when it began. I consider this feedback invaluable. It's difficult to overstate the importance of having someone familiar with your field but not part of your research team look at your paper and ask the right questions before it is exposed to the public--er, that part of the public which reads the publishing journal, anyway.


• It successfully filters out a lot of junk. There is a lot of bad research being done. I think it's good that a lot of it never sees the light of day. Granted, a lot of it does see the light of day when the peer review process fails, and a lot of good work doesn't see the light of day when the process fails in the other direction. An imperfect filter isn't useless, however.

Well, there's my defense of peer review. It's not perfect, and I think Joe Carter does a good job of condemning the ideological bias which creeps its way into peer review (I do some slamming of ideological bias myself here). However, considering the sheer volume of scientific papers, some sort of filter has to be applied. Letting each and every paper submitted stand equally before the scientific community to rise or fall on its merits is a nice idea, but impractical.