Peer Review, Pt. 1: The Way It Was

Are you ready to discover your college program?

Search Colleges
TheBestSchools.org is an advertising-supported site. Featured programs and school search results are for schools that compensate us. This compensation does not influence our school rankings, resource guides, or other information published on this site.

Part 1 | Part 2 | Part 3 | Part 4 | Part 5


Twitter Icon Tweet this!
Today's scholarly literature is exploding in quantity and deteriorating in quality.
Peer review was first institutionalized by the Royal Society of London in 1665 1. It's basically a good idea. Disinterested experts in a field vet a scholarly contribution and assess the worth of a paper before announcing it worthy. Peer review encourages quality and helps authors to sharpen their work. But peer review is not mandatory to assure quality. Plato's Republic, Euclid's Elements, and Darwin's Origin of the Species had no peer-review.

The assumption that today's peer reviewed paper has been vetted by experts and therefore has been awarded a blue ribbon for excellence is far from the truth. Peer review is not doing its job. Today's collection of scholarly literature is exploding in quantity and deteriorating in quality. After the ailing patient is examined and the diseases diagnosed, maybe we can prescribe some medicine to make it better.

Anonymous Peer Review

Peer review comes in different flavors. One is anonymous peer review, where the identity of the reviewers is kept secret from the author. Albert Einstein only had one anonymous peer review in his career and the paper was rejected 2. This happened in 1936. A decade and a half earlier in 1905, Einstein's annus mirabilis (remarkable year), he published four breakthrough papers. One introduced the world to special relativity 3. Another outlined the photoelectric effect---why metals spit out electrons when illuminated with light 4. He won a Nobel Prize for this once the theory was experimentally verified. A third paper explained that Brownian motion occurred because there were molecules randomly dancing around and bumping into little particles 5. In another paper, he supplied the foundation for the relationship between mass and energy summarized by the familiar equation E = mc2. Physicist Frank Tipler writes

“[All of Einstein's 1905] papers were published in Annalen der Physik, one of the major physics journals in Germany. But none of the papers were sent to referees. Instead the editors—either the editor in chief, Max Planck, or the editor for theoretical physics, Wilhelm Wien—made the decision to publish. It is unlikely that whoever made the decision spent much time on whether to publish. Almost every paper submitted was published. So few people wanted to publish in any physics journal that editors rarely rejected submitted papers. Only papers that were clearly `crackpot' papers—papers that any professional physicist could recognize as written by someone completely unfamiliar with the elementary laws of physics—were rejected.”6

All agree that Nobel Laureates Max Planck and Wilhelm Wien were peers of Einstein. During Einstein's time, Tipler estimates that “one would have to submit [only] three papers on the average to have an even chance that at least one of your papers would be `peer' reviewed by a [past or future] Nobel Prize winner.”7

Anonymous peer review as practiced today has been in force only since the end of World War II when pressure was applied to professors to write papers. The phrase “publish or perish” looks to have been coined soon after the war in 1951 by Marshall “The Medium is the Message” McLuhan 8. Prior to this, Professors were often discouraged from publishing. Karl Popper, one of the most influential philosophers of the 20th century, wrote “I was told that I should be well advised not to publish … and that any time spent on research was a theft from the working time as a lecturer for which I was being paid."9 Many scholars wrote papers in their spare time. Einstein published his first seminal papers while working at a patent office.

Industry bosses, now as then, often discourage their employees from publishing. They either don't want the distraction from the bottom line or don't want trade secrets disclosed. A great example is William Sealy Gosset, who worked at the Guinness Brewery in Dublin, Ireland. Gosset developed an important statistical model for small sample sizes. He was obliged to hide his affiliation and published his paper under the pseudonym “Student”10. The Student's T distribution is now taught in all introductory statistics classes.

For professors, things changed after WWII. There was a mandated explosion of scientific publication. If Einstein's one paper led to the atomic bomb that ended WWII, think what a thousand papers could do! Quality soon took a back seat to quantity. The broadening of scientific inquiry and the new glut of papers necessitated revising paper reviewing to an assembly line process.

Peer-reviewed papers are commonly thought to be vetted to ensure quality. The paper is assumed to have been thoroughly reviewed anonymously by disinterested top people in the field. After careful consideration, the peer-reviewed paper is deemed to be of sufficiently high quality to warrant publication in a scholarly journal that will be archived forever. Archiving used to be done in dusty tombs on the upper floors of University libraries. Now, archiving is done on the net.

When I write recommendations for students, I require they waive their right to read said recommendations. This allows me freedom to be open and candid. I was once on the Ph.D. committee of a slovenly and morbidly obese student. We'll call him Tim. Tim's t-shirts were stained and too small. His bare naked belly often spilled out under his shirt and hid his belt. (I think he wore a belt.) But scholastically, Tim was brilliant! In my recommendations, I told Tim's prospective employees of his gifted mind. But I also, diplomatically wrote “Beware. [Tim] is definitely not a slave to fashion.” Tim's waiver of access to my recommendation allowed me to both overly gush about his brilliance and openly comment on how he presented himself. The goal of anonymous peer review is similar. If I review your paper, I can be open and honest about what I think without the distraction of worrying about what you think about what I say.

But there's a consequence to anonymous peer review: anonymity diminishes accountability. Sloppy and inept reviews today often eclipse thoughtful in-depth reviews. The reality is nicely captured in a quote from physicist Frank Tipler 11.

“[Today's] 'peer' review is not peer review: the referee is quite often not as intellectually able as the author whose work he judges.”

Tipler is spot on.

The Peer Review Procedure

Here is the innocent sounding procedure followed today if you want a paper published in a scholarly journal.

Three Tiers for Peer-Review

There are three administrative tiers in the peer review process: the Editor-in-Chief, the Associate Editor, and the reviewers. The Editor-in-Chief and Associate Editor are titles with high academic currency. When looking at the resumes of prospective professors, Deans and Provosts interpret these positions to indicate that one's status in the field is substantive. Although it takes the most time when done right, the reviewing of a paper pays less than minimum wage. Unless mandated, Editor-in-Chiefs and Associate Editors rarely look at a paper in detail. They view themselves as judges waiting for the decision of the jury. A lot of time is needed to understand a paper and write a thoughtful and thorough review. Someone who would be a good reviewer can always earn more academic currency by spending their time writing their own papers or, better yet, composing a well-polished grant proposal asking for money to fund their own research.

All of the recognition one gets for reviewing a paper is a single pathetic line on the resume. When the time for raises, promotions and tenure decisions come up, the reviewer line buys the academic almost nothing. It's worse for researchers in industry. Your manager may fairly inquire, “How does your review of this paper help us sell more widgets?” The motivations for writing an in-depth review of the paper are fulfilling a sense of good citizenship, or on some occasions, satisfying an interest in the paper's content.

If leading researchers in a field don't do the reviews, who does? Often an Associate Editor will take whoever he can get. Even when a top tier reviewer is recruited, anonymity can lead to unaccountability, which can lead to shallow reviews. Only the Associate Editor will know you did a lousy job. On more than one occasion, reviews of my papers appear as if the reviewer hadn't even read the manuscript. Authors spend hours writing and polishing papers only to be insulted by review from a lazy reviewer. In other instances, a researcher may outsource review to a graduate student with the following instructions: “Read this paper, write a review and then let's talk. If I like what you've written, I'll put my name on it, mention you in some way, and submit your assessment as the review.” I know this is true since I've done it.

Peer review is popularly viewed as a stamp of approval on a scholarly publication. It becomes less impressive when we look at who is brandishing the stamp.

Part 1 | Part 2 | Part 3 | Part 4 | Part 5

____
Robert J. Marks II, Ph.D. is a Distinguished Professor of Electrical & Computer Engineering at Baylor University in Waco, Texas. The material in this column, though, does not necessarily represent the views of and has not been reviewed or approved by Baylor University.

Notes

[1] Select Committee on Science and Technology Written Evidence, The United Kingdom House of Commons, "The Origin of the Scientific Journal and the Process of Peer Review."
Back to passage

[2] Kennefick, Daniel (September 2005). "Einstein versus the Physical Review". Physics Today 58 (9): 43.
Back to passage

[3] Einstein, Albert (1905). "Zur Elektrodynamik bewegter Körper". Annalen der Physik 17 (10): 891–921.
Back to passage

[4] Einstein, Albert (1905). "Über einen die Erzeugung und Verwandlung des Lichtes betreffenden heuristischen Gesichtspunkt". Annalen der Physik 17 (6): 132–148.
Back to passage

[5] Einstein, Albert (1905). "Über die von der molekularkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen". Annalen der Physik 17 (8): 549–560.
Back to passage

[6] Tipler op. cit.
Back to passage

[7] ibid.
Back to passage

[8] Eugene Garfield (June 1996). "What Is The Primordial Reference For The Phrase 'Publish Or Perish'?". The Scientist 10 (12): 11. Garfield references M.Molinaro, C. McLuhan, W. Toye, eds "Letters of Marshall McLuhan," Oxford University Press, 1987.
Back to passage

[9] Karl Popper, Unended Quest: An Intellectual Autobiography, Updated Ed. (London and New York: Routledge, 1992), 136.
Back to passage

[10] Student [William Sealy Gosset] "The probable error of a mean". Biometrika 6 (1): 1–25. (March 1908).
Back to passage

[11] Tipler, F. J. "Refereed journals: Do they insure quality or enforce orthodoxy?" International Society for Complexity, Information and Design (2006).
Back to passage

Take the next step towards your future with online learning.

Discover schools with the programs and courses you’re interested in, and start learning today.

Woman working at desk