Peer Review, Pt. 3: Towers of Mostly Babble

Are you ready to discover your college program?

Search Colleges is an advertising-supported site. Featured programs and school search results are for schools that compensate us. This compensation does not influence our school rankings, resource guides, or other information published on this site.

 Part 1 | Part 2 | Part 3 | Part 4 | Part 5

There are two activities that rank high in academic currency: publications and attracting external funding to support research or academic programs (particularly those which supply lots of overhead money for your university). The law of supply and demand has spurned a glut of journals. There are also conferences where papers are published in conference records. Both the scholarly journals and the conferences post peer-review guards at the gate with the supposed goal of blocking the entry of unworthy papers. The collection of the papers deemed worthy is often referred to as “the literature.”

The number of authored papers often becomes a glittering merit badge worn by a researcher. Quantity is easier to assess and communicate than quality. Two decades ago, I declared1 “Pure publication quantity today has become a meaningless metric. One can publish almost anything.” I have since revised my opinion. Today I would remove the word “almost.” But no one listened to me and paper counting is now more deeply entrenched in academia than ever.

Authors are often asked to write short autobiographies in the third person at the end of their papers. In these biographies we often read self-congratulatory phrases like “Dr. Pythagoras is the author of over 500 journal and conference papers.” This is like saying “Dr. Pythagoras pounded 500 nails into various types of lumber.” The pounding of the nails is unimportant. It's what you've built that counts.

I once found myself in a discussion about publication count with a newly minted acquaintance at a neural network conference in Japan. He asked me how many publications I had. I told him. He looked at me with near disgust.

“Geesh,” he muttered. “That's nothing!”

He then told me how many publications he had. It was a lot more than me. This was 20 years ago. I have lost track of my friend, the publishing machine. I can tell you, though, that none of his papers has made any impact in the field of neural networks. Such worthless papers are called “write-only articles,” which is funny if you know what ROM stands for. I'm unsure whether my friend continues to publish today. Chances are his well-endowed paper count caused him to be promoted to an administrative position where he now evaluates faculty by counting beans. Final faculty promotion and tenure decisions are more often than not made by those who lean heavily on bean tallies. “The Dean can't read, but the Dean can count.”


And I confess that, yes, I report my publication count in my short third person autobiographies, but usually with a hedge. In a recent paper, my bio reads2 “[Dr. Marks] is the author of hundreds of journal and conference papers. Some of them are good.”3 And it's true! In fact, a few of them are very good.

So what is the impact of the ranking paper quantity over paper quality? Unsurprisingly the answer is: more papers. In the December 1994 issue of Scientific American, a figure labeled “Tower of journals” shows a stack of papers from the MEDLINE bibliographic database in the area of biomedical research.4 The stack of papers is placed next to the Washington Monument. The stack of papers is taller.

I looked at the trend since 1994 from an updated list of MEDLINE data I found online.5 As you can see from my plot in the Figure 1, the stack gets taller and taller each year and looks to literally be increasing exponentially. To place the sheer number of publications in perspective, some familiar landmarks are pictured in the background.6 The numbers are astonishing. Over 1.1 million scholarly publications in 2013! On average, MEDLINE lists two biomedical scholarly papers every minute, 24/7. As you can see in the figure above, the stack of papers now is almost as tall as the Empire State Building shown on the right. And MEDLINE only lists the journals it considers reputable. So the true numbers are larger.

But maybe we are being unfair. The biomedical field is broad and there is no way one can keep up on all areas of the research. So I did a tally of papers in computational intelligence.7 This is a favorite research area of mine and I have published a lot of papers in the field. (Some of them are good). Using the Scopus8 database, I generated the figure below with the giraffe in it. At first, this graph looks a lot better than the MEDLINE graph. Although buildings are too tall to put in the figure, we can include a six-foot man and a giraffe for perspective. But closer consideration reveals the problem has not gone away. In 2012 there were 53,446 papers published in the area of computational intelligence. That's 146 papers each day or six papers per hour. That's a lot less than MEDLINE's two papers per minute, but is still unmanageable. No wonder I find it harder and harder to keep up with the current developments in my field.


As shown in Figure 3, I performed an exponential regression on the MEDLINE data and got an R2 factor of 0.9834, which is decent. Exponential increases can be alarming.9 Notice that during recent years, the number of publications lie significantly above the exponential regression curve. Things look like they are getting worse at an ever increasing rate.10

If paper hard copies of all of today's publications were necessary, university libraries could not contain them. Many of today's publications are somewhere in the cloud and available only on the web.

And thank goodness! We have avoided an environmental calamity! George Kaub published a classic satirical paper in the Journal of Irreproducible Results entitled "National Geographic, the Doomsday Machine"11 prophesying global chaos because of the increase of print publications. In 1974, Kaub was concerned that accumulation of heavy glossy paper used by National Geographic Magazines was amassing so fast that catastrophe was imminent. He wrote

“These threats to our environment, our health and our mental wellbeing are real and with us, but not nearly as immediately catastrophic or totally destructive as the disaster which imminently faces this nation… [This] menace of monstrous proportions can be likened only to the entire country resting on a gargantuan San Andreas fault. Earthquakes, hurricanes, mud slides, fire, famine, and atomic war all rolled into one hold no greater destructive power than this incipient horror which will engulf the country in the immediate and predictable future.
“This continent is in the gravest danger of following legendary Atlantis to the bottom of the sea. No natural disaster, no overpowering compounding of pollutions or cataclysmic nuclear war will cause the end. Instead, a seemingly innocent monster created by man, nurtured by man, however as yet unheeded by man, will doom this continent to the watery grave of oblivion.

“But there is yet time to save ourselves if this warning is heeded.

“PUBLICATION AND DISTRIBUTION OF THE NATIONAL GEOGRAPHIC MAGAZINE MUST BE IMMEDIATELY STOPPED AT ALL COSTS! This beautiful, educational, erudite, and thoroughly appreciated publication is the heretofore unrecognized instrument of doom which must be erased if we as a country or continent will survive. It is NOT TOO LATE if this warning is heeded!”

Kaub's inspired doomsday prediction reminds me of the dire warnings we have today about manmade global warming and the self-sacrifice we must all endure. The difference of course is that today's warnings are serious.

The accumulation of National Geographic Magazines would be dwarfed by the mountainous pile of paper copies formed from all of today's so-called scholarly publications. We are all thankful to the cloud for making paper copies no longer necessary. It has allowed us professors to publish an ever increasing number of papers12 while successfully derailing Kalb's forecast of impending environmental doom.

 Part 1 | Part 2 | Part 3 | Part 4 | Part 5

Robert J. Marks II, Ph.D. is a Distinguished Professor of Electrical & Computer Engineering at Baylor Univesity in Waco, Texas. The material in this column, though, does not necessarily represent the views of and has not been reviewed or approved by Baylor University.

[1] Professor Bing Sheu,"Neural Networks and Beyond-An Interview with Robert J. Marks," IEEE Circuits and Devices Magazine, Volume 12, Issue 5, 1996 [DOI 10.1109/MCD.1996.537355]
Back to passage

[2] Baylis, Charles, Josh Martin, Matthew Moldovan, Robert J. Marks, Lawrence Cohen, Jean de Graaf, Robert Johnk, and Frank Sanders. "Spectrum Analysis Considerations for Radar Chirp Waveform Spectral Compliance Measurements." Electromagnetic Compatibility, IEEE Transactions on 56, no. 3 (2014): 520-529.
Back to passage

[3] I also use similar language in writing my short autobiographical introductions for talks. Audiences seem to appreciate the honesty. Here is a YouTube video showing such a reaction.
Back to passage

[4] Ralston, Rick, and Carole Francq. "Subscription statistics for collection and budget decisions." Indiana Libraries 14, no. 3 (1995): 65-71.
Back to passage

[5] MEDLINE data base
Back to passage

[6] I'm proud of the way I calculated the height of the background landmarks in the figure. I'm sitting in my office without a ruler wondering how thick the average scholarly paper is. Googling webpages on the thickness of paper didn't give any decisive answers, so I did a nerdy engineering thing. I got a dollar bill from my wallet and googled that a dollar bill measures 2.61×6.14 inches. I folded the dollar bill lengthwise. That's 1.305 inches. When placed next to a copy of Elements of Information Theory, second edition by Thomas M. Cover, and Joy A. Thomas, there were 626 pages in the 1.305 inches. My conversion rate was thus 626/1.305=479.69 journal pages per inch. This is the figure used to calculate the height of the landmarks in the figure assuming the average paper is six pages in length. Compared to the December 1994 Scientific American graphic, the results are conservative. Scientific American showed the stack of papers being taller than the Washington Monument in 1994. In Figure 1, the Washington Monument is taller.
Back to passage

[7] For computational intelligence I use the following search words : machine intelligence , artificial intelligence, computational intelligence, data mining, neural network and fuzzy. I performed a similar search over 20 years ago where similar phrases were used: R.J. Marks II, "Intelligence: Computational Versus Artificial," IEEE Transactions on Neural Networks, vol. 4, p 737 (September, 1993)
Back to passage

[8] Scopus
Back to passage

[9] Historically, exponential increases are not as alarming as they first appear. They are rather the start of an S-shaped sigmoid curve that will soon level off. We might be seeing this in the figure showing computational intelligence publications where there looks to be a leveling off in the last few years.
Back to passage

[10] I often have to examine such phrases and sometimes think what the phrase means is different than what the author meant. For those sufficiently nerdy and care, here is what I mean. “Getting worse” means the publication count is getting larger. Let p(t) denote the number of papers per year as graphed in Figure 1 where t is the year. By publication rate in papers per year, I mean dp/dt. (A constant rate would mean there were the same number of publications each year.) Thus “at an ever increasing rate” means that dp/dt is ever increasing or, equivalently, d2p/dt2>0 So the phrase could have also been written “the yearly publication count is accelerating.” Maybe I'll write a scholarly paper analyzing such phraseology. See how easy it is?
Back to passage

[11] Kaub, George H. "National Geographic, the Doomsday Machine." The Journal of Irreproducible Results 20 (1974): 22-23.
Back to passage

[12] See endnote #9.
Back to passage

Take the next step towards your future with online learning.

Discover schools with the programs and courses you’re interested in, and start learning today.

Search Colleges is an advertising-supported site. Featured programs and school search results are for schools that compensate us. This compensation does not influence our school rankings, resource guides, or other information published on this site.
Woman working at desk