Peer Review, Pt. 5: Artificial Unintelligence: will journals accept papers written by... a computer?

Are you ready to discover your college program?

Search Colleges
TheBestSchools.org is an advertising-supported site. Featured programs and school search results are for schools that compensate us. This compensation does not influence our school rankings, resource guides, or other information published on this site.

 Part 1 | Part 2 | Part 3 | Part 4 | Part 5

We know that real artificial intelligence has arrived when a computer can write soul stirring poems, compose grand music, pen engrossing novels and, yes, write scholarly journal papers. We're halfway there. There are free sites on the web that automatically write scholarly papers for you. The first, I believe, was SCIgen.1 Not surprisingly, SCIgen was written by computer science students at MIT. Go to the site. Enter up to five authors and hit Generate. A nice web formatted paper appears. Click on PDF and there's your paper ready to submit to a scholarly computer science journal. SCIgen automatically draws figures and references for your paper. The references typically contain fictitious papers where you are assigned authorship along with other notables such as Albert Einstein and Charles Darwin.

The computer generated SCIgen papers remind me of the pod copies in the 1956 black and white science-fiction classic Invasion of the Body Snatchers. Pods from outer space are placed next to sleeping humans. The pods take on human form and, during the copying process, the human dies. To an outsider at a distance, the finished pod copies look like humans. Interaction and closer inspection reveals they are phonies. Close relatives are able to notice the difference more quickly. The SCIgen papers are like this. They initially look pretty good. But a closer inspection reveals they are phony. And if computer science is your field, the phoniness is recognized more quickly.

Should we be surprised that phony papers generated by SCIgen have been accepted by conferences and journals? The pressure to publish has been applied to professors almost everywhere. Supply and demand dictates that journals and conferences be created to meet the demand. Many of these conferences and journals, motivated by profit, are not picky about the quality of the papers they accept. They are more interested in collecting fees. Although I'm not a big fan of peer review as it is currently practiced, there always needs to be a gatekeeper to ban entrance of garbage trucks.

A phony paper written by the computer program SCIgen was accepted at the ninthWorld Conference on Systematics, Cybernetics, and Informatics (WCSCI) in Orlando, Florida. After accepting the paper unreviewed, the WCSCI organizers found the SCIgen webpage where the paper's authors announced their triumph and were soliciting donations to allow them to travel to the conference and present the paper. After the discovery, the conference wrote “… since you gave the information in your web page that the paper was a fake one, we think we should not accept your registration even if you have total responsibility on the content of your paper (as a non-reviewed one).” Whatever that means.

There are quality controls on good journals and on journals that are trying to be good. Some journals do a better job than others. Every journal in my professional society, the IEEE, is considered quality and is good academic currency for promotion, tenure, and raises. For biomedical research, The National Library of Medicine controls the PubMed journals indexed by MEDLINE. The government bureaucracies have a tendency to maintain inertia and adding new journals to PubMed is a slow process. Harvard researcher Mike Shrime notes2 “If you want to find a reputable journal, you'd turn to PubMed, but the problem is that there are also many reputable journals that are not on PubMed.” Another source of papers is Google Scholar.3 Google Scholar includes all journals in its data base. But there are so many junk journals, you often must search through a lot of trash to find a treasure.

An apparent rebel, Shrime submitted an obviously phony computer-generated journal article to 37 junk journals. Seventeen of them accepted the paper for publication. The article, titled “Cuckoo for Cocoa Puffs,” didn't even go through an automatic paper generator like SCIgen. SCIgen at least makes a lame attempt at sounding legit. Shrime simply used a random word generator available on the web that usually fails to even write complete sentences.4 His paper begins:

“In an intention dependent on questions on elsewhere, we betrayed possible jointure in throwing cocoa. Any rapid event rapid shall become green. Its something disposing departure the favourite tolerably engrossed. Truth short folly court why she their balls. Excellence put unaffected reasonable introduced conviction she.”

Even though the word “cocoa” looks thrown in, it's all random gibberish. Shrime's designated authors were twentieth century media icon Orson Welles who scared the world with his War of the Worlds radio broadcast (1938) and then went on to write and star in Citizen Kane (1941), a movie most movie critics include on their all-time top ten lists. Welles died in 1985. The other author of Shrime's paper was Pinkerton A. LeBrain. Although the paper was accepted, it never went to press because Shrime refused to pay the required $500 processing fee.

Like Shrime, engineer Alex Smolyanitsky generated a phony journal paper. He chose Edna Krabappel, Kim Jong Fun and Maggie Simpson as the authors for his random text-generated paper titled “Fuzzy Homogeneous Configurations.”5 The middle author looks to be heir to a North Korean dictatorship and the other two are cartoon characters from The Simpsons. The paper was accepted by both the Journal of Computational Intelligence & Electronic Systems and The Aperito Journal of NanoScience Technology. Smolyanitsky continues to receive invoices from the journals for a $459 publishing fee.

SCIgen generates phony computer science papers. Another paper generator6, Mathgen7, specializes in phony mathematics. Fictitious author Professor Marcie Rathke had Mathgen produce the paper “Independent, Negative, Canonically Turing Arrows of Equations and Problems in Applied Formal PDE”. It was accepted for publication by the impressive sounding journal Advances in Pure Mathematics8. The editor's acceptance letter begins:

“Thank you for your contribution to the Advances in Pure Mathematics (APM). We are pleased to inform you that your manuscript … has been accepted. Congratulations!”

I'm not sure how a computer generated paper can be effectively reviewed, but the editors of this junk journal actually gave it a try. Remember while you read, the paper has been accepted. All it needs is some fine-tuning. In the review, the editor writes “We can't catch the main thought from this abstract.” This cracked me up! For some reason, the editor could not catch the main point of pure gibberish. He continues. “In this paper, we may find that there are so many mathematical expressions and notations. But the author doesn't give any introduction for them.” That's because no explanation for the paper exists.

Publication of Rathke's paper was again marred by the requirement of a $500 processing fee.

Phony SCIgen papers have also made it into reputable conferences. Springer, a large German based publishing house, was recently informed that a number of papers published in its edited volumes were generated by SCIgen.9 The authors of the phony journal papers had a different motivation than did undercover pranksters Shrime and Smolyanitsky. At the end of the day, publishing – once the crowning accomplishment of all aspiring academics – has descended into little more than an elaborate game of comparing who has more beans. The Springer authors were legitimately (or illegitimately as it were) trying to make their piles of beans higher. Springer contacted the authors of the SCIgen papers who “confirmed that their submissions were not intended as a hoax. The intention seems to have been to increase their publication numbers and to increase their standing in their respective disciplines and at their institutions.”10 Springer has retracted the papers, but you can still reference them in your papers if so inclined. An example of a Springer published SCIgen paper is:

Sun Ping. “Application of Amphibious Technology in the ReutoMail.” In Proceedings of the 2012 International Conference on Communication, Electronics and Automation Engineering, pp. 409-413. Springer Berlin Heidelberg, 2013.11

By simply listing the paper here, the citation count of Ping's paper as tallied by Google Scholar will increase by one. From the perspective of pure bean counting, Springer's publication impact factor will correspondingly increase a bit and Ping's h-index may also be helped. I'm uncertain whether or not Ping's paper is still included in Springer's book or not. If curiosity gets the better of you, and you have a few hundred dollars to spare, you can find out by buying a used copy on Amazon12 If Ping's paper is still in the book, let me know. I'd love to see a copy.

In all fairness though, compared to the large number of papers published by Springer, the number of SCIgen papers is small. To its credit, Springer is trying to scrub off some of the mud by taking steps to make sure publication of phony papers doesn't happen again.13 However, it's hard to ignore the fact that these papers made it through a supposedly “rigorous evaluation process” in the first place. It should rightfully make one wonder what kinds of papers Springer would reject. In the absence of reading the actual papers, the only foreseeable criteria remaining is ideology (i.e., sifting out dissenting ideologies, but only if they are obvious in the title, author, or abstract.) This too is an unfortunate reality within peer review.

One of supply side academic's prime currencies remains publication count. In promotion and tenure cases, “The Dean can't read, but the Dean can count.” One would think the clear case of bean abuse would enlighten academic administrators. It hasn't. They have instead decided to abuse different beans.

We'll talk more about this soon.

 Part 1 | Part 2 | Part 3 | Part 4 | Part 5

____
Robert J. Marks II, Ph.D. is a Distinguished Professor of Electrical & Computer Engineering at Baylor Univesity in Waco, Texas. The material in this column, though, does not necessarily represent the views of and has not been reviewed or approved by Baylor University.

[1] SCIgen - An Automatic CS Paper Generator
Back to passage

[2] Elizabeth Segran, "Why A Fake Article Titled `Cuckoo for Cocoa Puffs?' Was Accepted By 17 Medical Journals" Body Week
Back to passage

[3] Google Scholar
Back to passage

[4] Random Text Generator
Back to passage

[5] Fiona MacDonald, "A study by Maggie Simpson and Edna Krabappel has been accepted by two scientific journals," Science Alert, Dec 10, 2014
Back to passage

[6] I haven't kept up with all of the automatic document generators, but one I found an interesting is site that generates SBIR proposals. These are proposals to the United States government to support joint research by professors and small businesses. If one of these proposals gets funded by the government, I will be both impressed and depressed. http://www.nadovich.com/chris/randprop/
Back to passage

[7] Mathgen
Back to passage

[8] Mathgen paper accepted!: September 14, 2012
Back to passage

[9] "Springer fake paper tally up to 18" Retraction Watch.
Back to passage

[10] Ibid
Back to passage

[11] http://link.springer.com/chapter/10.1007%2F978-3-642-31698-2_58
Back to passage

[12] George Yang, Editor. Proceedings of the 2012 International Conference on Communication, Electronics and Automation Engineering, pp. 409-413. Springer Berlin Heidelberg, 2013. On Amazon:(Accessed March 30, 2015). On Springer's web site:(Accessed March 30, 2015).
(I think the high cost of the books is like a processing fee except paid by the reader.)
Back to passage

[13] My professional society, the IEEE, is also guilty of publishing SCIgen papers. Right now, I'm more interested in writing about Springer.
Back to passage

Take the next step towards your future with online learning.

Discover schools with the programs and courses you’re interested in, and start learning today.

Search Colleges
TheBestSchools.org is an advertising-supported site. Featured programs and school search results are for schools that compensate us. This compensation does not influence our school rankings, resource guides, or other information published on this site.
Woman working at desk