This is the first of a multi-part series on Peer Review. We’re starting right at the beginning, delving into its very origins, and finding out how it came to truly dominate the scientific discourse.
Let’s start with a simple story. Once upon a time, there was a man who wrote about the world. He would take his observations, and pass them to his followers. His followers would memorise and repeat his teachings to others. Thus, the knowledge would spread further, benefiting many people.
This story fits most ancient philosophers, like Galen and Aristotle. Their schools of thought would dominate their respective fields. It was an era when literacy was rare, and there was no printing press. Their students had to spend much time copying and memorising the work.
One could read these ancient philosophers and think :-
- “Wow, they were keen observers. What would happen if I went out and did the same thing? Would I see the world in the same way ?”
- “Wow, they were keen observers. There is nothing I can add. Checking their work would be disrespectful. Let’s see what new things we can do.”
You can understand why a lot of scholars chose the second path. As far as they were concerned, the experimental work was over. All that was left was putting all that work together. This was the medieval culture of disputation.
Debates were had, but they weren’t related to evidence. They were more concerned with rooting our inconsistencies, and unifying classical thought into one whole. The ideas of the Bible, Aristotle and Galen were strictly off limits. Any new idea proposed had to conform, and woe betide any that didn’t.
The Seeds of Dissent
“Throughout modern times, practically every advance in science, in logic, or in philosophy has had to be made in the teeth of the opposition from Aristotle’s disciples” Bertrand Russell
The medieval scholastic system relied on people like Aristotle being right all the time. Which was a problem, because Aristotle made some mistakes. Some were understandable errors, such as the believing that the sun revolves around the earth. Others, less so. Such as men having more teeth than women, and that heavy objects fall faster than lighter ones. These ideas had become enshrined, making them difficult, and even dangerous to dispute.
Which is why Copernicus kept his Heliocentric theory to himself for most of his life. It ran in direct opposition to the accepted dogma. He only released his work on the year of his death, when he couldn’t reap any of the consequences. That duty would fall to Galileo.
Galileo staunchly supported Copernicus’s ideas. He got into nasty squabbles with other scholars. Some of them would be converted by the evidence, but others reacted differently. They reported him to the Inquisition. He was ordered to stop promoting the heliocentric theory, and all the books promoting it were banned. He would die under house arrest.
Meanwhile, Andreas Vesalius courted controversy when he published his book on human anatomy. The book raised hackles. Galen had never dissected a real human, and got a lot of things wrong. Vesalius aimed to correct those errors, and was attacked for it. His rivals demanded that the Holy Roman Emperor punish him. Which wasn’t a wise move, because, it just so happened that Vesalius was the Emperor’s doctor. They were friends. The Emperor held an inquiry, but it was a foregone conclusion.
Bit by bit, more problems came to light. The old philosophers were soon shoved from their pedestals. The new generation of thinkers would replace them with… no-one.
On the Word of No-One
This new era of science owed a significant debt to one man, Francis Bacon. He wasn’t what you’d expect form a traditional scientist. A career politician, diplomat, advisor to the king, and in his spare time, a scientist. He set out to change the way science was made. His book, the Novum Organum, was his manifesto.
Like Aristotle, he believed that scientific ideas should stem from observations. But he realised that our immediate observations aren’t always true. Our senses limit us. False ideas can cloud our judgement. To understand the universe, we must overcome these limitations. No idea is sacred, no one immune from criticism.
Bacon’s ideas were soon the subject of living room discussions, coffeehouse arguments and letters flying across Europe. Scientists performing experiments began to share ideas more freely with each other.
This was “Peer Review”, at its simplest. It’s simply asking for the perspective of someone else who understood the work. Their criticism of the evidence would help to iron out any flaws, and make the work better. .
It was this hive of debate that birthed scientific societies across the world. The Royal Society was the most successful, possessing a charter from the king himself. Their motto embodied the new attitude of science.
“Nullius in Verba”. On the word of No-one.
It was the Royal Society who, in 1665, published the first scientific journal, “Philosophical Transactions”. It worked like any other magazine of the time. The editors published whatever they wanted to publish.
It’s purpose was to disseminate exciting news to scientists who wanted to hear it. Details of experiments and new observations filled its pages. It was to become the medium through which the great scientific debates of the day were had. Soon, more journals would crop up to capitalise on its success. Science was getting bigger and more complex. That would become a problem for journal editors. How did they handle papers that they themselves didn’t understand ?
The First Form of Peer Review
“Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author”
So goes preface to the 1731 issue Medical Essays and Observations. It describes the Royal Society of Edinburgh’s solution. The society already had members from many different scientific disciplines. So why not use them ?
It was a good plan. Specifically picking out the specialists to review work in their own field was obvious. As was ensuring their anonymity. Even then, people believed the shield of anonymity was essential for an honest critique.
This pre-publication review was not like today’s system. The main concern for these journals was picking out the most exciting story. They did not care whether what they reported was true, or even if it made sense.
If something erroneous got published, all responsibility fell on the author. People who disagreed with an article were given the chance to publish rebuttals. So if there was an argument to be had, it would play out publicly in the pages of the journal. if something was wrong, those errors and the fault of the author would be exposed for everyone to see.
This is why scientific conferences and informal correspondence were (and remain) so important. They act as informal testing grounds for new ideas. Scientists can show their research to their peers, and get feedback on it before embarrassing themselves in the pages of a journal.
Charles Darwin was famously reticent in publishing his work. He spent years collecting evidence, and sending letters to solicit opinions from his peers. So he only published when he was absolutely sure he’d addressed all the problems. Even then, it took some coaxing for actually go public.
He finally got his paper on evolution read before the Linnean society. There were none of the arguments and controversy he was expecting. This was perhaps a sign that his work was ready to be published. But, it was said that the main reason for that was that much of the audience were asleep. So this wasn’t exactly a watertight system. But the scientific discourse was still evolving.
Recruiting outside experts
Journals still had problems. Even the talent they could recruit from their own editorial boards wasn’t enough to keep up. Science was still becoming more complex. Editors would have to comprehend more and more obscure areas of science, often beyond their expertise.
The British Medical Journal was the first to reach out to experts working at the forefronts of science. These scientists had an intimate understanding of their field. In theory, that meant that they were the best equipped to gauge the merit of any new study.
But this came with its own set of problems. Science is a highly competitive field, and these new recruits weren’t always impartial when assessing their colleagues and competitors work. BMJ Editor Ernest Hart highlighted this in an 1893 lecture.
“It is a laborious and difficult method, involving heavy daily correspondence and constant vigilance to guard against personal eccentricity or prejudice or – that bugbear of journalism – unjustifiable censure.”
These are problems people still complain about to this day. Expert Peer Review depends strongly on the competence and honesty of the Peers reviewing. Nevertheless, the BMJ had kickstarted a revolution. They were no longer the medium to communicate science. They had become its gatekeepers.
This new system wasn’t initially popular. It was the era before photocopiers. Each submitted article had to use expensive carbon copies before being sent off for review. It was a lot of work to put into an article that may never be published. Most journals avoided it.
Not all scientists were thrilled with this new system either. In 1936, Albert Einstein had come to doubt his early ideas on gravitational waves. When he and his colleague Nathan Rosen tried to model them, they discovered some problems. They thought they had stumbled across proof that they didn’t exist. They submitted their findings to the Physical Review.
To Einstein’s dismay, the manuscript was returned . What’s worse, it came back with comments from a peer reviewer. Without reading the comments, he flew into a rage and penned this letter to the Physical Review:
We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the—in any case erroneous—comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.
P.S. Mr. Rosen, who has left for the Soviet Union, has authorized me to represent him in this matter.
In the process of submitting this paper to a different journal, his colleague, Howard Percy Robertson, politely drew Einstein’s attention to some errors in the paper. Einstein realised his mistakes, and walked back his previous assertion that gravitational waves didn’t exist.
The funny thing is, had Einstein read the comments from the Physical Review, he would have come to the same conclusion a lot earlier. In fact, if he had been truly observant, he might have realised that those comments had been penned by none other than Howard Percy Robertson.
Even the greatest of minds can make the occasional error. That’s why Peer Review is necessary. Peer review allowed scientists to sharpen their work before it reached the public eye. What’s more, it meant readers could put more trust in these articles.
Soon, those readers would demand the “Peer Review” stamp of approval from more journals. Not only had the invention of the photocopier made Peer Review cheaper, but other trends were making Peer Review incredibly attractive for journals.
Post-war economies invested more into research. As more scientists graduated, more of them needed to publish work. Journals were no longer worried about filling empty space. Now they had the opposite problem. They needed a fair way to decide which articles to publish. Expert Peer Review provided the answer.
Nature finally caved in 1967, and the rest of the scientific literature followed. By the 1980’s, “Expert Peer Review” had won the day. It now forms the backbone of science as we know it.
But our story doesn’t end there. Not by a long shot. In implementing this new system, Journals had made an implicit promise. That the work within their pages was scientifically valid and reliable. How well did they keep that promise ?
I’ll be exploring that question next Wednesday. Stay tuned !
References and further reading