Last week we saw how bad studies can get published when scientists commit fraud. Today, we see how Peer Review can allow erroneous studies to be published, and why people have been saying that Peer Review is broken.
The Arsenic Life Scandal
The hype machine began chugging away in November of 2010. NASA were sending out press releases. They were on the cusp of a major announcement in Astrobiology, the study of life beyond the stars. No-one knew what the discovery was. All they knew was that it was being kept secret.
What discovery could be so important, that even its announcement needed an announcement?
It wasn’t long before people jumped to conclusions. Had it finally happened ? Proof of extraterrestrial life. Soon the day was upon us. Anticipation was high as NASA declared they had found life on…... Earth. The disappointment didn’t stop there.
NASA claimed they had found lifeforms that built their DNA using arsenic. But there was a problem. Within days of its publication, scientists spotted some egregious errors in it. The experiments lacked important controls. Their data didn’t support their conclusions. But these flaws didn’t come from any intended malfeasance. They were caused by sloppiness and lack of experience. But the errors were so obvious to the readers that they were spotted immediately. Yet it still got published. How ?
How can Peer Review allow mistakes to be published ?
The Authority effect
In 1982, Douglas Peters and Stephen J. Ceci decided to troll scientists for the good of science. Their targets were prestigious psychology journals.
They wondered whether anyone would notice if they re-submitted famous articles under different names. They mocked up the manuscripts, with fake authors and fake universities. Twelve of these doctored manuscripts were delivered to their targets. Only three of them were caught out for plagiarism. The remaining nine went through the Peer Review process. That’s where it got weird.
These studies had gone through Peer Review already. Not only that, they were pretty well regarded. It should have been a foregone conclusion. They should have passed through Peer Review with flying colours. But they didn’t.
Eight of these papers were rejected. Not because they had been caught for plagiarism. Not because they were recycling ideas that were already known. Those reasons would make sense.
The reviewers picked apart flaws in the experiments. They objected to the writing style. according to them, these papers were not fit to print. If this was the majority opinion, then how did these get published in the first place ?
Peters and Ceci suspected it was due to the Authority effect. That was the only change they made. These fake authors and fake institutions were unknown within the field. The reviewers implicitly trust certain institutions, and by proxy trust their papers, without really critiquing them. Publishing from a famous institute got these papers a free pass through review.
The easiest thing to convince someone of is something they already believe. That even holds true for scientists.
In the late seventies, a fierce debate was raging in the world of psychology. They were arguing about motivation.
Let’s say you want to do something. Like say reading a long article series about Peer Review. Why are you reading it? What drives you ?
Is it curiosity? Do you enjoy hearing kooky tales about scientists ? Do you enjoy it for its own sake ? Then, I’m afraid you might be reading it because of “Intrinsic motivation”. This is where the desire to perform an action comes from within.
Or are you reading this article for other reasons. Are you researching for a school assignment ? Perhaps you need some background for talk you want to give ? Maybe you want to join in the discussion in the comments below ? Or even write your own article !
Those are sources of “Extrinsic Motivation”. You may not enjoy the article for its own sake. But you know reading it will benefit you in some other ways. You might get a good grade. You might impress someone with your newly acquired anecdotes. So you can say “Don’tcher know about that one time Einstein was wrong...” at a cocktail party and impress all your intellectual friends.
In the seventies, scientists were arguing over a simple question. What happens if we decide to reward you for doing something you already enjoy ?
What if you enjoyed reading this article, and then someone rewarded you for it ?
Surely, you would enjoy it more? Right ?
Some psychologists were surprised when they put this to the test. They sought out a group of avid puzzle solvers, and gave them puzzles to solve. When they paid them for their time, they spent more time on those puzzles. When that pay was taken away, everything changed. Those once avid puzzlers no longer played those puzzles as much. It seemed like rewarding them eroded their intrinsic love of puzzling. This is the “Overjustification effect”. It was controversial. Some scientists had experimental evidence to support it, others had evidence to dismiss it.
The point was that it was a fierce debate and it divided the community. It was the perfect environment to run an experiment on Peer Review.
Michael Mahoney of Penn. State University sent research papers to professors on both sides of the battle lines. To some of them he sent studies which confirmed their stance on motivation. To others he sent those very same studies tweaked to challenge them. This time, the results were not so surprising.
The Professors hated the papers that didn’t fit their world view. They were much more critical of those studies. They disputed the methods more, and were more likely to pick up small mistakes. Here is one example that illustrates the whole situation.
When Mahoney was making the manuscripts, he accidentally made a spelling error. That error made it into all the versions of the paper. It was spotted by 25% of the “friendly” reviewers. And by 72% of the unfriendly reviewers. People reviewed according to their own biases.
Confirmation bias can be a real problem. Editors often worry about it. Is a paper actually bad, or are the reviewers biased ? It’s something that editors always need to be alert to. They need to spot when reviewers are acting unjustly, and work around them. Editorial oversight can be a lifesaver for important papers, But it can also be a problem.
The Editors of scientific publications like Nature are pretty smart people. Many of them are scientists in their own right. Occasionally, they can review papers for themselves, and make their own decision that conflicts with those of the reviewers.
If their peer reviewers haven’t presented a convincing case, or clearly show their bias, it is the duty of the editor to overrule them. This is why editors can push through articles even if the peer reviewers are against them.
James F. Danielli, then editor-in-chief of The Journal of Theoretical Biology, faced such a choice. He had been sent a paper by a scientist named Lynn Margulis. In that paper, she put forward a controversial new theory.
Within the cells of every animal, plant or fungus, there are tiny little structures . These are called mitochondria. Margulis noticed that these structure resembled bacteria. She proposed that they were bacteria. She suggested that millions of years ago, these bacteria entered into a symbiotic relationship with our evolutionary ancestors.
These bacteria helped generate energy for the cell. In return the cell provided protection. Over time, they became more dependent on each other. To the point that today, we don’t see mitochondria as separate life forms. It was an outlandish idea.
A slew of journals rejected it before it reached Danielli’s desk. But Danielli took a bold move. He decided to push the paper through. That turned out to be a good call.
Soon, more evidence emerged to corroborate the theory. Mitochondria had their own DNA, whose lineage could be traced to free living bacteria. They had separate molecular systems, and even replicate like bacteria. Margulis’s ideas successfully traversed the breach between the fringe and the mainstream.
Fast forward to 2009. Lynn Margulis now sits in Danielli’s position. She was a member of the National Academy of Sciences. She had the power to suggest and review any paper for the society’s journal, “Proceedings of the National Academy of Sciences”.
Enter Donald I. Williamson. Like Margulis, he had a controversial theory. he noticed that a creature called a “Velvet worm” bore a strong resemblance to caterpillars. This was why he suggested that caterpillars and butterflies were once separate organisms. After a chance mating event between a velvet worm and an insect, their offspring could change into both forms. That was why caterpillars and butterflies look so different. It was an outlandish idea.
It wasn’t hard to see the parallels with Margulis’ own ground-breaking work. When she came across this paper she decided to make a similar gamble to the one that was made on her. She pushed it through to publication, against all protestations. This time, it was not a good call.
The entire basis of the paper was that velvet worms and caterpillars look alike. That wasn’t enough. Extraordinary claims require extraordinary evidence. Margulis had that to back up her work. Williamson did not. The genetic evidence showed that velvet worms and caterpillars were very different. And not related. Experiments to mate cockroaches with velvet worms have so far been unsuccessful.
The paper was immediately ridiculed.
Editorial discretion is sometimes essential to help good papers get published. But that can be a double edged sword. The people who make these decisions are just people. People can make mistakes too.
Peer Review is only as strong as the Peers who Review.
The scientists who peer review articles for journals are people. The editors who handle the Peer Review of those articles are also people. Even the scientists who do research, happen to be people. People can make mistakes.
Between 1.4 and 1.6 million papers get published every year. That doesn’t even account for all the papers that fail to pass review. The journals and their reviewers are under a huge amount of pressure. Under those circumstances, Peer Reviewers are often not on the top of their game. Considering their central role in the scientific process, it is surprising how poorly they are treated.
No-one gets any training to become a Peer Reviewer. all you have to do is register with a journal, or happen to have been suggested by some author somewhere. There is often confusion over what the actual role of the reviewer is. Are they to fulfill their time honoured role of determining whether a study is interesting? Or are they there to ensure the validity of the paper ? Some journals aren’t very clear on this at all.
Reviewers are often under severe time constraints. They have to juggle full time jobs, and must try to find time to review papers around that. The journals aren’t ready to pay their reviewers for their time. Without that kind of leverage, often the people who are selected to review papers do not see it as a priority.
Which is why the person who the editors think is reviewing isn’t always the person who is actually reviewing. A senior scientist can push their peer reviewing duties onto their students. Students who may not understand the papers they are given. They might not spot any errors, and let a flawed paper through. Or , they could misunderstand the paper and reject without any real grounds for it.
If you’re actually good at reviewing papers, journals will soon catch on, and you’ll find yourself inundated. So the journals are forced to select reviewers who aren’t good at assessing papers out of desperation.
Peer Review is only as strong as the Peers Reviewing, and we can only make Peer reviewers strong by investing in them. But Journals don’t invest time in evaluating reviewers. They don’t spend time training their reviewers. They don’t invest money in their reviewers. When you take that into account, it’s no wonder that Peer Review appears to be weak.
Is Peer Review Broken ?
With all of these flaws, it’s no wonder that people have been asking whether Peer Review is broken. Whenever a Scientific Journal publishes an erroneous paper, it breaks the trust that Peer Review engenders. That promise made at the end of part 1 of this series has categorically been broken. Just because an article has been Peer Reviewed does not guarantee its validity.
But as we’ll see next week, that promise might have been impossible to keep right from the very start. On Monday, I’ll talk about how the very nature of science means that many of the scientific studies that are published will eventually be proven wrong.
J Jonah Jameson, from Spiderman
References and Further Reading