Last time, we left the story with Peer Review reigning triumphant. Did the king truly deserve his crown ? Can a magic marker pen, a kidney and a gmail account help you cheat Peer Review ?
On the word of Peer Review
Peer Review focuses scrutiny on discoveries before they are published. It gave journals an extra layer of authority. Everything they published had been vetted by experts. This was becoming more important. The forces that compelled journals to adopt Peer Review ensured that their audience demanded it.
Last week, I talked about how the growth of science made the jobs of journals much harder. What I didn’t talk about was the effect that was having on the work of scientists. A scientist who studies bacteria may have to learn about a new antibiotic. That antibiotic will have been made by advanced chemistry.
That microbiologist may not have studied chemistry since high school. They were never trained to evaluate advanced chemistry. If they read a paper about an antibiotic created with advanced chemistry, they will run into trouble. They may not be able to spot the kind of errors that a chemist would spot.
Which is why Peer Review provides security to scientists as well. They know this paper has been sent to an expert who understands the paper. If there were any flaws, they would have been caught before publication. Thus, a scientist no longer needs to fully understand a paper to accept its results. Peer review removed the need for scientists to read critically.
Science shifted away from “On the Word of No-One” to “On the Word of Peer Review”.
Pressure to Perish or Publish
Science was no longer a hobby of the rich, it had become a profession. Governments and Industry were now employing scientists. But how do you work out whether a scientist is worth employing ?
The obvious way is to see what a scientist has discovered, by reading their work. But that takes a lot of work. So of course, employers tried to figure out a shortcut. They knew that if a work of science was important, other scientists would reference it.
Some journals are more widely read than others, and tend to get more cited. Therefore, if a scientist publishes in them, that scientist must get more citations. Thus, (by a somersault in logic) these highly cited journals must have better peer review. So a scientist who gets published in such a journal must be worth their salt. If they get published multiple times, then they must be even better.
It has become essential for scientists to publish in the most popular journals. Careers depended on getting the right publications. As the pressure to publish increased, so too did the pressure to cheat.
Science Scandals Sow Doubt
William T. Summerlin was a rising star in the dermatology world. What he was most famous for was his incredible transplants. To show how dramatic this was, he transplanted skin from a black mouse, to a white mouse, with no ill effects. He would show these mice off at talks. People would be struck by how healthy they were, and that they didn’t need any anti rejection drugs.
But something was off. These patches of fur would sometimes “Shift” their position on the body. Sometimes they would look greyer than usual, then other times they would look blacker. One day, a suspicious technician decided to test for sure. They dabbed a bit of alcohol onto the black patches. The patches washed off. They were ink. William T. Summerlin had fooled the entire scientific world with a magic marker pen. It was one of the first big scientific scandals to reach the public in the modern era. But Summerlin wouldn’t be the last scientist to try to fake it to make it.
Gaming Peer Review
One of the biggest complaints scientists have about the whole peer review process is its length. It takes time to review papers, and peer reviewers generally aren’t in a hurry to do reviewing. They have other jobs that they have to do at the same time. Which means it can take weeks, even months before they respond.
One enterprising scientist found a way around this. Hyung-In Moon worked on discovering new drugs based on natural compounds from plants. His reviewers responded within 24 hours of being asked for their comments on a paper, they responded. Those responses were nearly always favourable.
This aroused the suspicions of Claudiu Supuran, Editor-in Chief of the “Journal of Enzyme Inhibition and Medicinal Chemistry”. He noticed that many of Moon’s reviewers didn’t have professional e-mail addresses. They chose to review papers via Gmail, or Hotmail. Not unheard of, but not common. After some detective work, Supuran discovered that these Peer Reviewers didn’t exist. Hyung-In Moon had created them. How was this allowed to happen ?
When a Journal gets a paper, they don’t always know who the right person is to send it to. Which is why they ask the author for suggestions. The author is expected to recommend experts in the field. They can also warn the journal of competitors who may be biased against them. The author supplies the journal with all the contact details. The editor can then make the decision of who to ultimately send the review.
In recent years, that process has become automated. Instead of sending a letter, online forms are filled out. The author supplies postal and e-mail addresses to the editors at the journal It makes everything much faster and more convenient. But that convenience comes with a cost.
In the grand old days, it would be suspicious if I suggested a reviewer living at my address, or at an anonymous post box. I would be caught out pretty quickly. E-mails are not like that. People can have multiple e-mail accounts, not all of them through their institution. It can be a lot more difficult to assess how valid an author actually is.
That’s how Moon got away with using fake e-mails for so long. He had managed to get at least 35 papers out before someone noticed.
He wasn’t the only person caught vigorously reviewing himself. A “Peer Review Ring” involving 130 fake e-mail accounts was busted, leading to the retraction of 60 papers. A leading publisher, “BioMed Central”, recently retracted 43 articles because of this.
As far as fraud goes, this is a new one. Journals are still wrestling with how to deal with it.
There are still scientists who alter their experiments to make the results more impressive. But they’ve gotten smarter since Summerlin. Transplantation researchers would not be fooled with a marker pen again. Which brings us to how transplantation researchers were fooled again.
We’re lucky to be born with two kidneys, even though we can survive with just one. That’s why kidneys are the most commonly transplanted vital organ. Unlike many other organs, you can get them from living donors. But rejection is a huge problem. The immune system can attack a new organ, killing it off before it can function. Anti-rejection drugs prevent this from happening.
Enter Judith M. Thomas. Her research focused on finding new anti-rejection drugs. To investigate this, she performed kidney transplants on animals. If her drugs stopped an animal rejecting its new kidney, they would work in humans.
But it isn’t enough to just replace one transplanted kidney. Because, like humans, most animals have a spare that they can survive on. If the anti-rejection drugs failed, no-one would ever know. The new kidney would shrivel up and die, but the healthy one would do all the work. The animal wouldn’t show any symptoms. It would look like the drug was working, when it wasn’t doing anything. If you want to know whether an anti-rejection drug works, both kidneys need to be removed.
In 2006, she made a complaint about one of her subordinates. Juan Contreras hadn’t been removing the second kidney. The university launched an investigation. They soon realized that many of the animals weren’t getting the second surgery, going back years.
It soon came to light that other people had complained about this to Thomas. She aware of the fraud for a long time, and had allowed it to continue. That fraud had made her anti-rejection drugs look more effective than they actually were.
There are comparatively few reports of scientists directly manipulating experiments. It takes time and money to care for animals, to buy equipment and chemicals. In short, this kind of fraud is expensive, and wasteful. It’s also difficult to catch. Any data obtained from such experiments will look genuine. The papers themselves will look original. Only people directly involved with the project have a chance of catching the fraud.
The question arises. Why go to the trouble of performing these fraudulent experiments ? Why spend the money and the effort ? Why not just lie ?
This story starts with a legal case. A prestigious Lawyer, Richard Barr, was working with an activist group who were very worried about some drug companies. They suspected those companies had been selling products with some awful side effects. But he had hit a snag. He couldn’t find the evidence He needed help from an expert. Which is why he recruited a physician named Andrew Wakefield.
Wakefield had made his career as a gastroenterologist. He had become convinced that the measles virus could cause gastric problems. He suspected a link to Crohn’s disease, but a chance meeting with the mother of an autistic child would sent him down a very different path.
The child he treated had severe constipation. He suspected a link between the child’s autism and their constipation. He already suspected that measles could cause constipation. The mother noted that their child symptoms emerged after they had received the measles vaccine. A link began to form in his mind.
He hit a snag. He didn’t have any solid evidence to prove it. It may have all been a complete coincidence. The child may have happened to have had bowel problems and autism at the same time. It may have been sheer chance that these symptoms arose after the MMR vaccine. He needed evidence.
So Wakefield decided to put it to the test. He recruited autistic patients for a medical study. He needed a smoking gun. A sample of neurotypical children who developed autism after the vaccine. He needed to prove that children developed a specific form of “Regressive Autism”. This is where the child’s autism isn’t immediately detectable.
He recruited twelve children for the study. This was where the trouble started. Regressive autism is tricky to spot. Autism is a spectrum, and diagnosing it can be a hit and miss process. Very hit and miss. Even more so in this study. Starting with those twelve children.
Three of them weren’t even diagnosed with autism at the time of the study. Five of the children had showed developmental issues before they’d been vaccinated. Others only showed issues months after they received the vaccine.
You wouldn’t know any of this if you just read Wakefield’s paper. The details of the patients were altered. According to the paper, all twelve had confirmed autism diagnoses. All of them were fine before the MMR vaccination. They all developed symptoms within weeks of being vaccinated. Despite the hospital records telling a very different story.
But it went further. He also needed to find evidence that these children had an inflammatory bowel condition. Which meant that he needed to cut out biopsy samples from the children. These samples would be analysed by trained pathologists under the microscope. Frustratingly, the pathologists found no evidence of disease. But the paper says the children had “non-specific colitis”.
When real results don’t fit the theory, they can be changed with the stroke of a pen. In this case, those effects were devastating. The paper passed through peer review. When it got into the hands of the sensationalist media, it got out of control. Panic spread. People refused to take the MMR vaccine. As vaccine rates plummeted, the rates of autism.... stayed the same. Measles returned with a vengeance. It became endemic in countries where it had once nearly been extinct. The consequences of this paper were devastating.
Peer review was never designed to catch misconduct
Not too long ago, I met an academic who told me that misconduct was harmless. They said that if mistakes happen, they will be corrected. Which means other academics can make their careers by exposing those errors. So in the end, those errors keep scientists employed.
That is the kind of myopia that breeds misconduct. That’s the kind of attitude that leads to useless drugs and phony health scares. When a scientist doesn’t think, or worse, doesn’t care about the wider consequences. All they see is the cut-throat race to the top of the career ladder.
Peer Review was founded on the basis of honest critique. A scientist should want to have his research honestly critiqued. They should want to get to get closer to the truth. The peer reviewer should want the same. As such, the Peer reviewer must trust what has been presented to him. Which makes it very easy to lie to and get away with it.
The creation of the Office for Research Integrity was a tacit admission of Peer Review’s failure. These fraudsters run rings around the whole process. Journals couldn’t police science, so the government had to step in.
However, scientific fraud is rare. Out of the millions of papers that come out every year, very few are published with deliberately fraudulent results. So perhaps it’s unfair to judge Peer Review for failing to fulfil a function it was never designed for.
On Friday, I will be looking at how Peer Review does at the job it was designed for.
Calvin and Hobbes, Bill Watterson
References and Further Reading