The netizens of Pubpeer broke last years biggest science scandal, exposing the flaws in the acid-bath stem cell paper. Now they're under threat for commenting on the work of Dr Fazlul Sarkar. I'm delving into the complaints raised by PubPeer, to show you why this case is happening, and why it's important.
The unfortunate experience of Dr Sarkar
Last year, life looked rosy for Dr Fazlul Sarkar. He was on his way to a tenured position at the University of Mississippi. He prepared his family for the move, put his old house on the market, resigned his position at Wayne State university and got ready for his new life. A new life that wasn't to be.
Just eleven days before he was due to start work, the University of Mississippi, the job application was rescinded. The stated reason ?
Faculty members at the University of Mississippi had been on the receiving end of a number of e-mails from an anonymous source. These e-mails contained several posts from PubPeer, focusing on the work from Dr Sarkar's group. I'll get into the content of these posts later, but suffice to say they were not kind to Dr Sarkar's papers.
"At this point, we cannot go forward with an employment relationship with you and your group. With these allegations lodged in a public space and presented directly to colleagues here (I am not sure of the scope of the anonymous distribution), to move forward would jeopardize our research enterprise and my own credibility"
Dr Larry Walker, Excerpt from Dr Sarkar's Rejection Letter
Making matters worse, when Dr Sarkar attempted to regain his old job at Wayne State University, it was no longer available to him. Thus ended his scientific career, brought low by one website. PubPeer.
What is PubPeer ?
PubPeer went online in 2013 with one simple goal- To critique scientific papers in a public space, and to make that dialogue accessible to all. A scientific paper is the beginning of a larger discussion, and PubPeer wanted to be the venue for that discussion.
It also was there to fulfill an unmet need, to catch the mistakes that can slip through peer review. It arose in the wake of a number of high profile cases, in which massive flaws were discovered in scientific articles published in "reputable" scientific journals. There was the notorious Arsenic life paper, the Diedrik Stapel and Marc Hauser cases where either incompetence or outright misconduct forced the retractions of papers from the scientific corpus.
The commenters of PubPeer have been at the forefront of exposing scientific fraud, with their biggest coup being their rapid exhaustive coverage of all the flaws within Haruko Obokata's Stem Cell paper. Their criticisms triggered retractions from Nature, one of the most prestigious scientific journals out there. That such a highly regarded journal could fumble its peer review so disastrously provided more evidence that post publication peer review is crucial to science.
Wait, slow down a second, what is post publication peer review ?
Well, let's start with the basics. Peer review is everywhere. If you say, work as a journalist, you probably have to get your work looked over by an editor of some stripe before it gets published.
The basic idea is that you get a smart person who understands what you're trying to create to look over your work. They can point out the grammatical errors you miss, perhaps help you re-word things in a way that will help your intended audience understand it. They can even point to outright errors, so you can correct them before you get published, rather than after.
Scientific peer review is like that, except that there are vastly fewer people out there who understand what the scientist is attempting. There are very few experts out there in whatever field a certain person may be working. So the journal runs new publications by them first, just to check whether the facts are correct, and whether any other scientist will want to read that work. The criticisms they provide can be used to improve the work, and to make it suitable for publication.
In practice, this process is far from perfect. In fact there are enough problems for me to write a whole other article on them, but for the moment I'm going to focus on two.
- Peer review is only as strong as the Peers reviewing. These people tasked with going through papers are human, and they can make mistakes. They may not have the right specialty to understand the work they have been asked to critique. They may not have had the time to read through everything properly. This is how massive errors can slip through peer review.
- Individual specializations in science tend to be small fields, and very competitive. Which means that competitors may often end up critiquing each-others papers. Worse, a grad student may find themselves needing to apply for a job at the very lab whose work they trashed.
The first problem is why post publication peer review is necessary. Just because something has been published doesn't mean its findings are set in stone. There needs to be an ongoing culture of scientific criticism to root out these kinds of errors.
But, if this culture of criticism is to be successful, then it needs a solution to the second problem. Those who critique science need to be protected, otherwise they will never be able to speak freely about a papers problems. This is why scientific peer review is conducted anonymously, and why any website attempting post publication review also needs to guarantee the anonymity of its user base.
The anonymity of PubPeers users are now under threat.
Who is threatening the PubPeer users ?
Remember the unfortunate experience of Dr Sarkar, and how comments on the PubPeer website triggered them ? Now he's going after the websites users, demanding that PubPeer reveal their identities so that they can be sued.
PubPeers commenters stands accused of defamation. Essentially, the idea is that PubPeer has hosted false statements, and that those false statements have caused damage to the character of Dr Sarkar.
What have the PubPeer users been saying about him?
I'm going to go deeper into the exact criticisms further down the article. But first, we need to get a little science-y.
Most of the criticisms focus on a technique called "Western Blotting". I'm going to give you a quick rundown of what western blotting is, and how it has been abused in the past.
What is Western Blotting?
Let's say you have two test tubes full of cells. You want to prove that the cells in one of these test tubes is producing more of a specific protein than another. It looks like a job for Western Blotting* !
You can pop these cells open, and pull out the proteins form each of the test tubes. In the next step, the scientists will separate the proteins by their molecular weight, and they do this by holding a race.
The proteins are given a negative charge , and put on one end of a gel. The positive charge on the other end of the gel causes these proteins to be pulled forward. But it's a race, remember ?
The heavier molecules will run at a different pace to the lighter ones, causing them to separate. Proteins of specific sizes will form bands. The process is a lot like chromatography, which you may have experimented with at school. If not, here's a link to a good video about it.
Based on this technique, we can estimate the size of a protein. But we still don't know whether it's the protein we're looking for.
For that , we need to use antibodies that specifically attach to the protein we want to look for. These antibodies are labelled with a luminescent dye. If the protein is present, specific bands will start to glow. If there is a lot of protein there, they will glow brightly, and if not, they will glow very faintly.
The traditional way of measuring this is by sticking some photographic paper over them for specific periods of time. When exposed to light, the paper darkens. So the light emitted by these glowing antibodies causes the photographic paper to darken. The darker a band, the more of a protein there is within it. This is the Western blot.
But in order for a western blot to be valid, it needs to be standardized. Whilst a dark band might indicate that there's more protein, it could also indicate that the protein has been exposed to the glowing antibodies for a lot longer.
That is but one example of many factors that need to be controlled for if a western blot is to be taken seriously.
Western blotting is a really useful technique when it's used correctly. However, there have been some cases of "Blot Fraud" that have thrown this particular technique into disrepute.
How can this happen ?
Western blots can be altered through the diligent use of photo manipulation software. In 2009, 4 articles were retracted from the Journal of Clinical investigation for blot fraud. The frustration of the executive editor, Ushma S. Neill, is palpable in the first lines of her editorial. Furthermore, A journal targeted specifically at scientific editors called "Editors update" published a how-to guide for detecting image manipulation, and have an entire section dedicated to the humble western blot.
If you don't have the time, here is a summary of the main points raised regarding the most common errors in western blots.
- The undeclared Splicing of different gels together, making them seem like they've been run side by side with the same exposure.
- Running controls on different gels, negating the entire point of having the controls there.
- Beware of repeated controls, re-used and re-purposed for different gels.
- Be suspicious of low quality images.
- Watch out for bands that have been overexposed, to the point where they merge together.
These are the kinds of errors that the PubPeer commenters have leveled at Dr Sarkar.
Were those accusations justified ?
There was only one way for me to find out. I had to review all of the comments that are currently available on the website, to see whether they were fair or not. Which meant I not only had to read the comments, but double check the paper to see if those comments held true.
I put all of the evidence together in the slide show below. I keep a running score of criticisms that I deem to be fair, or unfair. It's all my opinion, and I've provided links to the original journals so that you can judge the validity of the accusations for yourself.
The Evidence against PubPeer
Wow, that was long ! I did not expect the Cylon!
I'm glad you made it through the slide show. I hope that's given you some idea about the basis for this case, and perhaps you've formed your own opinion on whether the commenters of PubPeer were fully justified in their critiques against Dr Sarkar.
Why did you automatically mark comments that lead to corrections as "Fair"?
If a Journal reads these criticisms, and decides that they are justifiable enough to demand an correction (also known as an erratum), and the authors publish an apology for that error followed by a correction, it means that the original comment was justified.
That's not to say that the publication of corrections isn't a good thing. But they don't automatically erase the existence of the errors in the paper, and nor do they absolve problems that still remain within the paper.
Does this mean Dr Sarkar is guilty of misconduct ?
I don't think so. If you read all of the papers, you'll find one repeated sentence. "All authors contributed to this article equally". There is no way for anybody outside of his laboratory or institution to know that, and here in my country at least, we have the presumption of innocence.
The truth is that we don't know, and we may never know for sure.
If he is innocent, it won't mean he'll get off unscathed. As Ushma S. Neill points out-
When questioned about data in a paper, many senior authors feign ignorance and blame the first author or data generator, but I find that disingenuous. If you are the senior author, it is incumbent on you to verify all of the raw data yourself. There is intense pressure to produce, and to produce high-impact results. Sometimes this can lead to a student doing anything to please. If the paper goes out with your name on it, you should be able to verify every single piece of data in it and take responsibility for it.
Why do you keep sticking up for this guy ?
Whilst I hope PubPeer will be a huge force for good in science, it worries me. If these criticisms prove anything for certain, it's that scientists don't always examine the evidence in articles before they are published. Considering that its their job to do this, how can we expect any more of the casual reader ?
It worries me because I know that there are so many scientists out there who will see a criticism, and not go to the same lengths I have to verify it. Maybe they won't even read it. They'll just see that it exists, and count that as a strike against the author.
That's only a problem if there are criticisms that are outright wrong, and misrepresent the content of a journal article, making it look like the authors are guilty. Whilst I don't believe this has occurred for the majority of the comments against Dr Sarkar, there is one exception that really sticks in my mind.
Towards the end of my slide show, you'll notice there was one comment that appeared to have completely fabricated a graph to make it look like a graph had been plagiarised. [EDIT 21/1/15- I didn't actually check hard enough, I'll explain] I checked the actual article, and the problem just wasn't there. It doesn't look like anyone fact checked it. No-one even called it out. This is the kind of skeevy behaviour that makes some scientists leery of PubPeer
Defending yourself against peer reviewers and other scientists comes with the territory. But many scientists do not want to have to defend their work against an army of internet commenters.
Wait , do you support the lawsuit against PubPeer?
Oh Hell No. I was just giving you the side of the argument that I haven't often seen addressed in this controversy.
Science by its very nature is self-critical field. Peer Reviewers aren't obliged to show wisdom or compassion, only to give their opinion on articles before they are published. Those opinions aren't always based on a close reading of the paper, heck, they aren't always based on fact. Yet we still address every last one of them as if they were legitimate.
Why ? Because when something as precious as your life's work is being criticized, even the most innocuous comment can feel like an attack. Every comment that knocks you back feels unfair, so it's impossible to tell whether they are really being unfair. Sometimes the criticisms that offend us the most are the ones that we need to hear the most. Those are the ones that make us better at our jobs.
That's why scientists aren't generally in the business of silencing criticism and dissent. To do so would knee cap one of science's greatest strengths, its culture of criticism.
Criticism is essential to the scientific process. It's what drives us to hold ourselves to better standards, forces us to continually improve what we do, and question what we think we already know.
Science should be controlled by the people with the best science, not by the people with the best lawyers. If we want to be able to hold science to the highest standards, we must be able to speak freely about it.
So, yesterday morning, I looked at my e-mails to find that PubPeer had left me a message. Funnily enough, it wasn't a legal threat. It very well could have been, because the message revealed a mistake made in this article! It was a big one, one that made PubPeer look a lot worse than it actually was.
What was the mistake ?
Remember how I made a big deal about a graph that looked like the commenters had photoshopped it ?
Yes, it was only seven paragraphs ago
Well, the reason I drew everyone's attention to that graph was that the one in the comment looked different to the one in the most recent article.
I get the feeling the word you italicised is going to be very important
Indeed ! You see if I had checked an earlier version of the article, this graph, it would have looked like this :-
In case you can't spot the error, the last two graphs are identical. Which means that the PubPeer comment was fair.
The graph was changed in later editions of the paper, without any correction being published. So when I went to read this paper, I had no idea that the Journal of Biochemistry had changed the graph.
So why is PubPeer not suing you?
Sometimes in science, people make honest mistakes. Both in scientific papers, and in the criticisms of those papers. Like I said before, the normal reaction of science is to address criticism, and not to silence it. To educate rather than litigate. Which is why I got a polite e-mail instead of a court summons.
So did you just critique PubPeers critique of your critiquing of Pubpeers critique of the scientific literature ? If so, what will happen if someone critiques this critique of PubPeers critique of your critique of PubPeers critique of the scientific literature, and you have to respond with another critique?
*Why is it called Western blotting ? Because it apes a similar technique used to detect DNA that was invented by a man named Edwin Southern, that was referred to as "Southern Blotting". As a joke, the creators who applied this to proteins called their technique Western Blotting, and another similar technique for RNA was called Northern blotting. Don't even ask about Eastern Blotting.
All images in the slide show come from the PubPeer comments, or the Linked journal articles.