Over the past few months, Facebook has used third-party fact-checkers to decide which COVID-19 news stories and op-eds are false or misleading. Recently, one determined that a Wall Street Journal opinion article by Marty Makary, MD, MPH, Johns Hopkins professor and editor-in-chief at MedPage Today, was misleading. Specifically, "Three scientists analysed the article and estimate its overall scientific credibility to be very low," according to HealthFeedback.org, Facebook's checker in this instance.
In his op-ed, Makary argued that COVID-19 will be mostly gone by April. I want to be clear: I am not interested in litigating Makary's opinion, which is what an op-ed conveys. April 30 will be here soon, and we will find out. Instead, I am interested in thinking about how an $800-billion company decides when an op-ed by a professor can be stamped "misleading."
How does this third-party fact-checking system work? Does it solicit reviews from the top academics in a field or discipline? Why is it targeting an op-ed? As I researched this piece, I discovered that the process is obscure. However, what was clear is who the reviewers are.
They are disproportionately academics on Twitter who have mega-follower counts. They mostly have similar worldviews, and advertise those views on Twitter. In a different case, a reviewer already tweeted criticism of the article before being selected as a "fact-checker." This isn't an independent or fair process -- it is cherry picking criticism from Twitter celebrities in order to extinguish dissenting opinions.
How it works
Facebook has asked HealthFeedback.org to fact-check at least some articles. The website appears to solicit between two and four reviewers per article, and compiles their feedback before rendering a verdict. The website appears to be new. It lists three article reviews prior to COVID-19 and seven after. There were 10 reviewers pre-COVID-19, and 19 since January 2020. I extracted data on all reviews.
Fact-checkers or Twitter celebrities?
Of the 19 fact-checkers selected since the start of COVID-19, 15 (79%) have active Twitter accounts. These folks are followed by an average of 42,000 followers (median 10,000). Four of the fact-checkers have served on more than one occasion.
Let's compare the COVID-19 reviewers against pre-COVID-19 fact-checkers and academics in general. Among the 10 fact-checkers for three reviews on HealthFeedback.org prior to COVID-19, there were no repeating reviewers. Half were on Twitter and their average follower count was 442 (median 130).
To compare these rates against average academics, I visited the Johns Hopkins University Department of Epidemiology faculty listing. I picked 10 academics and searched for their Twitter accounts. Just three of the 10 had Twitter accounts with an average following of 800 (median 120).
This should come as no surprise: the average scientist is not on Twitter, including many of the best and brightest. Yet, for pandemic fact-checking, HealthFeedback.org seems to choose from Twitter celebrities who average 40,000 followers.
The independent fact-checkers used by this website appear to be far more active than the average epidemiology faculty member on Twitter. Several have stated their thoughts on COVID-19 policy and support for continued restrictions, while Makary is suggesting we should begin planning to loosen restrictions.
No matter how anyone feels about who is right -- is this a fair process? They are reviewing an opinion article using scientists who are disproportionately on a website that allows them to advertise their opinion in advance. I promise you, I can search Twitter and find three people who will review Makary positively and three who will review him negatively, just by perusing one's past tweets.
A conflict of interest
In one case, it appears the fact-checker was chosen because he had already written a Twitter thread that was critical of the article. The review notes it was "lightly edited for clarity." In other words, a Twitter user announced they were critical of an article, and then their views were selected as independent fact-checking. That is a deeply problematic selection process, as independence typically implies a reviewer has not pre-judged the material.
This also appears to run counter to the process stated where the editor selects the article before choosing reviewers. Imagine if a juror was selected in a trial because they previously tweeted they were sure the defendant is guilty.
It all begins and ends on Twitter
The fact-checking summaries show a larger debt to Twitter. The article that dissects the Wall Street Journal op-ed by Makary also includes screenshots of tweets by Eric Topol, MD, and Caitlin Rivers, PhD, to bolster its claim. Thus, in addition to heavy reliance on academics who happen to be on Twitter as fact-checkers, it appears the editors are closely following the comments on Twitter to guide them, perhaps even toward which topics to select.
This is not independent fact-checking, but groupthink
Put together, the fact-checking is suspect. It appears to be a website with its own policy ideas about COVID-19, which is selecting academics who are popular on Twitter, and have declared a point of view -- and gives them the chance to extinguish ideas that oppose their own.
It feels like a high school clique. These are the popular kids. They are using their position to label views they disagree with as "misleading."
Some argue this is just the battle of ideas, just a debate. Debate would be if these reviewers wrote a rebuttal, but they go further. They use the brute force of the platform to literally label Makary's op-ed as misleading. That is a massive leap from debate. Labeling instantly usurps the reader of their ability to make up their own mind. It is antithetical to the spirit of the academy.
Facebook is a sea of garbage and this is what they police?
I have to state the obvious: Facebook is a sea of garbage. Illogical arguments, false claims, harmful views -- you can open Facebook and find whatever objectionable idea you wish. That's why I don't open it. And yet, there is only a tiny subset of stories that the organization has sought to label "misleading." And one of them is an op-ed by Makary? This whole scenario is bizarre!
Meanwhile, there are news stories that are patently wrong for which Facebook takes no action.
What I didn't find
I did not find any explanation as to which of the billions of articles and Facebook posts the fact-checkers are asked to review. How does the editor pick? I found only a vague explanation of the standard to judge an article, not how disputes among reviewers are handled. I did not find an explanation of how the reviewers are chosen and what happens if they decline. I did not find information regarding who is paying whom and how much. I did not find evidence of how appeals are handled.
This is an op-ed
I again have to mention the Kafkaesque point that Makary's piece is an op-ed, which by definition is his perspective based on his interpretation of data. Much like a Franz Kafka novel, Makary has gotten a show trial by Facebook, and is found guilty of misleading the public with his opinion article by a bunch of Twitter celebrities, whose opinion is well known. I don't know if Makary is right or wrong, and time will tell, but what I do know is this process is not acceptable or fair. Facebook should not call it "fact-checking." That's not right. It is just checking what their friends on Twitter have to say.
Vinay Prasad, MD, MPH, is a hematologist-oncologist and associate professor of medicine at the University of California San Francisco, and author of Malignant: How Bad Policy and Bad Evidence Harm People With Cancer.