How Citizen Journalism & Education Reform Can Save Us from Bad Science

This pandemic has brought a lot of issues to light, and I can’t stop thinking about the intersection of these four: “bad” science, gatekeeping, citizen journalism, and inadequate education...

* Note: Many of the links here are from people who cherry-picked information to drive their conclusions and that I may ultimately disagree with, but believe there may be a shred of truth that you wouldn’t find otherwise. 

This pandemic has brought a lot of issues to light, and I can’t stop thinking about the intersection of these four: “bad” science, gatekeeping, citizen journalism, and inadequate education. 

They’re often presented individually, but I see them as related. First, this crisis has brought to our attention how bad a lot of studies are, whose ideas spread unchecked in part due to credentialism - the authors are “experts,” after all. Traditional Media exacerbates this, both by failing to question bad science and by compounding underlying biases. In response, citizen journalists (including experts and first-responders) are sharing their thoughts directly, increasing transparency and holding journalists and scientists accountable. 

While this is an exciting phenomenon, it’s weakened by a population that lacks the basic statistical and scientific tools to draw their own conclusions on these matters, which we can solve by reforming education. 

It’s tempting to think of these as pandemic-specific phenomena, but in reality these are longstanding issues that this crisis has made more visible. Science and research shape our lives, from the foods we eat to our personal finance decisions. The challenges posed by bad science, poor journalism, and a population poorly equipped to evaluate scientific research are hardly new, and they will continue long after the pandemic ends if we do not take the steps to solve them. I’m excited about what’s developing in citizen journalism, and hope we’ll re-evaluate how our students spend their K-12 years as a result. 

Bad Science

Studies advance our understanding of the world - whether it’s the efficacy of certain drugs, what to include in our diets,  the outcomes of policies, or the biases of job interviewers. Individually, there are many ways that a study can go wrong, and for the most part these conversations happen quietly during peer-review within scientific and academic circles. More broadly, the replication crisis has called into question how readily we should accept reported results.

The pandemic has upped the stakes, and now studies (and their preprints!) are getting as much scrutiny in the public eye as ever. And indeed, many of the biggest studies focused on this pandemic have had flawed methods, statistical errors, agenda-driven authors, or misinterpreted results. We're seeing studies get reviewed with a fine-tooth comb, especially by those whose opinions on the pandemic are threatened by them. This is actually a good thing, provided the criticisms are valid and fair - and ideally, that we scrutinize studies that support our views just as rigorously. 

One problem is we’re seeing bad studies get referenced over and over again, in spite of heavy criticism - especially if they support a specific narrative. One consequence of the various methods used and populations studied is that different studies will report a range of conflicting outcomes. The spread of results enables partisans to selectively choose the studies that support their point of view. Confirmation bias is human nature, and our best bet is to mitigate its worst effects by arming society with information. 

Take the percentage of covid-19 infections are that persistently asymptomatic, for example:

• The Diamond Princess Cruise studies suggested that 18% were persistently asymptomatic (if cruise passengers are representative), and people routinely cite higher numbers from the same dataset

• a Chinese study suggested a 1% asymptomatic rate, another one suggested 78% asymptomatic rate (I know reports from China can be controversial, but it is still worth including)

• a Boston Homeless Shelter study found ~83% asymptomatic at the time of testing - if you think a Homeless Shelter is representative, especially a group where the non-positive were more likely to have symptoms

• An NYC study that screened pregnant women suggests 78% were persistently asymptomatic - if “pregnant” is not too confounding on age, gender, health, behavior.

• Another cruise ship study also suggested 80%, but again did not explicitly specify that any post-testing follow-up for symptoms occurred.

• A UAE Cancer patient study did follow up to see if symptoms developed and found 0% persistently asymptomatic - if cancer patients, an at risk group, is not too confounding.

We don’t just have a problem with bad science, we also have bad reporting on science. The studies by themselves may or may not be flawed, but the ways they’ve been misused and misinterpreted by the press and internet warriors trying to prove a point is concerning. The Boston Homeless Shelter study probably takes the cake on this, with reporters falsely claiming a widely-shared “not a single one had symptoms.” (It’s apparent the journalist did not read the study before publishing). 

Optimistically, the peer-review process would catch these issues before the study goes viral, but many of these studies and preprints go directly to the press, where the press’s dependence on credentialism, authority, and gatekeeping amplifies the problem.

Credentialism, Authority, and Gatekeeping

First of all, I want to acknowledge that experts play a very important role in society, and they are badly needed especially in times like this. What I’m going to emphasize here is that we sometimes treat them (or, perhaps they represent themselves) as infallible, and therefore we lack appropriate checks & balances. 

Credentialism compounds bad science by stifling criticism. What qualifies you to have an opinion on this? Who are you to question the scientists, public health experts, epidemiologists, doctors? A brief summary of how credentialism has protected flawed ideas:

• January: “The WHO says no person-to-person transmission, and this is not a big deal.” Youtube is censoring WHO-contradicting posts. 

• February:: “I’m a Doctor, I know coronaviruses, and this is not a big deal. ARDS? Seen plenty of that too.” (his 180°, a month later.). This is also when the FDA was shutting down testing. 

• March: “WHO says death rate is 3%” (at the same time South Korea was reporting 0.53%).

• April: “The CDC says you don’t need to wear a mask unless you are sick,” so did the WHO. (and people were shamed for this).

• May: the CDC combined antibody and viral tests, and Public Health experts continued advocating for lockdowns as Economists began to more vocally disagree...

Even at the time, there were serious doubts about the rationale behind these claims, but in real-time the claims were protected by gatekeeping: the idea that only certain individuals are privileged with the truth, and the rest of us have no grounds to disagree. Dramatic reversals like this are hardly pandemic-specific; consider how we’ve reversed norms around mammograms and changed guidelines about what foods are healthy. 

Authority

I want to briefly distinguish between credentials and authority. We treat them similarly, in that we generally assume authorities make decisions in consultation with experts. Most of the examples I just listed came from persons of authority, who might be balancing various forces that we’re not privy to. For example, a common excuse for Los Angeles’s Mayor Eric Garcetti’s obviously incorrect prediction is that it was justified in scaring Angelenos to stay home. Similarly, the CDC was fairly transparent about preserving scarce masks for healthcare workers.

Certainly, authorities get some things right, otherwise we wouldn’t trust them (case in point, the left’s general distrust of Trump) - but we need a mechanism for questioning decisions that may not ultimately be in our favor. Regardless of whether it’s incompetence or hidden motive, believing authorities because they are authorities is not always in our own interest nor does it further the pursuit of truth. Gatekeeping curbs our ability to question authority. 

Gatekeeping

Gatekeeping isn’t always bad and it’s often a useful heuristic, but we need to recognize that it’s not a proper logical counter-argument either. Some actual quotes that I’ve seen in the past weeks, some I agree with some I don’t:

• “Once again, a non-clinician schooling physician with over 20 years of practice how to correctly define asymptomatic” - an MD, responding to different metrics about asymptomatic rates

• “You’re not the one that has to tell parents that their kid died because they believed a fake article over their doctor.“ - an MD

• “At least my position is backed by CDC guidelines.” - spouse of a healthcare worker, criticizing mask wearing

• “These are just some observations by a random guy who disagrees with the US Surgeon General’s stance on masks” - an MD, criticizing a mathematician’s synthesis of mask research

(Lots of healthcare workers in my circles)

It’s not always so simple, of course. I criticized the “California herd immunity” theory from a fellow at Stanford’s Hoover Institute. It didn’t seem plausible to me with the information available, but it was certainly an added bonus that he is a historian and not an epidemiologist. 

People often claim a monopoly on truth. I am an epidemiologist. I am a doctor. I am a scientist. I am a trained statistician. Our egos at work, disgusted that someone who didn’t spend years and dollars suffering through the material like we did would dare question our expertise. 

Cherry-picking Experts

Now, a big problem with relying on experts is that, when it comes to controversial issues, there is an endless supply of “experts” who will endorse whatever side of the issue you lean towards. Experts are ultimately humans: while they, on average, will know vastly more about an issue than you do, they also have their own incentives, biases, and agendas; and even at full transparency, they may simply make different value judgments than you would. How do we know which “expert” to believe?

This is certainly not limited to the pandemic, either. Different doctors routinely give conflicting medical advice. You could be forgiven for having no idea if cheerios are healthy or not, or whether or not vegan diets are healthy or not, whether you need sun-exposure or whether it causes wrinkles. You can find economists on both sides of the minimum wage debate.

Again, confirmation bias and partisanship will often trump credentialism. We’ll prefer experts who confirm our existing beliefs, and we’ll blindly accept arguments that support our beliefs even if they don’t come from an expert. We can see it very clearly in which studies are scrutinized by whom, and who’s giving which bad studies a pass. 

The natural antidote to this is to diversify your sources, (or, when possible, defer to a survey of experts, which is subject to its own flaws). Even so, by relying on expert interpretations of the situation at hand, even if there is overwhelming consensus, we water down the truth. Experts in a field can be subject to groupthink, or they might be motivated by different core values than the general public. Public consensus has been wrong many times, and will be wrong again.

The Halo Effect for Experts

Meanwhile, when we rely on experts, one could easily confuse just who is an expert on what, exactly? Should we rely on Doctors to calculate mortality rates, a numerical exercise? Should we rely on public health experts to decide when to open up the country, without regard for the economics? Should we rely on politicians to pick our drugs, when scientists are skeptical? Can a survey of doctors signal the efficacy of a drug, can a billionaire with no medical credentials but lots of experience qualify?

We need experts to synthesize for us, because we can not afford to all have in-depth knowledge of everything. It’s a shortcut in a world where all the knowledge in the world is a click away, but it is overwhelming. However, we also need an escape hatch. We need a way to hold experts accountable, isolate truth from spin, and form our own opinions. 

Legacy vs. Citizen Journalism

In February, the narrative was that Silicon Valley was overreacting to covid-19 (yes, it’s hard to imagine now, but the press was downplaying through at least February). Balaji Srinivasan famously went head-to-head with journalists from Vox and Recode, who were looking to dismiss fears, and ridicule venture capitalist prep. Some failures of the traditional narrative to date:

• This will not be a pandemic (January 31st)

• The China travel ban is unjust (February 5th)

• Silicon Valley is overreacting (February 7th)

• “Escaped from lab” theory is a conspiracy theory (April 13th)

• HCQ is a disaster recommendation (March 24th), Remdesivir is comparatively responsible. (April 29th)

Journalism is a Game of Telephone

Most of us don’t receive our information straight from the source, or even secondhand through an expert, but rather through a bad game of telephone: the American media diet. Credentialism is exacerbated by mainstream journalists, who rely on experts and authorities to seed their content. 

Legacy Media at its best will hold leaders and scientists accountable on any issue or topic, but in modern times if often falls short of doing so. Mainstream journalists often fail to keep the experts in check, lack the background to evaluate studies or extrapolate results, cherrypick which experts they consult, or selectively critique experts based on the agenda or narrative that is being driven home. They may straight up get the facts wrong (and fail to correct it) or cite an expert in a different field. The dogged adherence to narratives and the lack of accountability poses a systemic risk. 

But who can blame journalists? After all, how “responsible” is journalism that quotes a random viral thread on Twitter for your article? They’ll never live it down if they end up being wrong. Credentialism is an essential safety net for the reputable journalist - but relying on mainstream journalists is not enough of a safety net for us.

The reality is that journalists are ultimately professional middlemen, standing between informers and the rest of us, and adding their own spin along the way. Middlemen are not necessarily a bad thing, as we certainly don’t all have time to synthesize the facts on our own, and great journalists make the relevant facts more accessible - but this crisis has shown both how problematic our dependence is, and how far many will go in search of truth.

In response, this crisis has been a boon for citizen journalism. Twitter, Medium, Substack, and Facebook have played an instrumental role. No one’s making a career change; rather truth-seekers of all backgrounds are synthesizing facts and sharing their findings. 

Citizen Journalism in Action

Experts are sharing their thoughts directly instead of going through a professional journalist, and they’re engaging with disagreement publicly in the comments. Everyday citizens are fact-checking the press. Statisticians are building their own models, and analyzing the numbers. Entrepreneurs and investors are weighing in on the efficacy of the Swedish model. Doctors & health-care workers are sharing their experiences with patients, drugs, and hospitals. Experts and independent thinkers are streaming their “peer reviews,” reading studies line-by-line, trying to understand the methodology and what’s really being shown. Some are even recalculating the results with better methods. And, almost all of them are sharing their sources, in plain view, so you can make your own decisions. 

Unlike Legacy Media, citizen journalists rarely use a “take my word for it” approach. They go straight to the source whenever they can, because heaven knows they’ll be asked for it - Balaji Srinivasan calls this “reproducible reporting.” Sometimes, they are the source. You get to hear it first-hand, instead of reading a journalist’s spin in an amplified game of “telephone.” And then you get to read the comments, where people can share their experiences and/or disagree.

Democratized information has its problems, of course. For one, it is a breeding ground for misinformation. A lot of these viral threads will ultimately be wrong; they will succumb to the same biases as journalists and experts, potentially worse, without the accountability that an institution demands of you or the foundation that credentials usually give. The sharing dynamics on these platforms incentivize clickbait and manufactured outrage over balanced truths. Information goes viral because of how it makes people feel, not its veracity. 

It will, however, be part of a more democratized process, where a rebuttal can be just as visible and accessible, and onlookers can make their own decision from the facts about who and what to believe. “Truth is like a vaccine.” There are some safeguards: an author must build their following, whether on the merits of their credentials, an existing following in another field, or organically through the ideas they share. Whether or not truth-seekers will rise to the top in this environment will ultimately depend on demand for the truth, and how the truth is packaged.  

In a world in which experts and everyone else have equal footing, it is tempting to look towards censorship as a solution. Regardless of your beliefs on free speech, censorship is at most band-aid. It doesn’t solve the underlying root cause: we have a population that does not think critically and is prone to believing partisan propaganda. The solution to these vulnerabilities is to immunize people by educating them.

Education

“Citizen journalism” helps democratize information, as people seek the raw facts and draw their own conclusions. However, it does not go far enough, because it lacks a solid foundation: an educated population that can appropriately contextualize facts and numbers. “The only way a democracy can survive is if the populace is educated enough to make informed, thoughtful, and critically sound decisions and not be easily manipulated.”

Innumeracy

We are overwhelmingly innumerate. This is not a statement about the quality of education in the US versus the result of the world, as far as I can tell it’s happening everywhere. Statistics has become the domain of the statisticians, rather than an essential part of an informed citizen’s toolkit. We’ve generally solved illiteracy, but still suffer from innumeracy. 

Without a solid foundation in both statistics and its misuse, most of us lack the means to synthesize this information, or, even worse, overestimate our ability to do so. I can’t tell you how many times I’ve seen doctors share faulty epidemic progression charts, and incorrectly explain them to others. Numbers can easily be twisted, and partisans share whatever cherry-picked data supports their position and people accept it unquestioningly. “People who are afraid of numbers have a really bad habit of looking to others for answers.” 

Vulnerability to Misinformation

As a result, we have a population that’s prone to believing misinformation and conspiracy theories. The goal should be neither to eliminate herd mentality nor our dependence on authorities and experts - it’s too much work for each person to go deep on every topic. The ideal is that people at least have the skillset to review the facts and decide for themselves on the topics that are important to them. 

Without education:

• We rely on others - academics, usually - to evaluate the methods and interpret the results of their study. Someone often under pressure to produce results or to please another body.

• We then rely on journalists to filter the studies that matter, and in doing so, implicitly rubber-stamp those studies for us.

• We lack the tools to validate what we see, synthesize the facts presented, or understand the tradeoffs that were made along the way. 

This is not specific to the pandemic. Studies and science shape our lives, from public policy to health to the economy and sociology. Carbon tax, minimum wage, nutrition, cancer treatment, mandatory vaccinations, teaching methods, drugs, sexism. 

Depth of Understanding

Some topics require a deep domain expertise to be understood. I’m in no place to evaluate research on black holes, for example. Others could easily be more accessible if we prioritized teaching basic statistical literacy, and how easily data can be manipulated. It’s incredible to me that so much of life is shaped by studies, and yet so few of us are able to understand the basics. Some publications have really taken this challenge to heart, but it’s clearly not enough. As Americans, we need a better access to the underlying nuances driving big decisions. 

Without access to the raw facts, and worse, a lack of ability to contextualize them, it is no wonder that a population as diverse as ours seems to only opine in two modes: conservative or progressive. “Independent thinking” has been watered down to mean independently choosing a party affiliation, without the self-awareness to realize that if you don’t break with your “side” on any major issues, then you are not, in fact, thinking independently. 

Start with K-12

With schools on hold and misinformation (ie. Plandemic) spreading rapidly, perhaps we’ll take this opportunity to realize what a squandered opportunity K-12 education has become. Thirteen years of school, and we ship most kids off to college without a foundational skill set. We can not expect better societal outcomes if we continue to fail our population on such a fundamental level. 

How can we improve? For one, it will likely take a radical rethinking of how we currently spend K-12 years. Education should primarily be valuable to its students, namely by preparing our young for the workforce, and we already fail to do that. Education should secondarily be valuable to society. With studies and science playing such a large role in our lives, we need to prioritize the subjects that enable our citizens to think for themselves on these issues as much as possible. 

Sometimes, this will merely be a matter of reading comprehension - reading a study and being able to understand what it is actually showing, or which quantities it is actually measuring. We don’t explicitly need to teach research reading comprehension, but the fact that we don’t do it undoubtedly discourages many people from even trying to understand it. 

We will also need a population that understands how statistics can be manipulated, through cherry-picking, spurious correlations, noisy estimates, poor comparisons, p-hacking, and choice of model. These are concepts that can be understood without an in-depth understanding of the underlying math, sometimes a visual can suffice. 

Ideally, our students spend years engrossed in the scientific method and research, building the muscle and confidence to engage with the material on their own. Just as we don’t stop teaching reading and writing, thinking about science is something we need to reinforce every year to ensure it’s truly absorbed. While periodic tables and physics formulas may be forgotten, learning how to think is much harder to undo, particularly once you realize these are skills that can be used every day, outside of school.

Lastly, it’s important to note that the antidote to misinformation is not censorship, it is education. An informed citizenry is less likely to be baited by conspiracy theories, less dependent on a biased press to deliver their facts, can think critically, makes their own decision about which sources to trust, and can draw their own conclusions. It will not create the conformist population that partisans dream of, but will democratize access to information and give us a better chance of surfacing truths. 

---

Many thanks to Dr. Boris Hanin for reviewing this with me. Your time is much appreciated!