When the Numbers Don’t Say What You Think They Say
There is a pediatrician who has been making a name for himself in anti-vaccine circles. He released a letter dated March 12, 2019, addressed to lawmakers where he uses some “data” from his medical practice to apparently try and show that vaccines are associated with autism. Never mind the recent and not-so-recent large-scale, well-designed and well-controlled studies that have found no association between vaccines and autism. No, folks. He just draws some numbers from his practice and wants to convince us otherwise.
The, uh, “data”
Check it out:
Out of a total of 3,355 patients, 715 are unvaccinated and 2,640 are “partially vaccinated.” (He states in the letter that almost all of his clients refuse at least one vaccine. More on that in a minute.)
In the unvaccinated group, only one case of “autism/ASD” is noted. In the partially vaccinated group, six cases of “autism/ASD” are noted. The pediatrician then concludes that his patients, because they’re unvaccinated and partially vaccinated, have less autism than what more respectable groups like the American Academy of Pediatrics (AAP) and the Centers for Disease Control and Prevention (CDC) have calculated to be the autism prevalence in the United States.
He also seems to imply that there is less autism in the unvaccinated group, as his data clearly show… Except that it doesn’t show that at all.
Some quick biostatistics
You can skip to the next section if you know how statistics work. If you don’t, here is a super-simplified way of looking at it.
When you flip a coin, you expect to get either heads or tails, right? If you flip a coin that is perfectly made, and you flip it ten times, you would expect 5 heads and 5 tails, right? What if you get 6 heads and 4 tails? You wouldn’t be too shocked. It kind of makes sense that a 6-4 split would happen. But what if you got 9 heads and 1 tail?
You’d probably be surprised, and that’s okay. On the average and in the long run, when you flip a coin ten times, you’re going to get a bunch of 5-5 splits, some 4-6 and 6-4, fewer 7-3 or 3-7, very few 8-2 and 2-8, and even fewer 1-9 or 9-1. You will get a 10-0 or 0-10 less than 1 in a thousand times that do ten flips.
In the long run, the distribution (known as the “binomial distribution“) of the coin tosses looks like this:
Over the centuries, people have calculated the probabilities of these things happening. When we do our own experiments, we compare the results with the probability tables of experiments like the coin flip I describe above. These probability tables exist for all sorts of data: continuous data like blood sugar levels, binomial data like sick/not sick outcomes, and even categorical data like “belongs to group A or B or C or D.”
The major rule is that if our results have a less than 5% chance of happening, our observation is said to be “statistically significant.” (For example, if we flip a coin ten times, and we repeat those ten flips a thousand times, and each trial of ten flips results far from the 5-5 split and closer to the 10-0 split, we say that our observation is statistically significant.) We call this the “p-value,” and it answers the question: What is the probability that we saw these results by chance alone, and not because there is really something going on here?
Remember the p-value.
Back to the data at hand
Okay, so back to the 1 in 715 compared to 6 in 2,640. Are those two proportions (0.0014) and (0.0023) different enough to show that there is an association between vaccines and autism/ASD, or are they just different out of pure chance? What is the p-value of the comparison of these two proportions?
To answer those questions, we do a statistical test called “Fisher’s Exact Test” because the proportions are so small (“the values of the expected cells are too small”). (PLEASE NOTE: If I had access to the entire dataset, I could do a logistic regression, a more “robust” statistical analysis that takes into consideration all of the other variables presented by the patients, like age, gender, socioeconomics, number of vaccines given, etc.)
So what does the test tell us? This is the output in R:
First, the p-value is 1, meaning that it is certain that vaccination status is not associated with autism/ASD status. At the bottom is the Odds Ratio of 1.63. This tells us that, based on this sample alone, those who are vaccinated have 63% higher odds of being diagnosed with autism/ASD. But then look at the 95% confidence interval above that: 0.197 to 74.895.
The 95% confidence interval says: “We are 95% confident that the true odds ratio in the population from where this sample was drawn is between 0.197 and 74.895.” In other words, there is a good chance that those who are vaccinated have between 80% less odds of autism/ASD and 7489% more odds of autism/ASD.” As you can see, it’s impossible to know based on these numbers what the true association is. There is even a good chance that the odds are the same between the two groups, not less nor more.
So we conclude that there is no association.
There are other problems with this pediatrician offering up his own practice for evidence of an association between immunizations and autism. First, it’s a biased sample for the simple fact that he is so friendly to unvaccinated families. Second, it’s a biased presentation of the data because the pediatrician seems very friendly to the idea that vaccines are bad. Third, there is a wealth of evidence from well-planned-out studies that account for confounding variables (like sex, age, socioeconomic status, family history, etc.) and control for them.
In my opinion, presenting this “data” without a proper statistical analysis and without a true discussion of the potential sources of bias is not fair. It’s not fair to the patients at the pediatrician’s practice. It’s not fair to those parents who are confused about all this talk about vaccines and autism. And, finally, it’s not fair to autistic children to be told that something we have more and more evidence of being genetic could have been “prevented,” further inculcating in them the idea that they are flawed when they’re not.