When the Numbers Don’t Say What You Think They Say
There is a pediatrician who has been making a name for himself in anti-vaccine circles. He released a letter dated March 12, 2019, addressed to lawmakers where he uses some “data” from his medical practice to apparently try and show that vaccines are associated with autism. Never mind the recent and not-so-recent large-scale, well-designed and well-controlled studies that have found no association between vaccines and autism. No, folks. He just draws some numbers from his practice and wants to convince us otherwise.
The, uh, “data”
Check it out:
Out of a total of 3,355 patients, 715 are unvaccinated and 2,640 are “partially vaccinated.” (He states in the letter that almost all of his clients refuse at least one vaccine. More on that in a minute.)
In the unvaccinated group, only one case of “autism/ASD” is noted. In the partially vaccinated group, six cases of “autism/ASD” are noted. The pediatrician then concludes that his patients, because they’re unvaccinated and partially vaccinated, have less autism than what more respectable groups like the American Academy of Pediatrics (AAP) and the Centers for Disease Control and Prevention (CDC) have calculated to be the autism prevalence in the United States.
He also seems to imply that there is less autism in the unvaccinated group, as his data clearly show… Except that it doesn’t show that at all.
Some quick biostatistics
You can skip to the next section if you know how statistics work. If you don’t, here is a super-simplified way of looking at it.
When you flip a coin, you expect to get either heads or tails, right? If you flip a coin that is perfectly made, and you flip it ten times, you would expect 5 heads and 5 tails, right? What if you get 6 heads and 4 tails? You wouldn’t be too shocked. It kind of makes sense that a 6-4 split would happen. But what if you got 9 heads and 1 tail?
You’d probably be surprised, and that’s okay. On the average and in the long run, when you flip a coin ten times, you’re going to get a bunch of 5-5 splits, some 4-6 and 6-4, fewer 7-3 or 3-7, very few 8-2 and 2-8, and even fewer 1-9 or 9-1. You will get a 10-0 or 0-10 less than 1 in a thousand times that do ten flips.
In the long run, the distribution (known as the “binomial distribution“) of the coin tosses looks like this:
Over the centuries, people have calculated the probabilities of these things happening. When we do our own experiments, we compare the results with the probability tables of experiments like the coin flip I describe above. These probability tables exist for all sorts of data: continuous data like blood sugar levels, binomial data like sick/not sick outcomes, and even categorical data like “belongs to group A or B or C or D.”
The major rule is that if our results have a less than 5% chance of happening, our observation is said to be “statistically significant.” (For example, if we flip a coin ten times, and we repeat those ten flips a thousand times, and each trial of ten flips results far from the 5-5 split and closer to the 10-0 split, we say that our observation is statistically significant.) We call this the “p-value,” and it answers the question: What is the probability that we saw these results by chance alone, and not because there is really something going on here?
Remember the p-value.
Back to the data at hand
Okay, so back to the 1 in 715 compared to 6 in 2,640. Are those two proportions (0.0014) and (0.0023) different enough to show that there is an association between vaccines and autism/ASD, or are they just different out of pure chance? What is the p-value of the comparison of these two proportions?
To answer those questions, we do a statistical test called “Fisher’s Exact Test” because the proportions are so small (“the values of the expected cells are too small”). (PLEASE NOTE: If I had access to the entire dataset, I could do a logistic regression, a more “robust” statistical analysis that takes into consideration all of the other variables presented by the patients, like age, gender, socioeconomics, number of vaccines given, etc.)
So what does the test tell us? This is the output in R:
First, the p-value is 1, meaning that it is certain that vaccination status is not associated with autism/ASD status. At the bottom is the Odds Ratio of 1.63. This tells us that, based on this sample alone, those who are vaccinated have 63% higher odds of being diagnosed with autism/ASD. But then look at the 95% confidence interval above that: 0.197 to 74.895.
The 95% confidence interval says: “We are 95% confident that the true odds ratio in the population from where this sample was drawn is between 0.197 and 74.895.” In other words, there is a good chance that those who are vaccinated have between 80% less odds of autism/ASD and 7489% more odds of autism/ASD.” As you can see, it’s impossible to know based on these numbers what the true association is. There is even a good chance that the odds are the same between the two groups, not less nor more.
So we conclude that there is no association.
In conclusion
There are other problems with this pediatrician offering up his own practice for evidence of an association between immunizations and autism. First, it’s a biased sample for the simple fact that he is so friendly to unvaccinated families. Second, it’s a biased presentation of the data because the pediatrician seems very friendly to the idea that vaccines are bad. Third, there is a wealth of evidence from well-planned-out studies that account for confounding variables (like sex, age, socioeconomic status, family history, etc.) and control for them.
In my opinion, presenting this “data” without a proper statistical analysis and without a true discussion of the potential sources of bias is not fair. It’s not fair to the patients at the pediatrician’s practice. It’s not fair to those parents who are confused about all this talk about vaccines and autism. And, finally, it’s not fair to autistic children to be told that something we have more and more evidence of being genetic could have been “prevented,” further inculcating in them the idea that they are flawed when they’re not.
EpidemioLogical 
As you well know, I’ve chosen a field where I work directly with US government agencies, typically as a contractor. It has historically paid better than my GS counterparts, although of late, the government is winning the technical worker arms race in outpaying qualified workers in technical fields. Currently, I command a premium amongst my peer contractors, as I have what those GS workers lack an they lack, being younger, experience an deep knowledge of the technologies we use and keep running, at all levels, from desktop support through advanced server services and cloud based services.
To a level that quite astonished a VP from Xerox’s field service, government services group, as we worked together to finish what was discovered to be a heavily misconfigured multi-function printer/copier/scanner/fax/surprised-it-doesn’t-make-coffee machine on a secure network. Apparently, he lost a critical employee and having a clearance, he had to assist. He was also shocked that I understood why Xerox paid out $800k per year for licensing support for secure network authentication technology, while making nowhere near enough to cover that licensing cost. Get a US government contract to provide MPD units and support them for the unclassified networks, you’re also required to support the classified units, which are uncommon, comparatively. As in, we have three such devices on one secure network, but have 30 or so devices in just one building and other buildings have such devices per office, leading to hundreds of such devices on our base. That, out of hundreds of such bases that *we* support.
He was relieved to be able to discuss LDAP with someone who was rusty, but familiar with the protocol, not have to explain certificate services (I had related the annoyance of having to halt a major project of replacing self-signed certificates long enough to give a class to an entire LAN-WAN shop on how certificates work and why precision in what something is named to a connecting device is more important than the actual server name, but the alias to the connecting device is what the certificate being requested is critical, so server bob isn’t certificate for bob, it’s the alias mail.my.organization.mil was critical, as users connecting seek mail.my.organization.mil, not bob and most certainly assant6ittn4997 (an in joke for one installation I was on, as that would nearly be a valid name, save for a slight obfuscation of a purpose in the naming convention, but those in would know precisely which base in Qatar I was on, which organization I worked for and a second joke, as the last would be normally sequential and in its heyday, that would basically say we had twice as many servers as users). We discussed a wide ranging area, while waiting for a reboot or an update to come down, get ported to the secured network or during a brief, required consultation break, all IT and even some life experiences, as we’re nearly the same age, he’s a few months younger than I am, so he doesn’t recall JFK being shot, I do.
We worked for 2.5 hours past my shift end, largely due to a very odd error on the part of his technician, who had initiated configuration, although an incident initially likely did distract him. A very serious security incident that he had precipitated by coat tailing one of our techs. Happened right in front of me and it was fast enough that I couldn’t stop him and he should have known better anyway.
But, I’ve seriously digressed from my point. Wildly, at length, but to give a hint, both my careers and history of holding a position of trust in the form of a security clearance is longer than many managers I interact with on a daily basis than said manager’s been alive.
The number one employer in the US of disabled people is the US government. We have deaf workers all around our installation. I’m really going to have to learn ASL, both to better communicate with them and well my tinnitus is growing worse of late – much worse, one ear sounds like my hearing aid was blocked and suffering varying tone feedback and my hearing aids need to be replaced (on my to-do list, gotta find an audiologist in this area first).
I’ve also dealt with many, many workers on the ASD spectrum, ranging from Asperger’s through rather “severe” workers nicknamed wallsliders, as they were so involved in thinking of a problem, they literally slid along the wall for guidance, allowing that much more “CPU cycles” to work on a muttered, insanely complex mathematical formula. That’s OK, I think aloud at times as well, working out a resultant set of policy in a very complex Active Directory environment and frankly, I come up with an accurate result a tad faster than some rather serious hardware.
Boolean algebra? Routine, child’s play – literally, for me. Regular Expressions? Can do them when I’m drunk, hell, on the hobbyist side, I have, get extremely complex filters created quite quickly, correctly and don’t require addressing again until conditions are added after.
Give me a command line, well, one network we had had annual “attacks” performed by our CERT, our higher command’s IA team and the NSA red and blue teams. While my peers dreaded those tests, I quite enjoyed it and both sides learned new things about offense and defense. I succeeded where they all failed, overall, for one reason, I’m not a specialist. In basic mathematics, I’m most certainly learning challenged, to put it mildly. I joke, not very far from the truth, that counting to 20 requires my removal of footwear and socks and I get into serious legal trouble when counting to 21.
But, I intuited the Lorentz transformation and general relativity’s link in junior high school, while being incapable of giving the months of the year in proper order, which I can, with concentration, do today.
Dyslexia means thinking differently.
Which is also true for those with Autism.
Where they’re socially awkward, at best, I’m socially fluid. We can think in similar algorithms, in advanced mathematical ways, I’m utter rubbish at calculating numbers without at least a slide rule or other form of calculator.
Even an abacus, it’ll just take me 5 – 10 minutes how to remember how to work the damnable thing, largely to filter out the time I mastered using the thing to work out binary problems.
Now, if the US government highly values people with autism, how in the hell, especially when they’re solving problems nobody else on the planet are solving, things like sonar processing, cryptography, breaking cryptography and signal processing, is getting things wrong, especially given our immense successes in all of those fields.
Especially, when they’re dealing, daily, with things I have trouble comprehending, outside of Infinite Fun Space and I’m a fairly bright fella, given less than half a chance.*
*One would need a bit of experience with the Culture to get that joke. Still leaving me as an Outside Context Problem. 😉
LikeLike
Pingback: Why Will Paul Thomas’ Patients Be Excluded from School in Oregon? – VAXOPEDIA
I would have asked that pediatrician what level of autism as defined in DSM V. There is a vast difference in required support.
Which actually depends on which genetic sequence is the cause of the autism spectrum behaviors.
LikeLiked by 1 person
Hola amigo – I have been trying to find this Dr. Gaven, I think Paul Thomas made him up. Most of his mentions have been scrubbed. Even if real, I think they violated HIPAA and basic research methodology.
c/s Professor Xeno
LikeLiked by 1 person
Basic research methodology, at the very least.
LikeLiked by 1 person