The CDC's data on covid vaccine safety signals

Compared to all previous non-covid vaccines there are many worrying signals in the Pfizer and Moderna vaccines

A standard way to determine whether there is a genuine concern that a new vaccine may be causing a particular type of serious adverse event (SAE), such as ‘myocarditis’, is to compare the proportion of adverse event reports associated with the new vaccine that include myocarditis with the proportion that include myocarditis in other vaccines.

For example, suppose that in Vaccine Adverse Event Reporting System (VAERS):

• For the new vaccine there are 1000 reports of SAEs of which 300 include myocarditis.

• For all other vaccines there are 2000 reports of SAEs of which 100 include myocarditis.

Then 30% of the reports for the new vaccine include myocarditis compared to just 5% for all other vaccines. This is used to calculate the proportional reporting ratio (PRR) which is simply 30% divided by 5% in this case, and here the PRR is equal to 6 for a myocarditis SAE. The fact that there is a much greater proportion of myocarditis events in the new vaccine compared to previous vaccines should be concerning.

To be considered a potential ‘safety signal’ by the Centers for Disease Control and Prevention (CDC), the PRR must be at least 2 and must also have a chi-square test statistic value of at least 4. The chi-square test statistic is a classical statistical test of significance that takes account of the total number of events and reports.

In this example the chi-square is 361 which is highly significant; but if, say we divide the values in the above example by 100 (so 3 out of 10 compared for the new vaccine and 1 out of 20 for all other vaccines), then although the PRR is still 6, the chi-square value is just 1.9 which would not be significant under the CDC’s rule.

The CDC conducted a PRR analysis of adverse events reported in the VAERS system for the Pfizer and Moderna covid vaccines for the period Dec. 14, 2020, to July 29, 2022. For each different type of adverse event reported, the CDC calculated the PRR of every SAE for each of the two covid vaccines against all other non-covid vaccines. They also computed the PRR of every SAE for Pfizer against Moderna.

Zachary Stieber of the Epoch Times obtained the results of the CDC analysis through a Freedom of Information Act request after the CDC refused to make the results public. Zach sent the spreadsheets with the results to myself and colleague Josh Guetzkow asking for comments. Zach has now published his article on this - with links to all the data - including some of the comments that Josh and I provided. The article is also published on the CHD website. Josh has also now published his own extensive article.

For completeness, here is the full list of comments I made to Zack (I made these after seeing Josh’s comments and so are in addition to what he had already covered):

• Since these PRRs are just the ratio of the proportions what it means of course is that all that it is ‘measuring’ is the extent to which there are different categories of SAEs between the covid vaccines and other vaccines. There must therefore be a whole bunch of SAE categories not listed here for which the PRR >2 for the non-covid vaccines (meaning the non-covid vaccines are relatively more dangerous with respect to those SAEs).

• Now while it is still important to determine the extent to which specific SAEs are much more prevalent in the covid vaccines, ultimately what matters most is simply the overall proportion of reports that are classified as serious.

  • For the 18+ age group the proportions are: 11.1% for the covid vaccines (73,178 out of 660,643) compared to 5.5% for non-covid vaccines (13,287 out of 242,091). That is a very important signal especially as this age group is where the vast majority of reports are. It is a highly significant difference. Therefore, if the CDC wish to claim that the probability of SAEs with the covid vaccines is not significantly higher than that of other vaccines the onus is on them to provide an alternative causal explanation for this difference.   

  • For the 12-17 age group: 6% for covid (1,893 out of 31,447) compared to 5.3% for no-covid (1,598 out of 30,084). This is a statistically significant difference.

  • For the 5-11 age group: 2.6% for covid (319 out of 12,329) compared to 3.8% for non-covid (1,111 out of 27,960) (surprising as this too is a significant difference in the opposite direction).

• Of course, we must also compare the total numbers of SAEs reported. The total number of SAE reports for the Pfizer and Moderna covid vaccines in just 19 months (73,178) is over five times the total number of SAE reports for all other non-covid vaccines (13,278) in 163 months (since 1 Jan 2009).

• Another incredibly important statistic is the proportion of deaths (which is only given for the 18+ age group) which is 14% in the covid vaccines (10,169 out of 73,178) compared to just 4.7% (618 out of 13,278) in non-covid vaccines. If the CDC wish to claim that the probability a covid vaccine adverse event results in death is not significantly higher than that of other vaccines the onus is on them to come up with some other causal explanation for this difference.  

• Note that there are many instances of differently named SAEs that really should be merged because they might be obfuscating the scale of some of the problems. For example, in the serious 18+ age group worksheet we have separately: cardiac failure acute, cardiac failure, infarction, myocardial strain, myocardial fibrosis – but not simply ‘myocarditis’. These are not mutually exclusive categories and spreading the same condition between categories obviously reduces the frequency counts per category, resulting in an underpowered chi-square test and a reduced chance of triggering a significant PRR under the CDC’s rule.

• Likewise, ‘myocarditis’ does not appear in the 18+ age group serious worksheet but it is in the serious 12-17 age group worksheet where it is the most serious after ‘chest pain’ (ignoring the covid specific material and generic stuff like ‘intensive care’). Presumably, in the older age group cases of myocarditis are simply subsumed within the other cardiac failure SAEs.

  Josh commented on this:

  “Keep in mind though that if a certain type of SAE was reported less than 3 times for all other vaccines, it will not show up as a safety signal here even if it was reported a million times. So it could be that there are many other cardiac failure AEs for 12-17 year olds that simply don't show up because they were never reported in the past. It's a huge blind spot with this method.”

• In the 18+ age group, non-serious AE, worksheet there are two categories “Menstruation irregular” and “Heavy Menstrual bleeding” both with a lot of cases and high PRR. Presumably these should have been merged (as there may be some overlap between them) but the key question is why these are not included at all in the serious categories? On a related note, since neither still birth nor miscarriage appear anywhere here does this mean they did not meet the threshold of N>=3, PRR>2 and chi-squared>4? 

  Josh commented on this:

  Yes, I agree about merging. There are other menstrual categories as well. Assuming there are few or no serious issues with the women reporting menstrual issues, then there is no reason it would show up in the serious reports (see definition in my previous e-mail as to what counts as a serious report). Regarding miscarriages (aka spontaneous abortions), yes that is my inference. I am assuming they did this for every AE/PT that had been coded from the reports, but we don't know for sure because we don't actually know what methodology they used. 

• For most of the SAEs considered the chi-squared is so high that, from a Bayesian perspective, the probability that the true rate of the SAE of the covid vaccines is not higher than that of the non-covid vaccines is essentially zero.  So, it is up to the regulators to come up with some other causal explanation for the difference if they wish to claim the probability of any of these SAE’s is not increased with the covid vaccines.

• Regarding the difference between Moderna and Pfizer: By far the most important difference is in the overall proportions of serious cases: for Moderna the proportion is 9.6% (33,003 out of 342,456) compared to 12.6% for Pfizer (40,193 out of 318,346). This is a highly significant difference that cannot be ignored (in fact, if you remove the many serious AEs that are specifically connected with injection site – for which Moderna has overwhelmingly more than Pfizer - then difference would be much greater still).  So again, the onus is on the regulators to come up with some other causal explanation for the difference if they wish to claim the probability of a serious AE with Pfizer is not significantly higher than that of Moderna.  Ideally, we would also like to compare these with all previous vaccines.

• The only other really interesting difference between Moderna and Pfizer is – as mentioned above - that there is a significantly higher rate of injection site SAEs with Moderna.

In summary: We do not believe that the PRR is an ideal method for finding safety signals because it only identifies significantly new, different or unexpected SAEs compared to other vacciness and does not consider underlying population rates for the SAE conditions. However, there are so many red flags for the two covid vaccines the CDC analysed that they should reconsider their decision not to halt further roll-out of these vaccines until these SAEs are more thoroughly investigated.