Vaccine efficacy 'cheap trick' by exclusion

Simply ignoring - rather than misclassifying recently vaccinated cases - also creates the illusion of efficacy.

Our main simulated example of the vaccine efficacy ‘cheap trick’ was based on the assumption that those who tested positive for infection within a short period after vaccination (e.g., 7, 14 or 21 days) were classified as unvaccinated. In a typical mass (placebo) vaccine roll-out this cheap trick guarantee the illusion of efficacy starting at over 90% before gradually waning to less than 20% after three months.

We made it clear that this assumption was not made in all the observational studies and randomised controlled trials that claim high efficacy for the covid vaccines, but that it was just one of the many variants of cheap trick that all lead to exaggerated efficacy claims (see also this paper by Funge et al for even more examples).

However, many said that our example was not representative because it was possible that, shortly after vaccination, cases were simply excluded rather than miscategorised as unvaccinated. Here we show that this also leads to the illusion of efficacy for a placebo vaccine (and, as before, also for a vaccine that increases infectiousness).

Simple Example

Consider an observational study in which 1000 people get the vaccine and 1000 do not, where the weekly infection rate is 1%. We will ignore the possibility that those infected are less likely to get reinfected.

After 4 weeks we will observe 10 infected cases in each of the two groups each week. So there will be 40 cases out of the 1000 unvaccinated - a rate of 4%. But if we exclude those infected cases within the first 3 weeks, there are just 10 cases out of the 1000 vaccinated among the vaccinated, a rate of 1%.

Since vaccine efficacy is defined as

When, expressed as a percentage, the calculated efficacy is 75% for this three week delay (for a two week delay the efficacy is 50%, and for a one week delay it is 25%).

Even for a vaccine that increases the infection rate for those vaccinated - say one where the weekly infection rate is 1.2% compared to the 1% for the unvaccinated, we would still get the illusion of efficacy. In our observational study, although in reality there will be 12 out of the 1000 infected each and every week, only the 12 in week 4 would actually be categorised as cases. So, the vaccinated case rate would be calculated as 1.2%, rather than the actual 4.8% (not obvious). This yields a vaccine efficacy of 70%.

How the illusion can be avoided

The illusion is an example of survival bias. If, instead of counting ‘people vaccinated’, we count number of ‘person weeks vaccinated’, then the above problem would be avoided because, for the first 3 weeks, the vaccinated would be considered not fully vaccinated. So, in total (for a placebo vaccine) we would have:

• 10 cases in 1000 person weeks fully vaccinated

• 30 cases in 3000 person weeks partially vaccinated

• 40 cases in 4000 person weeks unvaccinated

Using this counting method would result in the correct vaccine efficacy of 0% whether we consider fully or partially vaccinated compared to unvaccinated.

However, in practice the person weeks counting method is not used. During a vaccine roll out period, when a high proportion of people are vaccinated, the number of infected cases for those people ‘ever vaccinated’ are compared against the cases in those people ‘never vaccinated’. In the following simulated vaccine roll-out scenario we use this convention.

Efficacy of a placebo vaccine

In the example presented here we assume there is a population of 110,000 people of whom 100,000 are vaccinated over an 11-week period with a typical ‘ramp up’ (starting with 1000 vaccinated in week 1 and peaking with 30,000 vaccinated in week 6). Here we assume a fixed weekly rate of infection of 1%. Then we consider a placebo vaccine under each of the three exclusion periods: 7-day, 14-day and 21-day.

Any infections of those vaccinated inside the exclusion period are removed from the analysis (as opposed to being categorised as unvaccinated cases). So, the unvaccinated category contains cases that are genuinely never vaccinated.

With these assumptions we still get the following efficacy rates:

• 7-day exclusion period: 67% efficacy at week 2 tailing off to 10% by week 8

• 14-day exclusion period: 86% efficacy at week 3 tailing off to 12% by week 9

• 21-day exclusion period: 94% efficacy at week 4 tailing off to 13% by week 10

So, with the 21-exclusion period, which is the period used by the ONS for vaccine efficacy calculation, results in a 94% initial efficacy for a placebo vaccine (although as noted here the ONS do use the person years formulation to avoid the survival bias issue).

Efficacy for a vaccine that increases infectivity in those vaccinated

What if the infection rate of the vaccinated is higher than that of the unvaccinated, i.e., the vaccine enhances infectivity? It turns out that excluding cases shortly after vaccination also produce the illusion of efficacy for such a vaccine.

This time the only change from the placebo vaccine example above is that we assume the weekly infection rate for the vaccinated is 1.25% (as compared to the 1% for the unvaccinated).

With an ineffective vaccine we get the following efficacy rates:

• 7-day exclusion period: 58% efficacy at week 2 tailing off to negative efficacy by week 8

• 14-day exclusion period: 82% efficacy at week 3 tailing off to negative efficacy by week 9

• 21-day exclusion period: 90% efficacy at week 4 tailing off to negative efficacy by week 10

So, with an 21-exclusion period, as used by the ONS, this results in an incredible 90% efficacy for a vaccine that has a 25% higher weekly infection rate for those vaccinated.

Here is a video demonstrating all the above the results:

The excel spreadsheet:

Vaccine Efficacy Illusion Ignoring Recently Vaxxed Cases

42.1KB ∙ XLSX file

Download

Download

In these examples the illusion would be eliminated if, instead of considering the total ‘ever vaccinated’ as the denominator when calculating the proportion of cases in the vaccinated in any given week, we consider the total ‘fully vaccinated’ as the denominator.

The illusion is difficult to avoid

But, it turns out that even this adjustment fails to prevent the illusion occurring in observational studies. Let us explain.

Here we consider a simulated example in which we observe a vaccine rollout over a 10-week period for a population of 500,000 people in which 250,000 people get vaccinated at some point during the 10 weeks and 250,000 are never vaccinated during the same 10 weeks.

Again, we assume the vaccine is a placebo and that the weekly infection rate, for vaccinated and unvaccinated alike, is 1%.

We assume the vaccine rollout starts with 1000 vaccinated in week 1, peaking at 60,000 in week 8. Then, if we exclude cases that occur within 3 weeks of vaccination by the end of week 10 we will have counted a total of 2,550 cases (compared to the actual total of 8,950). Whereas the case rate for the ‘ever vaccinated’ would be 1.02% (2,550 cases from the 250,000 ever vaccinated) as there are only 110,000 ‘fully vaccinated’ people the case rate for the fully vaccinated would be higher at 2.32%. But the case rate for the unvaccinated over the 10 week period is 10%.

The vaccine efficacy for the ever vaccinated is therefore 90% and even if we focus only on the fully vaccinated, the vaccine efficacy of 77%.

As the video below shows, for the 1-week and 2-week exclusion periods, for the fully vaccinated we calculate vaccine efficacy of 71% and 75%, respectively.

Summary

For both a placebo and an ineffective vaccine which enhances infectivity the cheap trick, by exclusion, guarantees a high efficacy, which wanes until a point when a booster is required to restore efficacy and enhance profits.