By Sarena L. McLean MSc. (Epidemiology & Biostatistics), Health Sciences Researcher & Member of Doctors for Covid Ethics (2021)

The study “A Critical Analysis of All-Cause Deaths during COVID-19 Vaccination in an Italian Province” by Alessandria et al. aimed to evaluate the impact of COVID-19 vaccination on all-cause mortality while correcting for immortal time bias (ITB). Conducted in the Piedmont region of Italy, the study utilized a retrospective cohort design using a dataset from the Italian National Healthcare System, residents aged 10 and older, from January 1, 2021, to April 30, 2022 (Alessandria et al., p. 2). The authors aligned follow-up periods to ensure comparability between vaccinated and unvaccinated groups, dividing the cohort into groups based on vaccination status: unvaccinated, one dose, two doses, and three or four doses.

The study found an increased risk of death associated with receiving one or two doses of the vaccine, with less clear results for three or four doses. Covariates analyzed included gender, age, hypertension, diabetes, chronic obstructive pulmonary disease (COPD), cardiovascular disease, kidney diseases, cancer, and SARS-CoV-2 infection status. The authors used Cox proportional hazards models to estimate hazard ratios for all-cause mortality and employed Restricted Mean Survival Time (RMST) and Restricted Mean Time Lost (RMTL) for additional measures of survival and life expectancy (Alessandria et al., p. 4).

1. Strengths of the Study

The study used an innovative ITB correction, designed for accurate vaccination impact assessment. Additionally, the use of a large dataset from the Italian National Healthcare System provided a comprehensive dataset for analysis. Importantly, the authors chose to complete an all-cause mortality analysis versus examining COVID-19 death statistics, which are rife with misclassification errors. The all-cause mortality is not only more robust but also captures the impact from other indirect effects of COVID-19 such as delayed medical treatments.

2. Areas for Improvement

2.1. Overall Presentation

The article requires clarity, and a few items are missing. For example, when beginning to read the abstract, the authors lay out the analysis problem in general, which is an excellent consideration. However, the authors would best help the reader orient themselves first by providing an overview or context. We know nothing about the work, so help us follow along with you. There is no clear research question, and all reviewers want to see clear hypotheses, objectives, and purpose. I struggled at first to understand the unstratified sample size. Do not make work for the reader is always good advice.

There are many indications the authors were innovative and thorough in their approach and diligent in their process. Unfortunately, there were details and explanations missing that created a lot of questions during the review. It is challenging to discern if there are methodological or statistical analysis issues or are simply missing pieces of information creating a conversation about items that are already addressed and valid. The comments below are intended to demonstrate what arose in this review.

2.2. Descriptive Statistics

More detailed descriptive statistics are needed to provide context for the study population and their baseline characteristics. These statistics are vital for both lay readers and academics to understand the foundation of the investigation and provide essential information to assess both the statistical approach and interpret the findings. As a health science researcher, I am always looking for the measures of central tendency because I am interested in the distribution of these variables. Additionally, this item relates to the above point about unclear or missing details. The scientific method is predicated on the ability to replicate a study. We are unable to do that in this instance because the information is unclear. For example, I am assuming the authors chose not to provide details about vaccine manufacturers or types of vaccines for a substantial reason. However, there is no discussion about this. We are all challenged by word counts in publishing our work; yet some background and explanatory information is crucial.

2.3. Covariates/Confounders

Multicollinearity

The study includes several covariates that may have the potential to cause multicollinearity issues. Age and sex are unlikely to present problems. Cancer and infection are only moderately correlated with other conditions, while hypertension and COPD, though more correlated with other comorbidities, are still manageable within the model. However, cardiovascular disease is highly correlated with multiple conditions, notably hypertension and diabetes, which can complicate the analysis. The most significant concerns are diabetes and kidney disease, which are strongly correlated with each other and with other chronic conditions. This high correlation, known as multicollinearity, which can severely impact the stability of the regression coefficients, making it difficult to determine the individual effect of each variable. For example, the study’s regression model might show a misleadingly high effect of vaccination on mortality if the true effect is confounded by the combined impact of diabetes and kidney disease, leading to unreliable results. I would have preferred to read the authors’ comments on the associations among their covariates briefly in prose or shown in a table.

Limited Covariates

The covariates were limited in this study. This is generally problematic, because it limits the ability to control for factors that can significantly influence health outcomes. Specifically from a social epidemiological perspective, missing covariates highly correlated with poor health outcomes, such as socio-economic status (SES), access to healthcare, and mental health status, can lead to residual confounding where we may miss the opportunity to see what is truly driving the relationship(s) resulting in poor outcomes (Alessandria et al., p. 4). Importantly, these same factors were directly impacted by COVID-19 response measures. For instance, individuals with lower SES often face greater barriers to healthcare access and may have challenging living conditions, which can increase their vulnerability and result in an increase in both the burden of disease, and mortality. The study did include clinical covariates such as chronic diseases like hypertension, diabetes, COPD, cardiovascular disease, kidney diseases, cancer, and SARS-CoV-2 infection status, but broader social determinants of health were overlooked (Alessandria et al., p. 4).

Statistical Methods

An early question about the model was the exclusion of deaths in the first 2-week time period after vaccination. For example, we know that cardiac death occurs in that time frame; thus, the omission of this time period could impact the results.

It was unclear from the paper whether the authors examined and could assure readers that the data met the assumption criteria for the Cox model statistical analysis. The assumption required for a Cox model to be robust and valid is that the hazard ratios for the covariates must be constant over time (Alessandria et al., p. 5-6). Confirming this would be helpful for readers, as it would address concerns about the robustness of the statistical analysis. There is some information in the notes for Table 3, which do demonstrate the authors were addressing this issue along with a reference to the Schoenfeld’s test (Alessandria et al., p. 4). However, in my view, it remains unclear if the assumption is sufficiently addressed, particularly with respect to the confounders. By providing more comprehensive evidence of how the proportional hazards assumption was tested and addressed for each covariate (and confounders) including detailed plots or test statistics and explaining the stratification rationale would strengthen the study’s validity and address concerns effectively.

If the proportional hazards assumption is violated in a Cox regression model, it can cause several problems. The estimates of the hazard ratios may be biased, meaning they do not accurately show the true relationship between the covariates and the risk of the event occurring. This can lead to incorrect statistical tests and confidence intervals, causing wrong conclusions about the effects of the covariates. In the context of the study by Alessandria et al., if this assumption is violated, it could compromise the validity of their findings about the impact of COVID-19 vaccination on all-cause mortality. The observed effects might be due to changes over time rather than a true relationship, leading to erroneous results.

While Kaplan-Meier survival curves and a simplified Cox model do not directly address immortal time bias as comprehensively as the authors’ approach, these methods can be adjusted to partially mitigate ITB. Incorporating time-dependent covariates can help align risk periods correctly. As Tabachnick and Fidell (2013) explain, using time-dependent covariates within Cox regression can effectively handle violations of the proportional hazards assumption, providing more reliable results.

The advanced and complex nature of the original analysis, combined with missing or unclear foundational information, such as the research question and descriptive statistics, makes it difficult to determine if the authors’ findings are robust. For example, when reviewing Table 3.0, the covariate ‘Infection’ with SARS-CoV-2 shows hazard ratios of less than 1 compared to the population without infection, which is curious. Amon subjects who had received a single dose, the HR was 0.58, which suggests that the SARS-CoV-2-infected group had 42% lower hazard or risk compared to the non-infected group. I did not find any outcomes listed in the paper that suggested a lower risk made sense given the outcomes referred to are death and COVID-19-related deaths (Alessandria et al., p. 3).

Finally, an important suggestion is to clearly state what you are not going to do. It clears up any confusion and lays a boundary for why we do not venture into areas that are beyond the scope and resources we have in our work. Overall, it takes away a critique of the reviewers.

3. Conclusion

There could be valuable findings in this study, but the presentation is hindered by insufficient foundational information and very complex data analysis. Simplifying the methods and ensuring clarity in objectives and descriptive statistics would enhance the study’s reliability and accessibility. It would allow the reader to move beyond the initial information without so many questions. In this study, the investigators have been bold to embrace an untraditional statistical approach thus, to truly add to the body of literature, the detail is important. By embracing a more rigorous review process, we uphold the integrity of science and contribute to more reliable and impactful research.

For the authors, consider revising the study to address these concerns and recirculate their work. It is innovative yet complex, so clarity is key. I give credit to the authors for stepping out boldly to address gaps in the research.

4. References

Alessandria, M. et al. (2024) A Critical Analysis of All-Cause Deaths during {COVID}-19 Vaccination in an Italian Province. Microorganisms 12:1343 http://dx.doi.org/10.3390/microorganisms12071343

Tabachnick, B. G., & Fidell, L. S. (2019). Using Multivariate Statistics (7th ed.). Pearson.