Article Text
PDF
Abstract
Objectives Highly visible hospital quality reporting stakeholders in the USA such as the US News & World Report (USNWR) and the Centers for Medicare & Medicaid Services (CMS) play an important health systems role via their transparent public reporting of hospital outcomes and performance. However, during the pandemic, many such quality measurement stakeholders and pay-for-performance programmes in the USA and Europe have eschewed the traditional risk adjustment paradigm, instead choosing to pre-emptively exclude months or years of pandemic era performance data due largely to hospitals’ perceived COVID-19 burdens. These data exclusions may lead patients to draw misleading conclusions about where to seek care, while also masking genuine improvements or deteriorations in hospital quality that may have occurred during the pandemic. Here, we assessed to what extent hospitals’ COVID-19 burdens (proportion of hospitalised patients with COVID-19) were associated with their non-COVID 30-day mortality rates from March through November 2020 to inform whether inclusion of pandemic-era data may still be appropriate.
Design This was a retrospective cohort study using the 100% CMS Inpatient Standard Analytic File and Master Beneficiary Summary File to include all US Medicare inpatient encounters with admission dates from 1 April 2020 through 30 November 2020, excluding COVID-19 encounters. Using linear regression, we modelled the association between hospitals’ COVID-19 proportions and observed/expected (O/E) ratios, testing whether the relationship was non-linear. We calculated alternative hospital O/E ratios after selective pandemic data exclusions mirroring the USNWR data exclusion methodology.
Setting and participants We analysed 4 182 226 consecutive Medicare inpatient encounters from across 2601 US hospitals.
Results The association between hospital COVID-19 proportion and non-COVID O/E 30-day mortality was statistically significant (p<0.0001), but weakly correlated (r2=0.06). The median (IQR) pairwise relative difference in hospital O/E ratios comparing the alternative analysis with the original analysis was +3.7% (−2.5%, +6.7%), with 1908/2571 (74.2%) of hospitals having relative differences within ±10%.
Conclusions For non-COVID patient outcomes such as mortality, evidence-based inclusion of pandemic-era data is methodologically plausible and must be explored rather than exclusion of months or years of relevant patient outcomes data.
- Quality measurement
- Health services research
- Healthcare quality improvement
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Many hospital quality reporting stakeholders in the USA and Europe have excluded pandemic-era data from rankings, ratings and pay-for-performance programmes, indicating they do not intend to hold hospitals accountable for pandemic-era outcomes.
WHAT THIS STUDY ADDS
US hospitals’ COVID-19 patient burdens explained only a small amount of the variation in 30-day mortality among elderly non-COVID patients during the early months of the pandemic, and hospitals’ risk-adjusted performances did not vary meaningfully on average when high COVID burden months were excluded from the analysis.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
For non-COVID patient outcomes, evidence-based inclusion of pandemic-era data in hospital quality reporting is methodologically plausible and must be explored more rigorously rather than exclusion of months or years of patient outcomes data which may mask deteriorations in hospital quality unrelated to the pandemic.
Introduction
The 2022 edition of the US News & World Report’s (USNWR) Best Hospitals rankings used hospital outcomes data through the year 2020,1 representing the first time that quality outcomes reported by USNWR have overlapped with the era of the COVID-19 pandemic. USNWR methodologists were confronted with a decision as to what extent hospitals should be held accountable for non-COVID patient outcomes during this period of unprecedented stress to the healthcare system. Other quality reporting stakeholders faced a similar quandary, including the Centers for Medicare & Medicaid Services (CMS) which chose to exclude all data from 2020 in the Hospital-Acquired Condition Reduction Program penalty2 and to exclude 6 months of outcomes data from the Overall Hospital Star Rating.3 The Leapfrog Hospital Safety Grade has used a similar pandemic-era data period exclusion for several measures.4 In the UK, the National Health Service suspended its pay-for-performance (P4P) programme, the Quality and Outcomes Framework, for the duration of the pandemic to allow hospitals to focus on COVID-19 care without the additional stress of meeting performance targets,5 while several other nations across Europe saw changes in reimbursement for non-COVID care, though not necessarily based on performance.6 Ostensibly, these decisions were made with the best of intentions to prevent potentially unfair comparisons in outcomes due to pandemic-era stresses on hospital operations such as high COVID-19 census, which was the key pandemic-era concern prioritised for prediction by many institutions, including ours, throughout the pandemic.7
Critically missing from these decisions, however, have been publicly available analyses to support or refute the pre-emptive methodological decisions to implement months-long or years-long data exclusions for non-COVID patient outcomes. On the contrary, the established paradigm in hospital quality reporting has been to use risk adjustment to reduce the substantial differences between hospitals’ patient populations, providers and processes and allow for creation of meaningful performance benchmarks. The unprecedented nature of the pandemic appears to have steered quality measurement stakeholders away from this well-founded, data-driven approach, seemingly without any examination or presentation of evidence to support this paradigm shift. Inappropriate data exclusions could potentially limit the ability to accurately assess hospital quality, safety and experience, potentially mislead patients making decisions about where to seek care and mask improvements or deteriorations in quality unrelated to the pandemic that may have occurred. Here, we mirrored one of the most well-described and specific exclusion methodologies (put forth by USNWR) to assess to what extent hospitals’ COVID-19 burdens were associated with their 30-day mortality rates among non-COVID patients during the early pandemic to inform whether widespread non-COVID outcomes data exclusion is necessary.
Methods
We used the 100% CMS Inpatient Standard Analytic File and Master Beneficiary Summary File to include all Medicare inpatient encounters in the USA with admission dates from 1 April 2020 through 30 November 2020 (to allow for full 30-day mortality follow-up through the end of 2020). To reduce the possibility of results being impacted by inclusion of small hospitals with zero or few events, we restricted our analysis to hospitals with at least 25 non-COVID mortalities and tabulated the hospital-specific monthly proportion of encounters containing a COVID-19 diagnosis code (U07.1). Next, we excluded all encounters with a COVID-19 diagnosis in order to calculate hospital-specific observed/expected (O/E) 30-day mortality ratios among non-COVID patients using logistic regression to adjust for age, sex and individual comorbidities in the Elixhauser Mortality Index,8 using a similar modelling approach to the USNWR methodology.1 This approach allows for an approximately normal distribution of O/E hospital performance as well as controlling for any potential increases in non-COVID patient comorbidity burden during the pandemic. We further removed hospitals with O/E ratios above the 99th percentile and below the 1st percentile as outliers due to potentially incomplete or inaccurate claims data. Using linear regression, we modelled the association between hospitals’ COVID-19 proportions and O/E ratios, testing whether the relationship was non-linear using a 5-knot cubic spline function and Wald t-test for the linear combination of spline terms.
Finally, we calculated the alternative hospital O/E ratios after selective data exclusions mirroring the USNWR data exclusion methodology.1 Namely, we used the identical risk modelling approach described above, but for this alternative analysis we first excluded all monthly hospital-specific encounters from any month in which that hospital’s proportion of inpatients with COVID-19 was either more than the national average or more than 15%.1 Finally, we calculated and plotted the distribution of the pairwise relative difference of hospitals’ O/E ratios comparing their performance under this alternative analysis (in which monthly data were excluded) versus the original analysis (in which no monthly data were excluded). These pairwise relative differences serve as an indicator of the extent to which intrahospital performance changed from the application of these widespread monthly data exclusions.
Results
We analysed 4 182 226 encounters from across 2601 US hospitals with a mean (SD) age of 72.3 (13.0). The c-statistic for the 30-day mortality model was 0.76. The association between hospital COVID-19 proportion and non-COVID O/E 30-day mortality was statistically significant (p<0.0001) with r2=0.06. A 1% increase in COVID-19 inpatient proportion (eg, 5%–6%) was linearly associated (figure 1) with a 0.014 (95% CI 0.012, 0.016) increase in O/E mortality (eg, 0.900 to 0.914). The test for non-linearity was not significant (p=0.73).
Relationship between hospital-specific COVID-19 burden and observed/expected (O/E) 30-day mortality among non-COVID Medicare patients hospitalised during the early COVID-19 pandemic. COVID-19 burden was defined as a hospital’s proportion of Medicare patients who were hospitalised with COVID-19 during the time period. Expected 30-day mortality was adjusted for age, sex and Elixhauser comorbidities. The relationship between COVID-19 burden and non-COVID O/E 30-day mortality was approximately linear and statistically significant (p<0.0001), but weakly correlated (r2=0.06).
In the alternative analysis in which hospital-specific months with greater-than-national-average COVID-19 proportion were excluded, there were 2 750 794 encounters (65.8% of original cohort) across 2571 hospitals (table 1). The c-statistic for the 30-day mortality model in this cohort was also 0.76. The median (IQR) pairwise relative difference in hospital O/E ratios comparing the alternative analysis with the original analysis was +3.7% (−2.5%, +6.7%), with 1908/2571 (74.2%) of hospitals having relative differences within ±10% (eg, an O/E=1.10 or 0.90 vs an original O/E=1.00 when no months were excluded) and 2427 (94.4%) within ±25% (figure 2).
Histogram of relative difference in hospital-specific 30-day mortality observed/expected (O/E) comparing all data versus data excluding months of non-COVID patient data based on hospital-specific COVID burden. Y-axis is the total number of eligible hospitals (n=2571); x-axis is the relative difference in hospital-specific 30-day mortality O/E ratio comparing inclusion of all pandemic-era data for non-COVID patients versus a methodology of excluding hospital-months of non-COVID patient data based on the hospital-specific COVID burden (1.0=no difference).
Comparison of hospital-level characteristics when including all data versus excluding high-COVID burden months
Discussion
Our analysis indicated that during the initial period of the pandemic (April through November 2020), only 6% of the variation in US hospitals’ risk-adjusted non-COVID 30-day mortality could be explained by their COVID-19 census burdens. Further, this relationship was approximately linear, making it potentially amenable for adaptation as a post hoc hospital-level ‘COVID-census-adjustment’ to outcome rates used in quality reporting. We found that roughly three-fourths of hospitals performed within ±10% relative difference in observed versus expected 30-day mortality among non-COVID beneficiaries regardless of whether we included or excluded hospital-specific months of data with greater than the national average COVID-19 burden. Excluding hospital-months based on proportion of inpatients with COVID-19 resulted in minimal change to hospitals’ benchmarked performances at the expense of discarding the patient outcomes assessment for more than one-third of non-COVID encounters during this time frame.
Limitations include our usage of American data only, which was necessitated by our access to US Medicare claims in the absence of a centralised European or global claims data set. Similarly, the decision to include only hospitals with least 25 non-COVID mortalities during the study period due to statistical df considerations resulted in the exclusion of small US hospitals which may have shouldered a greater proportionate COVID-19 burden. However, our final cohort of 2601 hospitals is comparable to the CMS Overall Hospital Star Rating in which approximately 3000 hospitals are ultimately eligible for a rating each year and represents a majority of US hospitals. Second, our analysis relied on claims data and was therefore unable to adequately distinguish between exclusion of COVID-19 cases which may have been healthcare acquired rather than the main cause of admission, though there is evidence that hospital-acquired COVID-19 infections were not highly prevalent9 in the early stages of the pandemic. Third, a more thorough economic impact analysis of the small proportion of hospitals that had significant differences in performance based on the methodology applied would be needed prior to implementation of any new methodological approaches in specific P4P programmes. For example, in the USA it would be prudent to investigate specific Hospital Readmission Reduction penalty implications for all hospitals, or likewise in the UK to investigate the hospital-specific implications if the Quality and Outcomes Framework P4P was not suspended during the pandemic. In fact, it may be more appropriate for these stakeholders themselves to conduct and publish these analyses publicly in the interest of transparency. Here, though, we specifically mirrored the USNWR exclusion methodology which is a hospital quality ranking system which does not have implications for hospital payment in order to first share a broad example of how pandemic-era data could plausibly be analysed and how results may or may not vary regardless of data exclusion methodologies. Similarly, most P4P programmes focus on specific conditions and procedures (such as hip and knee replacement or heart failure), and further investigation or alternative methodological approaches in those specific cohorts may yield different results than in our all-non-COVID patient 30-day mortality analysis. Because of sample size considerations, the early pandemic months may not be sufficiently powered in some conditions and thus be limited to large academic medical centres, whereas our approach to include all non-COVID conditions allowed for inclusion of most US hospitals. Finally, our analysis relies on a framework which assumes that hospitals’ COVID-19 census burdens were a key influencer of hospital operations and quality outcomes during the pandemic, though other factors could be explored such as staffing, burnout and turnover.
A recent JAMA article showed that leading up to the pandemic, the USA experienced a decade of improvement in adverse patient events from 2010 to 2019.10 The decade after the pandemic will bring improvements in electronic health record sharing and standardised collection of clinical data, and hospitals must find innovative ways to improve quality outcomes and increase accountability to their patients through transparent public reporting, rather than exclusion, of such data. In fact, such accountability may be of greater importance during times of uncertainty, such as the pandemic, which led to a decreasing amount of public trust in medical and scientific leaders.11 It is imperative in the years ahead to extend analyses to understand hospital quality outcomes and performance, and hold hospitals accountable for the quality of care provided during the pandemic.
The observed versus expected 30-day mortality rates among non-COVID Medicare beneficiaries during the first year of the COVID-19 pandemic in the USA varied only slightly in correlation with their burden of hospitalised COVID-19 beneficiaries, and exclusion of data during high-COVID census months did not meaningfully change the majority of hospitals’ benchmarked mortality performance. For non-COVID patient outcomes such as mortality, evidence-based inclusion and adjustment for pandemic-era data is methodologically plausible and must be explored more rigorously prior to the pre-emptive and paradigm shifting exclusion of months or years of patient outcomes data.
Ethics statements
Patient consent for publication
Footnotes
Twitter @Pollock_BD
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; internally peer reviewed.