Problem

In October 2015, the Centers for Medicare and Medicaid Services (CMS) mandated that Medicare-participating hospitals report the sepsis bundle compliance rate as a quality process measure. Our healthcare system had been active in sepsis quality improvement (QI) in the decade predating the CMS mandate, with a standardised clinical practice for sepsis, ongoing monitoring of Surviving Sepsis Campaign (SSC) sepsis resuscitation bundle compliance, quarterly calculation of sepsis-related standardised mortality ratio (SMR) and implementation of computerised decision support systems (CDSSs) designed to detect potential episodes of sepsis in real time to alert clinicians.1 At the start of this study, our SSC bundle compliance was 33%, our sepsis-related hospital mortality was 14.5% and our sepsis-related SMR (observed/expected mortality calculated using APACHE IVa severity adjustment) was 0.75.

We convened a system-level QI team in the summer of 2015 with the specific aim of improving the CMS sepsis bundle compliance from 30% to 40% across 20 acute care hospitals in Banner Health within 1 year.

Background

Sepsis is a syndrome of life-threatening organ dysfunction caused by dysregulated host response to infection.2 Sepsis leads to 20% of all intensive care unit (ICU) admissions and is the most common cause of death in non-cardiac ICUs,3 leading to over 180 000 fatalities annually in the USA.4 Rivers and colleagues performed a randomised controlled trial in 2001 that showed a resuscitation protocol designed to rapidly achieve specific haemodynamic goals reduced the mortality of sepsis from 46.5% to 30.5% (p<0.009).5 Subsequently, SSC and the Institute for Healthcare Improvement promoted the concept of employing a sepsis resuscitation ‘bundle’—a list of interventions including goal-directed fluid and vasopressor resuscitation, blood cultures and antibiotics that all must be completed within a specific time window in order for compliance to be achieved.3 SSC posited that sepsis bundle compliance could be used as a process measure encompassing sepsis quality of care. An international SSC-sponsored QI project promoted the use of sepsis bundle compliance as a quality process measure and allowed individual hospitals to develop their own compliance interventions—including screening of patients to detect the onset of sepsis and protocols to implement bundle interventions. One hundred and sixty-five ICUs and emergency departments (EDs) participated and succeeded in improving their mean sepsis bundle compliance rate from 10.9% to 31.3% over the course of 2 years.3

However, three subsequent randomised controlled trials that cumulatively enrolled 4201 patients at 138 EDs and ICUs internationally showed that sepsis resuscitation bundles provide no survival benefit compared with usual care.6–8 Nevertheless, CMS mandated that Medicare-participating acute care hospitals in the USA report sepsis bundle compliance as a quality measure starting in October 2015.

Measurement

Since we chose the CMS sepsis bundle compliance rate as our outcome variable, we were able to ‘piggy-back’ our QI effort onto the the data abstraction system required to report this variable to CMS. This gave us access to two full-time data abstractors with associated support personnel whose workflow followed a detailed specifications manual provided by CMS9 entailing a procedure for sampling patients’ medical records post discharge and determining which patients should be included in the numerator and denominator of the compliance rate. Inclusion in the numerator required measurement of lactate, drawing blood cultures, administration of appropriate intravenous antibiotics, administration of fluids and vasopressors (if necessary), reassessment of haemodynamic status and repeat measurement of lactate (if necessary). The first three interventions needed to occur within 3 hours of presentation of sepsis and the remaining interventions within 6 hours of presentation of septic shock (see box). Failure of any single element results in failure of bundle compliance.

We established statistical process control (SPC) charting of monthly CMS sepsis bundle compliance rates with 99% control limits (±3 SD), as described by Benneyan et al.10 Pooled system-level CMS data collection achieved denominators in the range of 135–172 patients per month, including a total of 811 patients with sepsis in the first 6 months of the study. The monthly system-level compliance rate in the preintervention phase of our study ranged from 16% to 31% with a mean of 25.8% and baseline rate of 28.5% for purposes of SPC analysis (after exclusion of data from the second month of the study that fell below the lower control limit and was therefore not representative of the process at equilibrium).

Design

The system-level QI team was composed of critical care and emergency medicine physicians and nurses, quality specialists, clinical educators, clinical informatics personnel and CMS data abstractors. Facility-level QI teams were developed who shared our goal and with which we could exchange coordinate clinical education. Our system QI team met twice monthly, once to carry out Plan-Do-Study-Act (PDSA) cycles and once to feed back data, discuss progress and share ideas for interventions.

The CMS data abstraction team developed a data bank for each sampled patient that failed bundle compliance to support analysis of aggregate data by our QI team and allow access to detailed findings of the data abstractors not reported to CMS. This included their analysis of why each patient failed compliance. Information technology staff worked closely with our QI team to expedite design and implementation of CDSS systems in our Cerner Millenium (Cerner, Kansas City, Missouri, USA) electronic medical record. Cerner Discern was used to programme CDSS logic. We had previous success using CDSS to promote QI in the care of patients with sepsis.1 11 Although our work was organised around multiple PDSA cycles, we loosely conceptualised the first 6 months of the study as data gathering/planning and the second 6 months as the period in which our major CDSS interventions would occur.

Strategy

PDSA cycle 1 (October 2015–March 2016): during this phase of our study, we revised our sepsis standardised clinical practice and carried out system-wide education regarding the importance of sepsis bundle compliance. As we began to report SPC data to facility-level teams, it became clear that monthly CMS sepsis bundle compliance rates were not statistically interpretable at the facility level because of the small sample size stipulated by CMS (yielding a median of only seven patients per facility per month (range 0–16)). This resulted in dramatic swings in monthly facility-level sepsis compliance rates, strongly influenced by individual patient outcomes (see figure 1). This led to consternation of facility-level QI teams, since their efforts resulted in seemingly random changes in the outcome month by month.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Monthly sepsis bundle compliance rates at a single acute care hospital. Dashed lines indicated control limits. Note that a fall in compliance from 60% to 0% that occurred in April and May 2016 was statistically insignificant.

We analysed a sample (n=572) of patients to determine the most commonly missed bundle elements. In order of frequency, these included: repeat lactate assessment (30%), blood cultures (19%), antibiotics (16%), initial intravenous fluids (15%), initial lactate assessment (12%) and interventions related to shock (7%). Sixty-four per cent of failures were due to missing laboratory tests and 36% directly pertained to therapy (fluids and antibiotics).

Our data abstractors noted that this analysis didn’t capture the importance of documentation rules on compliance failure. In particular, it was sometimes difficult to precisely determine ‘time zero’—the onset of clinically recognised severe sepsis—given the vagaries of CMS specifications in relation to the limitations of our electronic medical record. Accurate determination of time zero is critical to determining compliance since all compliance goals needed to be accomplished within specified time frames after time zero. Time zero was defined as the earliest time in which all criteria for severe sepsis were documented; however, physicians might not document an associated infection until after administering treatment for sepsis. This would lead to compliance failure due to bundle elements being implemented before time zero. By CMS rules, if a physician’s discharge summary contained a notation such as ‘patient admitted with severe sepsis’, time zero for sepsis would be determined to be the time of admission, even if the patient did not have any clinical or laboratory findings of sepsis at that time. Other patients failed compliance because the duration of fluid administration was not specified in resuscitation orders. Much of our monthly discussions with facility-level QI teams focused on explaining CMS documentation requirements. By the end of this cycle, we had established our baseline compliance rate but had not improved it.

PDSA cycle 2 (April–June 2016): we developed CDSS interventions to optimise compliance with the most commonly missed bundle elements: repeat lactate assessment and blood cultures. We designed and implemented CDSS logic that would reflex-order a repeat plasma lactate level for any patient with a lactate result > 2 mmol/L. We also designed a CDSS to automatically order a lactate and prompt orders for blood cultures whenever broad-spectrum antibiotics were ordered for a patient with organ system dysfunction (a proxy for the diagnosis of sepsis). A chart audit of 30 patients identified by this alert logic showed a positive predictive value of 80% for the diagnosis of sepsis. A progressive rise in sepsis bundle compliance was observed (in May and June 2016) as these CDSSs were implemented.

PDSA cycle 3 (July–September 2016): our QI team decided to focus on improving compliance with therapeutic bundle elements (fluids and antibiotics), which we thought more likely to directly influence patient outcome than ordering laboratory tests. We failed to achieve consensus on how to improve fluid resuscitation compliance because clinicians in the system-level and facility-level teams considered the quantity of fluid required by CMS (30 mL/kg) unsafe in some patients, particularly those who were already fluid overloaded or who had cardiomyopathy. Therefore, we focused on timely and appropriate antibiotic administration. Chart audits showed that 80% of failures to give timely and appropriate antibiotics occurred in the ED. Forty per cent of antibiotic non-compliance was due to delays in ordering and 60% to delays in administration. These delays occurred despite documentation that serious infections were immediately recognised in the initial patient evaluation 75% of of the time. Chart audits revealed that antibiotics were often ordered as ‘routine’ priority for septic patients. After discussion, our pharmacy enacted a policy of automatically delivering all first-time intravenous antibiotic orders as ‘stats’. We also found that the administration of vancomycin before piperacillin/tazobactam repeatedly led to compliance failure because it delayed administration of piperacillin/tazobactam outside the time window required for compliance. We met with clinical and administrative personnel from our facility EDs and supported their efforts to improve early recognition and treatment of sepsis with data analysis such as that described above and educational materials.

However, before these interventions had been fully implemented, CMS changed the specifications for sepsis bundle compliance on 1 July 2016.12 Implementation of the new specifications was associated with an immediate decrement in our system bundle compliance rate, and our data abstractors reported many misclassification errors due to the changes. Under the new specifications, laboratory abnormalities due to chronic underlying conditions or medications were assumed to be related to sepsis unless a physician specifically documented otherwise. Failure of the physician to explicitly document that a patient’s elevated creatinine was due to end-stage renal failure or that an elevated international normalized ratio (INR) was due to warfarin administration could result in the patient being misdiagnosed with organ system failure due to sepsis. The most impactful change was that, prior to July 2016, only patients with hyperlactaemia or persistent hypotension after an initial intravenous fluid bolus were required to receive 30 mL/kg of intravenous fluids. The new specifications required that patients with even a single abnormal blood pressure (systolic <90 mm Hg or mean arterial pressure <65 mm Hg) receive 30 mL/kg of fluids. The rate of bundle compliance failure due to inadequate intravenous resuscitation subsequently increased by 33%—accounting for two-thirds of the decrement in overall CMS sepsis bundle compliance observed after the specifications were changed.

Lessons and limitations

Our QI effort was associated with a modest increase in CMS sepsis bundle compliance temporally associated with CDSS implementation designed to assist with ordering blood cultures and lactate levels. Later, we decided to focus on rapid recognition of sepsis and early administration of antibiotics in the ED, but CMS changed the definition of bundle compliance confounding our ability to determine whether this intervention was beneficial.

Our decision to use the data abstraction system required by CMS greatly enhanced our ability to collect and analyse data but exposed our main outcome variable to definitional change during the course of the study. This constituted a major methodological flaw. It is our hypothesis that the changes CMS made in July 2016 most likely caused the reduction in sepsis compliance rate we observed in the final 3 months of our study, based on temporal association and the observations of our data abstractors. We feel that the internal validity of our study was thereby weakened, but the generalisability remained strong since the specification changes affected all CMS  participating hospitals in the USA.

We made several observations regarding the use of sepsis bundle compliance as a process measure for QI. A good quality process measure should be proven related to important patient outcomes, relatively simple to measure and interpret, flexible enough to allow for reasonable clinical judgement/patient variation and stable enough to track progress over time.14–16 By the end of our study we concluded—in agreement with a recent editorial by Klompas and Rhee13—that the CMS sepsis compliance rate does not satisfy these criteria. Individual bundle elements are not proven to benefit patient outcomes with the possible exception of early antibiotic administration.17 18 The requirement to give 30 mL/kg of intravenous fluids regardless of the patient’s clinical fluid status may cause harm. The CMS measure is overcomplicated, requiring consideration of 84 variables per patient and entailing a multitude of onerous documentation requirements for nurses and doctors. Compliance failure is more often due to missing laboratory values or documentation than to suboptimal therapy. CMS did not establish benchmarks for performance and changed the measure less than a year after introducing it, confounding efforts to track progress of related QI projects. CMS seems cognizant of the difficulties hospitals are having with this mandate and has indicated that sepsis bundle compliance rate validation data from the first three-quarters of the mandate (October 2015–June 2016) will not impact the annual payment update for participating hospitals in 2018, nor be publicly reported.