Problem

Sepsis, a dysregulated host response to infection, is a serious health concern globally.1 In the USA alone, more than 750 000 individuals are afflicted annually,2 with a cost of over $20 billion per year.3 Many healthcare systems face the challenge of detecting sepsis early with high accuracy. The Cape Regional Medical Center (CRMC), a 242-bed acute care hospital located in Cape May Court House, New Jersey, is one such hospital that has aimed to improve sepsis-related patient outcomes through earlier recognition. This quality improvement initiative details CRMC’s collaboration with Dascena (Hayward, California, USA) to revise their sepsis management system through the implementation of a sepsis forecasting algorithm.

Previously, CRMC had no formal sepsis detection protocol for their emergency department (ED) patients. Patients in all other units were assessed for the presence of two or more systemic inflammatory response syndrome (SIRS) criteria during twice-daily nurse screenings.4 While the SIRS criteria were designed to detect sepsis development, in practice SIRS has demonstrated a low specificity, resulting in a high false alarm rate.5 Additionally, the infrequency of the sepsis screens at CRMC sometimes resulted in delay of treatment during the critical early intervention period.6 In particular, patients who rapidly developed sepsis over a few hours were not always identified in a timely manner with the twice-daily screenings. Consequently, CRMC implemented a sepsis prediction algodiagnostic (algorithmic diagnostic) developed by Dascena to aid in accurate sepsis evaluation.

The sepsis prediction algorithm implemented by CRMC is a machine learning program that provides risk scores that indicate the likelihood of sepsis onset. Through a pre-implementation and post-implementation analysis, we assessed improvements in sepsis-related in-hospital mortality rate, length of stay and 30-day readmission rate with the use of the machine learning algorithm in CRMC’s emergency and hospital patient populations.

Background

Severe sepsis is characterised by organ failure and carries a mortality rate of over 10%, and the escalation to septic shock presents with refractory hypotension and a mortality rate near 40%.7 Despite the high frequency and the poor associated outcomes of sepsis, the heterogeneity of both infection types and host responses makes the early and accurate diagnosis of sepsis difficult. Even consensus definitions of clinical sepsis presentation are difficult to achieve, as indicated by the recent proposed redefinitions of the stages of sepsis (sepsis −3).8 However, it has been shown in several studies that early recognition of sepsis, and compliance with sepsis treatment bundles, can lead to a reduction in patient mortality and length of stay.6 9 Electronic health record (EHR) data are becoming generally more widely available, and represent a rich if complex data source that can be applied to the prediction and detection of sepsis.

Prospective studies of EHR-tool usage in clinical settings have most often been rules-based,10 using predetermined score thresholds to rank patient sepsis risk,11 but demonstrate suboptimal sensitivity and specificity.12 Machine learning algorithms provide potential advantages over rules-based clinical decision support systems, allowing for site-specific adaptability, customisation and predictive capabilities. The machine learning algodiagnostic (MLA) used in this report has been validated in several retrospective studies,13–15 demonstrating a sensitivity and specificity of 0.93 and 0.91, respectively,13 and robustness to missing data.14 Experiments comparing this machine learning algorithm’s predictions with SIRS criteria have also shown that the algorithm’s predictions are more sensitive and specific four hours in advance, as well as at the time of sepsis onset.14 Other prior works applying machine learning algorithms to the EHR in the detection or prediction of sepsis include the retrospective studies of Henry et al 16 and Nachimuthu and Haug,17 and the pilot studies of Sawyer et al.18

Baseline measurement

We collected baseline measurements prior to the revision of CRMC’s sepsis management system. Previously, CRMC used a manual sepsis scoring system, tabulated for all non-ED patients twice per day. Nurses checked each patient every 12 hours, or on identification of a potential source of infection, to determine if they met at least two of the SIRS criteria.19 If a patient met two or more SIRS criteria at the time of assessment, a nurse ordered the nursing sepsis bundle, which comprised a lactate panel, microbiology cultures, a procalcitonin test, a complete blood count panel, a basic metabolic panel, a hepatic function panel and other applicable tests for organ dysfunction assessment. Furthermore, the physician assessed the patient for severe sepsis and accordingly administered all or a portion of the physician sepsis bundle. The physician bundle included the nursing bundle elements together with the following:

• fluid resuscitation (normal saline 30 mL/kg intravenously)

• broad spectrum antibiotics (meropenem or aztreonam, levofloxacin, vancomycin)

• invasive or non-invasive haemodynamic assessment

• haemodynamic support (norepinephrine, dopamine, phenylephrine, dobutamine, vasopressin).

Before the implementation of the machine learning algorithm, there was no formalised sepsis screening process for ED patients, but similar interventions were made for patients suspected of or diagnosed with severe sepsis or septic shock.

We extracted retrospective data from 1 November 2016 to 31 January 2017 from each of CRMC’s EHR systems: the Allscripts EHR (San Jose, California, USA) for ED patients and the Cerner Soarian EHR system (Kansas City, Missouri, USA) for intensive care unit (ICU), progressive care unit (PCU) and medical/surgical patients (2East and 4East units). These data represented all measurements recorded for the patient encounters in the baseline time frame.

The primary outcome assessed in this project was the sepsis-related in-hospital mortality rate at CRMC. In addition, the two secondary outcomes measured were average sepsis-related hospital length of stay and the sepsis-related 30-day readmission rate. Patient encounters were included in the sepsis-related outcome metrics if they met two or more SIRS criteria at some point during their stay. Further, only patients over the age of 18 in one of the participating units were included in the assessment. The baseline sepsis-related in-hospital mortality rate was 30/407 (7.37%). The average sepsis-related length of stay during the baseline data collection period was 3.35 days, and the baseline sepsis-related 30-day readmission rate was 188/407 (46.19%).

Design

This project was designed as a prospective quality improvement study. We assessed baseline measurements (pre-implementation) in addition to post-implementation metrics. The first post-implementation phase spanned from 7 February 2017 to 14 April 2017; we additionally analysed post-implementation steady-state measures during the period 20 April 2017 to 20 May 2017, beginning 1 month after the completion of all alert modifications. These data were collected through the EHR systems during the quality improvement project. To oversee implementation of the machine learning algorithm, we formed a quality improvement team overseen by Dr Andrea McCoy, Chief Medical Officer of CRMC. This team comprised clinicians and administrators from CRMC, as well as members of Dascena’s onboarding staff; this team met regularly throughout the post-implementation phase.

The post-implementation phase involved real-time data collection with the use of the MLA. We collected data for all patients in the included units who were over the age of 18. By analysing trends in the collected measurements and their degree of similarity to prior sepsis cases, the algorithm was designed to accurately identify which patients were at the greatest risk of developing severe sepsis. The algorithm used vital sign measurements and, optionally, lab results when present to generate severe sepsis predictions. One of each vital measurement (systolic blood pressure, diastolic blood pressure, heart rate, temperature, respiratory rate and blood oxygen saturation level) was required to generate a prediction score. If a vital measurement was not provided during a given hour, forward-filling imputation (estimation of missing data based on nearby values) was used to gap-fill the missing data.

The algorithm generated a sepsis risk score between 0 and 100 for each patient. Initially, if a patient’s score met or exceeded the threshold of 80, a healthcare provider was called and informed of possible severe sepsis. The CRMC standard of care process, unchanged from the baseline period, was then executed to assess and treat the patient.

Further, to estimate the potential effectiveness of the machine learning algorithm on the CRMC patient population, a side-by-side performance comparison was conducted on an EHR-extracted CRMC data set of 1665 retrospective encounters collected between 1 January 2017 and 30 April 2017. The machine learning algorithm demonstrated significantly higher sensitivity and specificity than the Modified Early Warning Score (MEWS),20 the SIRS criteria, the Sequential (Sepsis-Related) Organ Failure Assessment (SOFA) score21 and the quick SOFA (qSOFA) score22 for a gold standard of severe sepsis, as defined by the patient meeting two or more SIRS criteria and having two or more organ dysfunction lab results (table 1). We additionally assessed the algorithm’s ability to detect sepsis as defined by the Sepsis-3 consensus definition.7 The Sepsis-3 gold standard was defined by suspicion of infection, identified by an abnormal white blood cell count alongside an order of antibiotics within a 24-hour period, and organ dysfunction, identified by an increase in SOFA score of ≥2 points. Sepsis-3 onset was operationalised as the first time both criteria were met within the same hour.

We selected severe sepsis as a gold standard for this quality improvement initiative because CRMC uses this definition in diagnosing patients and initiating the sepsis bundle. Despite the 2016 publication of the Sepsis-3 definition, the majority of US hospitals use older definitions of sepsis in diagnosing patients. This is due to several factors, including the relatively recent publication of the Sepsis-3 definition by Singer et al,7 the use of older definitions of severe sepsis and septic shock by the Centers for Medicare and Medicaid Services in their  SEP-1 (sepsis core measure bundle) guidelines, and conflicting opinions about the diagnostic utility of the new definition.23 Additionally, the Sepsis-3 definition is quite similar to the chosen gold standard of severe sepsis, as both definitions require evidence of systemic infection alongside organ dysfunction. We have demonstrated success in the past with implementing the Sepsis-3 definition as a gold standard in retrospective studies,13 and the algorithm’s strong performance on CRMC retrospective data under the Sepsis-3 gold standard suggests that the algorithm would likely perform well at CRMC under an updated gold standard, as well.

The MLA was applied to patients throughout the course of their stay (ie, in the ED and/or the hospital). In addition to the use of the algorithm’s prediction scores, CRMC nurses continued tabulation of SIRS criteria every 12 hours for patients in non-ED units, continuing standard procedure as prior to the implementation of the machine learning algorithm; thus, the use of the algorithm did not pose any additional risk to patients. Furthermore, to customise the alerting system to local practice, the quality improvement team regularly incorporated feedback from clinical leadership and end users through the Plan-Do-Study-Act (PDSA) cycles. Modifications resulting from this feedback are provided in the Strategy section. Approval by the institutional review board and informed patient consent were not required in this project as it was a quality improvement initiative at CRMC and did not constitute human subjects research.

Strategy

We conducted three PDSA cycles to evaluate processes and incorporate clinical feedback during this quality improvement initiative.

Lessons and limitations

This quality improvement initiative was implemented in a community hospital, which typically has cost and mortality outcomes distinct from large academic medical centres.25 26 Often it can be difficult to effectively translate advances in clinical research to a community hospital setting.27 In addition to the successes of this project, there are several limitations related to the generalisability and design of this quality improvement initiative. Because this work was conducted in a 242-bed community hospital in southern New Jersey, it does not provide evidence that the same improvements would necessarily be achieved at a larger medical system or in a different geographical region. Furthermore, this initiative did not have a randomised design. Thus, we cannot draw conclusions about cause-and-effect relationships between the updated sepsis management system and readmissions, length of stay or mortality. Rather, we have documented the steps to an effective implementation of the machine learning algorithm, which was likely associated with such improvements.

Further, it is important to note that we did not analyse this patient population based on International Classification of Diseases (ICD) coding for sepsis, severe sepsis or septic shock. Rather, we based our analysis on meeting two or more SIRS criteria during the patient stay. We chose this method both because ICD code documentation is known to be an inaccurate indicator of clinical diagnosis28 and because of the predictive design of the machine learning algorithm. The algorithm may have enabled earlier intervention and averted development of sepsis or escalation to severe sepsis or septic shock. In addition, due to this patient selection, the absolute values of the outcome metrics (ie, mortality rate, length of stay and readmission rate) may not be directly comparable to other published studies and benchmarks, which often segment populations based on ICD coding; thus, we have chosen to focus on the relative, pre-implementationand post-implementation differences in this report.

Possible confounding factors relate to the personnel involved in sepsis care and management, the frequency of SIRS criteria tabulation, and variation in the baseline period and quality improvement period. Variations of clinical staff in the five units during the pre-implementation and post-implementation time periods may have influenced outcomes such as bundle compliance and sepsis-related length of stay through the level of care individual clinicians delivered. Increased bundle compliance during the post-implementation period, due to either increased awareness of sepsis caused by the algorithm or to other confounding factors, may have resulted in the improved outcomes noted in this study. However, increased bundle compliance may also represent evidence that the alerting system is appropriately drawing attention to patients with sepsis and prompting timely care; the cause of increased compliance cannot be determined by this quality improvement initiative alone. Furthermore, the use of the machine learning algorithm may have increased clinician awareness and unknowingly prompted clinicians to check SIRS criteria more frequently than every 12 hours. Finally, differences in contextual elements of the time periods could have impacted the results. If the pre-implementation period, for example, was a particularly busy time period for the hospital relative to available staff, a greater number of sepsis cases could have been overlooked.