Article Text
Abstract
Problem Antibiotic administration to inpatients developing sepsis in general hospital wards was frequently delayed. We aimed to reproduce improvements in sepsis management reported in other settings.
Context Ninewells Hospital, an 860-bed teaching hospital with quality improvement (QI) experience, in Scotland, UK. The intervention wards were 22 medical, surgical and orthopaedic inpatient wards.
Design A multifaceted intervention, informed by baseline process data and questionnaires and interviews with junior doctors, evaluated using segmented regression analysis of interrupted time series (ITS) data.
Measures for improvement Primary outcome measure: antibiotic administration within 4 hours of sepsis onset. Secondary measures: antibiotics within 8 hours; mean and median time to antibiotics; medical review within 30 min for patients with a standardised early warning system score ≥4; blood cultures taken before antibiotic administration; blood lactate level measured.
Strategies for change The intervention included printed and electronic clinical guidance, educational clinical team meetings including baseline performance data, audit and monthly feedback on performance.
Effects of change Performance against all study outcome measures improved postintervention but differences were small and ITS analysis did not attribute the observed changes to the intervention.
Lessons learnt Rigorous analysis of this carefully designed improvement intervention could not confirm significant effects. Statistical analysis of many such studies is inadequate, and there is insufficient reporting of negative studies. In light of recent evidence, involving senior clinical team members in verbal feedback and action planning may have made the intervention more effective. Our focus on rigorous intervention design and evaluation was at the expense of iterative refinement, which likely reduced the effect. This highlights the necessary, but challenging, requirement to invest in all three components for effective QI.
- Healthcare quality improvement
- Audit and feedback
- Hospital medicine
- Performance measures
Statistics from Altmetric.com
Background
Outline of problem
Sepsis is associated with significant morbidity, mortality and healthcare costs. Timely effective therapy can improve outcomes for patients with severe sepsis on admission to hospital and in the intensive care unit (ICU),1 ,2 and the evidence has been summarised in guidelines for severe sepsis management3 with key elements condensed into two care bundles.4 Implementing the bundles has been associated with improved clinical outcomes for patients admitted to hospital with severe sepsis.5 It is not clear whether similar improvements are achievable for patients who develop sepsis while in general wards.
In pilot work, we found that only 20/35 (57%) of patients with sepsis received antibiotics within 4 h of onset.6 The questions for our improvement project were:
-
What are the deficiencies in the management of patients who develop sepsis while inpatients in general wards in our hospital?
-
What barriers and facilitators exist that may impact on good sepsis management and on implementation of an intervention to improve care?
-
Will a multifaceted intervention lead to measureable improvements in the care of these patients?
Context
Ninewells Hospital is an 860-bedded, teaching hospital serving a population of about 400 000, with most acute services, including accident and emergency, acute medical and surgical admissions units, intensive care, renal dialysis and cardiac catheterisation, available on one site. The intervention wards were the 22 general medical, general surgical and orthopaedic wards, excluding acute admissions units. There is an established history of quality improvement (QI) activity in Ninewells Hospital as part of the Scottish Patient Safety Programme.7 There is a mature antimicrobial stewardship programme, and an Infectious Diseases service. There are large numbers of doctors in training, rotating through multiple clinical units and contributing to on-call rotas and out-of-hours cover. A Hospital at Night team led by senior nurse practitioners coordinates and aids the delivery of care overnight to hospital inpatients. There is no rapid response or medical outreach team, but there is a cardiac arrest team and a doctor on call for each specialty (within normal working hours only for some smaller or subspecialties).
Assessment of problems
Case identification
The study case definition was a patient with sepsis occurring ≥24 h after admission to hospital, either as a first episode or following a period of ≥24 h without meeting the systemic inflammatory response syndrome (SIRS) criteria (ie, having two or more of the following: heart rate ≥90 beats/minute; respiratory rate ≥20 breaths/minute; temperature <36°C or ≥38°C; white blood cell count <4 cells/mm or ≥12 cells/mm).2 ,8 Sepsis was defined as SIRS due to suspected or proven infection.8 In preliminary work, we found that screening patients who had blood cultures taken in our target wards had good performance characteristics for identification of patients with sepsis (sensitivity 12/16, 75% (95% CI 54 to 95%); specificity 55/80, 69% (95% CI 59 to 79%); positive predictive value 12/37, 32% (95% CI 17 to 48%); negative predictive value 55/59, 93% (95% CI 87 to 100%)). Other methods to identify potential cases for screening, for example, using SIRS or standardised early warning system (SEWS) scores, were considered but would have been dependent on many people regularly reporting potential cases to us, so could not be reliably implemented in this project.
Measures for improvement
We identified candidate measures for improvement and refined and prioritised these using Failure Modes Effects Analysis9 with a multidisciplinary team.6 We also showed that extraction of the measures from medical notes was a reliable way to obtain the required information.6 Our primary outcome measure was the proportion of septic episodes with antibiotic administration within 4 h of onset.6 ,10 Secondary outcome measures were the proportions with antibiotics within 8 h of sepsis onset,11 medical review within 30 min if the SEWS score was ≥4,6 ,12 blood cultures taken before antibiotics were administered,3 and blood lactate level measured,3 ,4 and the mean and median time to antibiotics. SEWS score calculation, based on observation of vital signs, and responding to high scores is part of routine nursing care across Ninewells Hospital.
Quantifying problems in sepsis management
We collected baseline data, including concordance with the study measures for improvement, by screening patients who had blood cultures taken in target wards for 6 months from September 2008 to February 2009. We excluded patients <18 years old, screened in previous 48 h, and immunosuppressed due to chemotherapy or organ transplant. We screened all eligible patients, or took a random sample using an online program13 if there were too many patients to screen in the time available. The time of sepsis onset was defined as the time on the observations chart of the deterioration (meeting SIRS criteria as a minimum) related to the septic episode.
Identifying barriers and facilitators: questionnaires and interviews with doctors
We developed and pilot-tested a questionnaire among trainee doctors to quantify knowledge, opinions and attitudes relating to sepsis recognition and management. Agreement with statements was measured on a 7-point Likert scale, from ‘strongly disagree’ (=1) to ‘strongly agree’ (=7). Questionnaires were distributed by email and at regular teaching sessions. Respondents could remain anonymous but were asked to consent for interview. We invited a purposive sample of consenting respondents, reflecting a range of grades, specialties, knowledge, opinions and attitudes to interview. The aim of the interviews was to inform the intervention by elucidating potential barriers to, and facilitators of, the timely recognition and management of sepsis. Interviews were digitally recorded, transcribed verbatim and analysed by thematic content analysis using a framework approach to index and summarise the findings.
Results of assessment
Magnitude of problems in sepsis management
Prior to the intervention, we screened 999/1341 (74%) total eligible blood culture episodes, of which 291/999 (29%) met the study case definition (table 1). Time to antibiotic administration could only be assessed for episodes in which antibiotic therapy was initiated or changed (241/291 episodes). The mean time between sepsis onset and receipt of antibiotics was 11.0 h (95% CI 9.3 to 12.7 h), with a median of 6.0 h (IQR 2.5 to 13.3 h) (table 2). Only 91/241 (38%, 95% CI 32 to 44%) patients received antibiotics within 4 h of sepsis onset. Analysis of the individual steps in the process revealed that the longest delay was from first recorded medical review to antibiotic prescription (mean 7.2 h, median 2.5 h), which was over twice as long as the delay between sepsis onset and medical review (mean 3.0 h, median 1.0 h). There was little delay between recorded antibiotic prescription and administration times (mean 0.9 h, median 0.0 h). Performance against secondary outcome measures ranged from 11% (95% CI 7 to 14%) for lactate level measured to 80% (95% CI 74 to 85%) for blood cultures taken before antibiotics were administered (table 2).
The long delay between medical review and antibiotic prescription highlighted that decision making among junior doctors contributed more to delays than barriers to administration, and supported our intention to direct the intervention primarily at them. It also highlighted the need to be directive with target timeframes in any clinical guidance or protocol, and to encourage doctors to prescribe on clinical grounds, without waiting for confirmatory tests or senior review.
In each individual ward, an inpatient developing a new septic episode was an infrequent and unpredictable occurrence. The median number of cases per month per ward was 2 (range 0–6) across general medical wards, 4 (0–10) across general surgical wards and 1 (0–3) across orthopaedic wards. This sporadic nature of sepsis onset meant that testing of the intervention ward by ward in real time was not a viable option. It also meant that performance feedback had to be at directorate, rather than individual ward, level for there to be enough cases at each monthly time point.
Questionnaires and interviews
Totally, 147/423 (35%) questionnaires were returned. Knowledge-based questions around sepsis definitions were answered poorly with only 16–56% of respondents correct in each of five questions. The lack of knowledge of definitions mandated their inclusion in intervention materials. Knowledge was no different between doctors of different grades.
Reported opinions and attitudes were positive, with most doctors agreeing that guidelines improve patient care, and that a local sepsis guideline would be useful. Agreement with statements on doctors’ confidence in their ability to recognise and manage sepsis was strong (median responses=6), but was not supported by knowledge. Doctors who were confident that they could determine when a patient has severe sepsis were no more likely than those who were not confident to correctly identify which clinical features among a list of ten options would or would not indicate severe sepsis. There were 13/97 (13%) confident doctors who scored 10 out of 10 vs 4/27 (15%) less confident doctors, χ2 test p=0.85).
A purposive sample of 10 (from 40) respondents took part in interviews. Identified barriers and facilitators are summarised in online supplementary appendix table 1. Some facilitators could be useful in designing and implementing the intervention, such as ensuring Hospital at Night nurse practitioners are involved. Some barriers were predictable, such as lack of knowledge and/or confidence, and incidences of poor communication. Others were less expected, such as the strength of the antibiotic restriction message from interventions to reduce Clostridium difficile infection, with the result that some doctors will not start antibiotic treatment without laboratory or radiological evidence of infection, even when a patient has clinical signs of sepsis. Some reported barriers were beyond the scope of this intervention, such as those relating to recent changes in doctors’ working patterns.
The findings indicated enthusiasm for an intervention to improve sepsis management, but uncovered misplaced confidence among doctors about their knowledge. Providing clinical guidance including definitions may deal with some knowledge deficits, as long as overconfidence does not prevent doctors consulting the guidance. We aimed to highlight knowledge gaps in the intervention to encourage consultation of clinical guidance even when doctors think they know what to do.
Strategy for change
We developed and implemented a persuasive intervention consisting of clinical guidance, education, audit and feedback, following MRC guidance for the development of complex interventions14 and the IHI Model for Improvement9 as far as possible. To develop the clinical guidance, we modified published resources4 ,9 ,15 to address local problems identified in exploratory work. We piloted and tested several versions of the guidance based on recent cases or case scenarios, with frontline staff using the Plan, Do, Study, Act (PDSA)9 method, and made required changes before implementing the final version. The PDSA cycles included different staff members and clinical areas to develop generic, rather than ward-specific, guidance. We were unable to roll out the intervention one ward at a time using real-time data due to the sporadic and infrequent occurrence of sepsis within each individual ward, as described above. That approach would have necessitated many months testing in each ward before each stage of roll-out.
Education sessions involved presentation at regular teaching or department meetings, or sessions arranged for the project, from June to October 2009. This time interval spanned the early August date when rotating junior doctors change specialty, so it could potentially capture all those who would be working in intervention wards during the feedback period. Although voluntary, education sessions were attended by over 300 clinical members of staff. Content focused on the importance of early recognition and treatment of sepsis, knowledge deficits identified in questionnaires, and the areas for improvement revealed by local data. We then introduced the new clinical guidance and plans for performance feedback. In the time allocated for discussion, the most common issue raised was the perceived conflict between antibiotic restriction to prevent C difficile infection and timely antibiotic administration in sepsis.
After distributing the guidance and delivering the education sessions, we restarted data collection and fed back performance against the primary outcome measure each month by directorate. We disseminated annotated run charts in poster format to each ward, and by email to all relevant nursing and medical staff. The text accompanying the run charts was modified each month to reflect recent performance and any feedback from the clinical teams. Monthly performance was discussed opportunistically with available clinical staff by a member of the study team (CM) during data collection and posting of the monthly run chart posters. In email and verbal feedback, clinical staff members were positive about the aims and patient benefits of the project, and reported positively on the utility of the clinical guidance. Reasons given for poor performance in particular months included staff absence and shortages, perceived conflict between antibiotic restriction and promotion, and too many improvement interventions and/or clinical guidelines being introduced at once. Some of these reported issues were not amenable to changes within our intervention, but we strongly and repeatedly emphasised that there is no actual conflict between interventions to reduce inappropriate broad spectrum antibiotic use and this intervention to promote prompt antibiotic administration to patients with sepsis. We highlighted that both types of intervention were supported by the local Antimicrobial Management Group, and our sepsis guidance directed users to prescribe antibiotics according to local policy. During the feedback period, we were unable to investigate whether there were additional reasons for poor performance.
We screened 1354 (95%) of 1431 episodes, identified 346 cases of sepsis (table 1), and could measure time to antibiotic in 297 cases (table 2), between October 2009 and March 2010. Demographic and clinical characteristics were similar to preintervention (table 1). The lower occurrence of vascular catheter-associated infections in the postintervention period may just be random variation, or may be due to the introduction of central venous catheter bundles close to the end of our baseline data collection period.
Effects of change
We used segmented regression analysis of interrupted time series (ITS) data to determine effects of the intervention on the primary outcome measure and any secondary measures for which before-and-after analysis (table 2) and/or run charts suggested a possible effect. In before-and-after analysis, performance against all study outcome measures improved postintervention, but differences were small and most were not statistically significant (table 2). However, there was a statistically significant 9% increase in the primary outcome measure of the proportion receiving antibiotics within 4 h (p=0.04), and a 14% increase in the proportion of patients who had a lactate level measured (p<0.001). We plotted run charts of performance by study week for each outcome measure (as distinct from the feedback run charts of monthly data for the primary measure only) to identify non-random (or ‘special cause’) variation using standard rules.16
The principal observation on visual inspection of run charts for the primary measure (figure 1) and for the proportion with a lactate level measured (see online supplementary appendix figure 1) was the marked variation in preintervention and postintervention periods. All the run charts met at least one rule for non-random variation, but there was no clear association with the timing of the intervention. Secondary outcomes measures which we evaluated by ITS analysis were the median time to antibiotics and the proportion of episodes where lactate was measured. The ITS models were adapted from those described by Wagner et al17 and were weighted to adjust for the denominator in each study week. We tested each model for autocorrelation using the Durbin–Watson statistic18 and the Breusch–Godfrey test,19 corrected significant autocorrelation by adding lag terms (and by log transformation of the lactate model), then used Akaike's Information Criterion (AIC)20 to select the best fitting models. There were no statistically significant intervention effects (p>0.2, see online supplementary appendix table 2).
Lessons and messages
Learning from this work
Among hospital inpatients, sepsis is not recognised and treated promptly. We developed an improvement intervention applying MRC and IHI guidance, and informed by extensive exploratory work. There was an improvement in performance against the primary outcome measure, and some small improvements among secondary outcome measures, but on rigorous statistical analysis we could not attribute these changes to the intervention. There is insufficient reporting of such negative intervention studies in the literature, giving the impression that QI interventions will inevitably be effective. The publication bias in favour of intervention studies with positive findings appears to be particularly evident in QI work. Many ‘positive’ QI studies are not analysed robustly enough to conclude definitively that observed changes were attributable to the intervention,21 with reporting limited to uncontrolled before-and-after. Equivalent analysis of our study findings would have labelled the 9% change in our primary outcome measure as an intervention effect. A recent multifaceted intervention implemented in eleven ICUs to improve adherence to ventilator-associated pneumonia prevention guidelines, reported an overall 8% (95% CI 2.7 to 13.3%) increase in guideline adherence from 50.7% to 58.7% over a 2-year period, despite repeated intervention exposure and reported staff awareness of the recommendations.22 The article and accompanying editorial acknowledge the modest change but report it as an intervention effect.22 ,23 Welcome efforts to improve the scientific rigour of QI reporting are being made,21 ,24 but are not yet universally applied.
Our study was funded through a Clinical Academic Fellowship, which imposed some limitations for an improvement study. First, the expectation for such a fellowship is for ‘research’ evaluation, in this case segmented regression analysis of ITS data which requires an intervention to occur at a single time point rather than allowing iterative adaptation of the intervention in response to effect. Further related limitations arose from the requirement to complete the fellowship within a fixed timeframe with the majority of the work carried out by the research fellow (CM). This requirement meant there was no time available to investigate potential reasons for limited effect, or to adapt the intervention (even if the method of evaluation had allowed for this), during the feedback period. If resource had allowed, investigating the effect of contextual factors, focusing on clinical areas and time periods of highest and lowest performance, may have provided useful insights. Reinforcing the educational components during the feedback period may also have been beneficial. Other potential biases were due to the study case identification method, which could miss patients with the worst care, and the self-selection of questionnaire respondents and volunteers for interview, either of which might have resulted in an underestimation of the magnitude of identified problems.
Our intervention was modelled on a previous QI intervention in our hospital, which significantly improved timely administration of antibiotics to patients with community-acquired pneumonia.25 That intervention was focused on a single clinical team (Acute Medical Admissions), whereas, our sepsis intervention was spread across several wards and the feedback was by Directorate, so did not focus on the performance of specific clinical teams. Engaging with senior clinicians across multiple clinical units and teams simultaneously proved challenging. Although the intervention was presented, discussed and agreed at senior level across the three Directorates, the consultants were not required to take an active role. A recently reported investigation into determinants of antimicrobial prescribing behaviour emphasises that local senior practice within a clinical team influences behaviour much more than policy or protocols, and that organisational approaches to policy implementation focus too much on junior staff.26
Following Cabana et al’s publication of a framework for overcoming barriers to physicians’ adherence to guidelines,27 there have been a number of publications highlighting the need to use social and behavioural approaches to understand and influence prescribing behaviour, including a review of the determinants of hospital antibiotic use and possible improvement strategies.28 This literature highlights the need to investigate stakeholder barriers at the local level to maximise the impact of improvement interventions. We used this type of approach, where barriers and facilitators are explored and addressed, or used where possible, and we applied potentially effective improvement strategies,28 but thorough application of some QI implementation methods was limited as above. Some of the barriers to our intervention would require service redesign and changes at the organisational level to overcome.
Moving forward
The problem of delays in the recognition and treatment of inpatients with sepsis will be common among other acute hospitals, but data on this population are lacking. The use of blood culture sampling showed potential in facilitating case identification, but should be tested in other acute hospitals.
Since we carried out this work, three meta-analyses have synthesised the evidence from reports of interventions including audit and feedback, using behavioural theory, to identify intervention components that may enhance effectiveness. Such components included setting a target or behavioural goal29–31; a supervisor or colleague delivering feedback more than once, in verbal and written formats31; providing specific, frequent and written suggestions for improvement30; and an action plan.29 ,31 Two of these potentially key components were missing from our intervention. First, we did not deliver verbal feedback to all staff. Second, we did not involve senior clinical team members in the delivery of feedback or in action planning when performance was below the agreed target. With hindsight, building review of sepsis cases by the responsible clinical teams with reflection on gaps in the quality of care into the intervention may have made it more effective.32
Valuable learning from our study has been used to inform various aspects of a Scotland-wide sepsis improvement collaborative33 as part of the Scottish Patient Safety Programme7 which aims to reduce mortality among hospital inpatients by 10% by 2014. The sepsis arm of the collaborative involves central reporting by NHS Boards of performance against process measures that are collected by clinical teams. In NHS, Tayside, the published Sepsis Six15 bundle is now used in admitting units across two sites with spread to inpatient wards in progress. The design of the tool which incorporates the bundle underwent much iteration, tailoring and testing before the final version to be implemented was agreed. The frequent occurrence of sepsis in admitting units allowed for real-time testing of the tool using SIRS and SEWS scores to identify potential cases with sepsis. Two clinical areas with very specific needs, the emergency department and obstetrics, have adapted the tool to suit their local context. Consultant engagement and leadership have been central to adoption of the bundle in pilot units and in spread to other units. Embedding process measurement and reflection on the data into routine practice is an on-going challenge, but has been aided by shared learning between units and is critical to sustained effectiveness. This intervention is being spread to general wards using annotated run charts of data collected by clinical teams as the main driver for change.32
In conclusion, we implemented a carefully designed multifaceted intervention targeting sepsis management, but rigorous analysis did not confirm any significant intervention impact. The lack of iterative refinement of the intervention likely contributed to the lack of effect. Learning from this study has helped inform the work of the Scottish Patient Safety Programme Sepsis Collaborative.
Acknowledgments
Many thanks to Ross Martin and John Wallace in Medical Microbiology in NHS, Tayside, for providing lists of patients who had blood cultures taken, and to the very many staff in NHS, Tayside, who assisted and supported intervention implementation and provided useful feedback. Many thanks also to the doctors who completed the questionnaire and took part in interviews.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Footnotes
-
Contributors CM, BG, JE and PGD designed the overall study, and all contributed to data analysis and interpretation. CM carried out the majority of the work for the project during a PhD fellowship. JP contributed to specific aspects of design and collected data. PTD gave statistical instruction and support for the interrupted time series analysis including interpretation of the results. DN contributed to the design of specific study components and the logistics of intervention implementation, and is taking relevant aspects forward with the Scottish Patient Safety Programme, along with CM. CM wrote the first draft of the article. All authors critically revised drafts and approved the final version for publication.
-
Funding The work was carried out during a Scottish Government Chief Scientist Office Clinical Academic Training Fellowship (reference number CAF/07/06) awarded to CM and supervised by PGD, BG and JE. The fellowship award included salaries for CM and JP and research and training costs for CM. The other authors contributed as part of their routine work and received no extra funding. The funders and sponsors had no direct involvement in the design or conduct of the work, or the preparation of the manuscript.
-
Competing interests All authors have completed the Unified Competing Interests form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: CM and JP received salary and research costs from the Scottish Government Chief Scientist Office for the work submitted, other authors received no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.
-
Ethics approval The clinical aspects of the study were approved by Tayside Committee on Medical Research Ethics A (REC reference number: 08/S1401/34). Informed patient consent was not required as we used observational clinical data, extracted from records created for routine care, and anonymised for analysis. The Caldicott guardian for inpatients gave permission to access this information. Tayside Committee on Medical Research Ethics B approved the questionnaire and interview study (REC reference number 09/S1402/10). The study Sponsor was the University of Dundee Research and Innovation Services.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Previous presentation Preliminary findings of this study were presented at Federation of Infection Societies annual scientific meeting: Infection 2009, 12th November 2009, Birmingham (FP9), and at the 22nd European Congress of Clinical Microbiology and Infectious Diseases, 31st March 2012, London (O171), with publication of the abstract in conference proceedings (Clin Microb Infect 2012; 18: 23).