Article Text
PDF
Abstract
Background Surgical Safety Checklists (SSC) have been implemented widely across 132 countries since 2008. Yet, despite associated reductions in postoperative complications and death rates, implementation of checklists in surgery remains a challenge. The aim of this study was to assess the impact of a patient safety programme over time on SSC use and incidence of clinical errors.
Design A prospective longitudinal design over three time points and a retrospective secondary analysis of clinical incident data was undertaken.
Methods We implemented a patient safety programme over 4 weeks to improve surgical teams’ use of the SSC. We undertook structured observations to assess surgical teams’ checklist use before and after programme implementation and conducted a retrospective audit of clinical incident data 12 months before and 12 months following implementation of the programme.
Results There were significant improvements in the observed use of the SSC across all phases, particularly in sign-out where completion rates ranged from 79.3% to 94.5% (p<0.0001) following programme implementation. Across clinical incident audit periods, 33 019 surgical procedures were performed. Based on a subsample of 64 cases, clinical incidents occurred in 22/16 264 (0.13%) before implementation and 42/16 755 (0.25%) cases after implementation. The most predominant incident after programme implementation was inadequate tissue specimen labelling (23/42, 54.8%). Clinical incidents resulted in minimal or no harm to the patient.
Conclusions The benefit in using a surgical checklist lies in the potential to enhance team communications and the promotion of a team culture in which safety is the priority.
- checklists
- compliance
- never events
- patient safety
- surgery
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:©http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Estimates indicate that over 234 million patients undergo surgical procedures worldwide every year.1 Adverse events in surgery were reported to occur in up to 14% of patients,2 while mortality rates were reported to be between 1% and 4%.3 4 In an effort to improve morbidity and mortality rates associated with surgery, the WHO Surgical Safety Checklist (SSC) was developed in 2008. In the decade since its introduction, implementation of the WHO SSC has become increasingly widespread and it is now used in over 132 countries worldwide.5 Many studies,6–10 including several meta-analyses,11–13 have reported reductions in associated postoperative complications since the introduction of the SSC. In more proximal outcomes, Haynes et al 6 reported improvements resulting in increases in correct surgical site marking from 56% to 83%, and general improvements in safety processes from 34% to 57%.
Problem description
Despite that checklist use has become more widespread, results have not been so impressive and have been mixed or inconclusive.14 15 Some studies have shown that although 100% compliance with the SSC was documented, in reality, checklists were completed less than 10% of the time when operating room (OR) staff were observed.16 Checklists may contribute to improved patient safety outcomes; however, they are often implemented as part of multicomponent quality improvement initatives.14 Additionally, there is mounting evidence identifying contextual and organisational challenges in relation to ease of checklist adoption into workflow patterns,17 18 and their true impact on patient safety.16 19 20 These aspects require high commitment to practice change at all levels, particularly relative to physician involvement and organisational support.21 Unsurprisingly, many healthcare organisations achieve only partial implementation success when they are initiated.22 23
To date, few studies have evaluated the observed use of the SSC longitudinally following the implementation of a patient safety programme. The objectives of this study were to: (1) assess checklist item completeness rates before and after implementation of a patient safety programme designed to improve surgical teams’ participation in safety checks using the SSC; and (2) describe the impact of the intervention on the prevalence of clinical errors.
Methods
Study overview
We conducted a prospective longitudinal study using structured observations and undertook a retrospective secondary analysis of electronic database audit. The primary outcomes included observed SSC participation and completion rates, while the secondary outcome was period prevalence of clinical errors. Following baseline measures, we implemented a patient safety programme over 4 weeks and collected observational measures at 3 and 12 months’ postimplementation. We undertook the audit of electronic secondary outcome data retrospectively, 12 months before and 12 months after programme implementation. Online supplementary file 1 details the data collection time points, method of collection and measures across implementation phases.
Supplemental material
Context
The study setting was a 750-bed university hospital in Queensland, Australia. The facility had 16 commissioned operating rooms, catering for all specialties except transplantation, and performs approximately 16 000 surgeries each year. Surgical teams consisted of anaesthetists and surgeons and their registrars, and nurses working in circulating/instrument and anaesthetic roles. As the programme was implemented department-wide, all staff (n=150) from surgery, anaesthetics and nursing were invited to participate in the preimplementation and postimplementation using structured observations. The clinical incident data were drawn from a consecutive sample of patients who underwent elective, emergent and emergency surgeries at the study site over a 24-month period between October 2014–2015 and November 2015–2016. The Pass The Baton (PTB) programme was delivered in November 2015.
Intervention
The ultimate goal of the WHO SSC is to ensure that surgical teams consistently follow several critical safety steps, thus minimise the most common and avoidable risks endangering the well-being of patients.1 The original 19-item three SSC checkpoints include: sign-in (preinduction), time-out (surgical pause prior to commencement of the procedure) and sign-out (checks performed during or just after skin closure). The WHO recommends that a designated coordinator (often the circulating nurse) review verbally each checkpoint with the appropriate team member(s) to confirm that all key actions for that phase of care have been performed.1 Having a dedicated person lead the checklist process ensures that safety steps are not missed or rushed in an effort to progress with the next phase of the procedure. However, while this person is often the circulating nurse, any member of the surgical team can coordinate the checklist.1
At the study hospital, the WHO SSC was adapted to meet the needs of the local context and included 20 items across the sign-in, time-out and sign-out checkpoints. Although mandated throughout the OR department, the SSC was haphazardly used and inconsistently completed, with varying levels of staff participation.24 The patient safety programme, PTB, was implemented department-wide, across all surgical specialties to increase item coverage and team participation across the three checkpoints. Development of the PTB programme was informed by a series of studies15 18 24 conducted at the study site. The most significant barriers to using the SSC as recommended were workflow, limited knowledge about timing/content of checks, a lack of clinical leadership and dissonant attitudes.18 24
Prior to implementing the PTB programme, two team members (BMG and JL) had initial discussions to identify other interested staff members. BMG has had a long-standing association as a researcher in the OR department, and is well known to many of the nursing and medical staff. JL is a senior nurse, working as a perioperative educator. In this role, she has implemented other types of practice improvement initiatives. As a staff member, JL is highly regarded, and considered an opinion leader. JL identified a mix of nurses working across all perioperative areas to champion PTB programme. Both BMG and JL approached several physicians in surgery and anaesthetics to garner support. The implementation team consisted of anaesthetic/circulating/ instrument nurses, a senior physician and two researchers. The clinicians on the team were experts in their respective fields and were highly respected influential leaders within the department. The programme was coproduced with staff across surgery and anaesthetics. A logic model (ie, a diagram describing structures in place to deliver the programme, the intended activities and intended short, medium and long-term outcomes)25 was developed a priori by the implementation team. The PTB programme, delivered over 4 weeks, contained process strategies designed to facilitate behaviour change, including change champions, audit and feedback, education, and prompts and reminders.21 To address the issues identified with workflow,18 the PTB intervention itself was nurse led, and involved the allocation of the anaesthetic nurse to pass on case-relevant information obtained at the sign-in checkpoint (in the induction room) to other surgical team members who were in the OR, at an appropriate time. The operating surgeon led the time-out checkpoint and the circulating nurse was the designated to lead sign-out. The process to execute the time-out and sign-out checkpoints involved the deliberate use of a scripted checklist in a ‘call and response’ process, adequate engagement by the OR team, appropriate timing to consider workflow, coverage and duration for all items of the checklist.
Measures
Structured observations
We evaluated checklist participation/completion using behaviour descriptors based on previous literature.26–28 The data collectors, both experienced OR nurses, received study-specific training in observational methods. The three-part structured observational tool had 70 items, including 50 items having a ‘yes’/‘no’ response to indicate items covered, team members who were present and/or initiated each checkpoint, whether there was active participation, time/location of checks and free text to document field notes. Prior to data collection, the observational tool was pilot tested and inter-rater reliability was established. During pilot testing of the tool in the field, observers further clarified and cross-checked ratings for items, and consensus achieved. For each procedure, we tallied the number of team members who were present, initiated and participated in the execution of the SSC. We timed each checkpoint in minutes/seconds and measured length of surgery in minutes. Observers also documented field notes to give context around the interactions, actions and behaviours of team members.
Clinical incident data
In this study, we defined a ‘clinical error’ as any error in judgement or conduct in the clinical setting, regardless of whether or not the error leads to or causes an adverse event.29 30 We specifically focused on clinical errors such as surgical injuries, wrong site surgery, mistaken patient identities, mislabelling of human tissue specimens and retained foreign adjuncts/objects. The Severity Assessment Code (SAC) classification system31 was used in coding of clinical incidents/adverse events (online supplementary appendix) Characteristics of clinical incidents were coded based on cause, type, reason/issue, study phase and patient outcome (ie, none, minimal, medium, high level of harm) in an Excel database.
Data collection
We collected observational and clinical incident data over 27 months, spanning October 2014–2015 (preimplementation) through to November 2015–2016 (postimplementation). Observers recorded whether all items in sign-in, time-out and sign-out were attempted. Observers recorded three measures of quality: whether all specified information was communicated, whether all members were present and whether there was active participation. We also collected case-related information relative to team membership, personnel present, length of surgery and specialty. For the retrospective clinical incident audit, the database administrator drew data from the clinical incident information database. The lead author received these deidentified data in an encrypted Excel database. Participating and reporting of clinical incidents in the system is compulsory, and all public hospitals in Queensland have an electronic local risk management system. Once the system has been populated with incident data, the reports are linked to a state-wide repository. Local hospitals around Queensland submit incident data directly to the repository that collates, cleans, stores and uses it for the purposes of analysis. Online supplementary file 1 tabulates time points, methods of data collection and measures for data collected prospectively and retrospectively over the study period.
Analysis
Observational and clinical incident data were analysed using the Statistical Package for Social Sciences (SPSS; V.23, IBM). We cleaned the data and checked a random sample of 20% for accuracy. Descriptive analyses included absolute (n) and relative frequencies (%) to analyse categorical variables (role, qualifications, surgical specialty, checklist item completion, clinical error type), while means/SD were used for continuous data (numbers participated in checklist and number of items used). We calculated the mean compliance rate for each section of the checklist based on the sum of the number of items in each section divided by the total number of items (ie, behaviours/actions actually observed) in the checklist. We used inferential analyses using t-tests to examine mean differences across time points relative to the WHO SSC completion rates (sign-in, time-out, sign-out). Inter-rater reliability was calculated using interclass correlation coefficient (ICC) for observational data. A p value of <0.05 was considered significant and 95% CIs were used.
Ethics
Ethics approval was given by the ethics committees of the hospital (HREC/13/QGC/154) and the university (NRS/06/14/HREC). Participants gave written consent prior to observations. We advised participants of their rights to discontinue participation anytime during the data collection period. Following ethics approval, we sought permission to obtain deidentified clinical incident data from the Director-General, Queensland Health, as required by the Public Health Act (2005).
Results
Structured observations
Table 1 presents descriptive results of team and case-related characteristics across the three time points. In total, 77 surgical teams including anaesthetic and surgical consultants and registrars, residents and nurses were observed over the study period using the WHO SSC.
Descriptive summary for case-related results of observed cases
Table 2 displays descriptive results specifically in relation to checklist initiation and participation, and the proportion of item completion across its three checkpoints. The majority of teams observed across all phases of the study had five to six corei members32 present for each case. Across study time points, checklist compliance ranged from 75.9% to 82.8%, respectively. Significant improvements were noted in item completion rates across each checkpoint, particularly, in relation to sign-out where completion rates ranged from 79.3% to 94.5% (p<0.0001) following implementation of the PTB programme. Inter-rater reliability between observers across phases was adequate (ICC 0.65, p<0.001).
WHO checklist observation results
Clinical incident data
Table 3 presents the results of the clinical incident audit. During the preaudit and postaudit periods, 33 019 surgical procedures were performed, with an increase of 491 cases in the post-test period. A total of 75 incidents among 72 patients were sampled consecutively across the 2-year audit period (ie, at baseline there were 20 patients who had clinical incidents but some had more than one so there were 22 incidents). After implementation, 40 patients had 42 incidents, again some having more than one incident. However, only 64 incidents among 60 patients were included in this analysis as some were recorded during implementation of the PTB programme and 12 months after implementation (November 2016). We performed a descriptive analysis on the subsample of 64 cases, 22/16 264 (0.13%) before implementation and 42 (42/16 755 (0.25%)) after implementation (table 3), indicating a marginal increase (although non-significant) in the postimplementation period. The number SACii3 clinical incidents rose by 20 following implementation and the most predominant incident was inadequate tissue specimen labelling (23/42, 54.8%). The majority (20/22, 90.9% and 37/42, 88.1% respectively) of incidents across both phases did not result in any harm.
Clinical incident data
Discussion
We believe that the observed improvements in SSC use following implementation of the PTB programme were due to several factors that commonly lead to successful implementation of any process-based practice. These include having a comprehensive stakeholder-driven programme, using a staged stepwise process providing clear communication and real-time feedback, engaging active leadership, and process simplification and modification.15 33 Our results showed improvements in checklist participation and item usage rates, following the implementation of the PTB programme. PTB was intended to simplify the checking process through addressing behavioural deficits and contextual factors identified during observations, described in earlier work.18 24 In other research, the most common reasons health professionals identified for non-compliance were forgetfulness in using the SSC or in addressing some of its elements,17 26 or a lack of time to complete it.15 17 33 We observed substantial improvements in the use of sign-out—improvements that were sustained 12 months after implementation of the programme. Prior to implementation of the programme, we did not observe sign-out during baseline observations, thus any improvement in performance from baseline would be significant. Vogts et al 34 suggest that sign-out use is often low because this section is not explicitly linked to a specific event in patient management, unlike sign-in and time-out—which may have catastrophic consequences if these checkpoints are not completed. Sign-out may reduce the likelihood of the most common ‘never event’, a retained foreign object during surgery.1
In our study, inadequate tissue specimen labelling was the most common clinical incident—the ramifications of which can potentially have devastating effects for patients. Labelling errors can result in inappropriate therapy or withholding treatment in patients with unrecognised malignancies.35 Observed improvements in use of the checklist did not result in concomitant reductions in clinical incidents. There are three possible explanations for this finding. The first relates to the possibility that incidents may have been under-reported. The second reason centres on the observed increase in the reporting of SAC 3 incidents following implementation of the PTB programme. This increase may be the result of simultaneous improvements in safety culture over time. The third explanation concerns the reach of the PTB programme, which was not as widespread as we would have liked. During the study period, staff changes and turnover occurred, and likely impacted programme spread across all surgical specialties and teams. The impact of team and staff factors and the stability of the workforce may also contribute to the dilution of capacity, that is, skills and attitudes.36
Despite a decrease in observed item compliance in the sign-in phase after implementation, our results attest to the overall sustainability of the evidence-informed PTB programme. Clearly, a variety of factors may create conditions that enable initial implementation, their presence or influence often diminishes over time.36 Unintentional ‘slippage’ can occur because of contextual factors such as a lack of resources, staffing conditions or competing demands.37 We attribute sustainability of the PTB programme to factors related to both outer (policies) and inner (ie, culture, structure), the programme itself (fit, adaptability), implementation processes (ie, fidelity monitoring, efforts to align the intervention and the setting) and the capacity to sustain (ie, interpersonal processes such as leadership, support). Beyond implementation of PTB, the clinical leads on the implementation team used fidelity maintenance strategies such as audit and feedback, reminders, and provided real-time informal education and training support. The PTB programme was co-constructed with stakeholders,21 and ‘normalised’ in everyday practice over time.38 The implementation team included credible leaders from nursing and surgery, which contributed to its sustainability and spread. Following evaluation of the PTB programme at this facility, the programme was implemented at two smaller satellite hospitals within the health services district. Spread of the PTB programme to these facilities was augmented by the leadership of senior nursing staff who work across these hospitals. While implementation fidelity is important, modification and evolution of any programme and intervention may need to occur in response to shifting priorities and contextual influences, or in light of new evidence.36 Nonetheless, the importance of having an ongoing education programme cannot be understated given the continuing challenges brought about by staffing changes and attrition.
Limitations
Few studies have used prospective approaches to evaluate the implementation of practice change interventions.39 This is one of the first longitudinal studies to prospectively and retrospectively evaluate a patient safety programme designed to improve surgical teams’ participation in, and use of, the SSC. Nonetheless, we acknowledge some limitations. First, the use of a single hospital site and sampling methods of prospective data collection may limit generalisability and increase selection bias. Second, we assessed surgical teams’ use of the SSC with direct observations, with results showing moderate consistency between raters, likely due to some variation in interpretation of events. To minimise this, we developed a data dictionary, and the two observers were experienced OR nurses trained in observational research methods. Third, the observational nature of the study may have given rise to the Hawthorne effect. Yet it is likely that this diminished over time as participants became accustomed to the presence of the observers. Fourth, at the time of the implementation, the PTB programme was not mandated, therefore, participation was essentially voluntary, potentially leading to self-selection. However, since its introduction, the PTB programme has been included in the health services policy, and has been subsequently introduced at two similar hospital sites across the health services district. Fifth, clinical incident data are secondary, self-reported and represent a non-random sample of errors from an unknown universe of errors. As such, these data cannot generate incidence rates or capture the entire universe (ie, denominator) of errors. Finally, there may have been some misclassification in the coding of harm and other variables. Despite these limitations, our study has considerable strengths in terms of its longitudinal nature and use of data triangulation. These results offer unique insights into SSC use and the types of clinical incidents identified in surgery. We anticipate these results will drive further improvements in checklist implementation and use beyond the facility where the PTB programme was developed and implemented.
Conclusion
Our study showed improvements in team participation and item adherence. Yet, there were no reductions in clinical incidents after implementation of the PTB programme. Despite this, our results suggest an increase in the reporting of clinical incidents, particularly during the sign-out period (ie, specimens, surgical count) increased after implementation. Adaptive changes in the department culture brought about through the introduction of targeted programmes such as PTB, although taking longer to achieve, may have a greater impact and lead to more sustainable improvements than simply concentrating on the technical aspects of the checklist. Clearly, the benefit in using a surgical checklist lies in the potential to enhance team communications and the promotion of a team culture in which safety is a high priority.
Supplemental material
References
Footnotes
↵i Core membership is defined as team members who had worked together, weekly or fortnightly, for a minimum of 3 months.32
↵ii SAC=Severity Assessment Code is a rating system used to classify and prioritise an incident. Ratings from 1 to 3 are based on the severity of the outcome for the patient. See online supplementary appendix.
Contributors BMG planned the study and led all aspects of the project. ELH analysed the data. APM, ELH, KH and TKW provided input in relation to methodology and subject matter. BMG, TG, JL and TKW collected the data. BMG drafted the manuscript. All authors participated in interpretation of the study results, assisted in manuscript revision and approved the final version of the manuscript.
Funding BMG was supported by a National Health and Medical Research Council (NHMRC) (Australia) Translation into Practice (TRIP) Fellowship and the Australian College of Perioperative Nurses (ACORN) Research Grant.
Competing interests None declared.
Patient consent Not required.
Ethics approval Ethics approval was given by the ethics committees of the hospital (HREC/13/QGC/154) and the university (NRS/06/14/HREC).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The data contributing to these analyses are held on a secure database held by the lead author.