Evaluation of a patient safety programme on Surgical Safety Checklist Compliance: a prospective longitudinal study

Problem description

Despite that checklist use has become more widespread, results have not been so impressive and have been mixed or inconclusive.14 15 Some studies have shown that although 100% compliance with the SSC was documented, in reality, checklists were completed less than 10% of the time when operating room (OR) staff were observed.16 Checklists may contribute to improved patient safety outcomes; however, they are often implemented as part of multicomponent quality improvement initatives.14 Additionally, there is mounting evidence identifying contextual and organisational challenges in relation to ease of checklist adoption into workflow patterns,17 18 and their true impact on patient safety.16 19 20 These aspects require high commitment to practice change at all levels, particularly relative to physician involvement and organisational support.21 Unsurprisingly, many healthcare organisations achieve only partial implementation success when they are initiated.22 23

To date, few studies have evaluated the observed use of the SSC longitudinally following the implementation of a patient safety programme. The objectives of this study were to: (1) assess checklist item completeness rates before and after implementation of a patient safety programme designed to improve surgical teams’ participation in safety checks using the SSC; and (2) describe the impact of the intervention on the prevalence of clinical errors.

Study overview

We conducted a prospective longitudinal study using structured observations and undertook a retrospective secondary analysis of electronic database audit. The primary outcomes included observed SSC participation and completion rates, while the secondary outcome was period prevalence of clinical errors. Following baseline measures, we implemented a patient safety programme over 4 weeks and collected observational measures at 3 and 12 months’ postimplementation. We undertook the audit of electronic secondary outcome data retrospectively, 12 months before and 12 months after programme implementation. Online supplementary file 1 details the data collection time points, method of collection and measures across implementation phases.

Context

The study setting was a 750-bed university hospital in Queensland, Australia. The facility had 16 commissioned operating rooms, catering for all specialties except transplantation, and performs approximately 16 000 surgeries each year. Surgical teams consisted of anaesthetists and surgeons and their registrars, and nurses working in circulating/instrument and anaesthetic roles. As the programme was implemented department-wide, all staff (n=150) from surgery, anaesthetics and nursing were invited to participate in the preimplementation and postimplementation using structured observations. The clinical incident data were drawn from a consecutive sample of patients who underwent elective, emergent and emergency surgeries at the study site over a 24-month period between October 2014–2015 and November 2015–2016. The Pass The Baton (PTB) programme was delivered in November 2015.

Intervention

The ultimate goal of the WHO SSC is to ensure that surgical teams consistently follow several critical safety steps, thus minimise the most common and avoidable risks endangering the well-being of patients.1 The original 19-item three SSC checkpoints include: sign-in (preinduction), time-out (surgical pause prior to commencement of the procedure) and sign-out (checks performed during or just after skin closure). The WHO recommends that a designated coordinator (often the circulating nurse) review verbally each checkpoint with the appropriate team member(s) to confirm that all key actions for that phase of care have been performed.1 Having a dedicated person lead the checklist process ensures that safety steps are not missed or rushed in an effort to progress with the next phase of the procedure. However, while this person is often the circulating nurse, any member of the surgical team can coordinate the checklist.1

At the study hospital, the WHO SSC was adapted to meet the needs of the local context and included 20 items across the sign-in, time-out and sign-out checkpoints. Although mandated throughout the OR department, the SSC was haphazardly used and inconsistently completed, with varying levels of staff participation.24 The patient safety programme, PTB, was implemented department-wide, across all surgical specialties to increase item coverage and team participation across the three checkpoints. Development of the PTB programme was informed by a series of studies15 18 24 conducted at the study site. The most significant barriers to using the SSC as recommended were workflow, limited knowledge about timing/content of checks, a lack of clinical leadership and dissonant attitudes.18 24

Prior to implementing the PTB programme, two team members (BMG and JL) had initial discussions to identify other interested staff members. BMG has had a long-standing association as a researcher in the OR department, and is well known to many of the nursing and medical staff. JL is a senior nurse, working as a perioperative educator. In this role, she has implemented other types of practice improvement initiatives. As a staff member, JL is highly regarded, and considered an opinion leader. JL identified a mix of nurses working across all perioperative areas to champion PTB programme. Both BMG and JL approached several physicians in surgery and anaesthetics to garner support. The implementation team consisted of anaesthetic/circulating/ instrument nurses, a senior physician and two researchers. The clinicians on the team were experts in their respective fields and were highly respected influential leaders within the department. The programme was coproduced with staff across surgery and anaesthetics. A logic model (ie, a diagram describing structures in place to deliver the programme, the intended activities and intended short, medium and long-term outcomes)25 was developed a priori by the implementation team. The PTB programme, delivered over 4 weeks, contained process strategies designed to facilitate behaviour change, including change champions, audit and feedback, education, and prompts and reminders.21 To address the issues identified with workflow,18 the PTB intervention itself was nurse led, and involved the allocation of the anaesthetic nurse to pass on case-relevant information obtained at the sign-in checkpoint (in the induction room) to other surgical team members who were in the OR, at an appropriate time. The operating surgeon led the time-out checkpoint and the circulating nurse was the designated to lead sign-out. The process to execute the time-out and sign-out checkpoints involved the deliberate use of a scripted checklist in a ‘call and response’ process, adequate engagement by the OR team, appropriate timing to consider workflow, coverage and duration for all items of the checklist.

Measures

Data collection

We collected observational and clinical incident data over 27 months, spanning October 2014–2015 (preimplementation) through to November 2015–2016 (postimplementation). Observers recorded whether all items in sign-in, time-out and sign-out were attempted. Observers recorded three measures of quality: whether all specified information was communicated, whether all members were present and whether there was active participation. We also collected case-related information relative to team membership, personnel present, length of surgery and specialty. For the retrospective clinical incident audit, the database administrator drew data from the clinical incident information database. The lead author received these deidentified data in an encrypted Excel database. Participating and reporting of clinical incidents in the system is compulsory, and all public hospitals in Queensland have an electronic local risk management system. Once the system has been populated with incident data, the reports are linked to a state-wide repository. Local hospitals around Queensland submit incident data directly to the repository that collates, cleans, stores and uses it for the purposes of analysis. Online supplementary file 1 tabulates time points, methods of data collection and measures for data collected prospectively and retrospectively over the study period.

Analysis

Observational and clinical incident data were analysed using the Statistical Package for Social Sciences (SPSS; V.23, IBM). We cleaned the data and checked a random sample of 20% for accuracy. Descriptive analyses included absolute (n) and relative frequencies (%) to analyse categorical variables (role, qualifications, surgical specialty, checklist item completion, clinical error type), while means/SD were used for continuous data (numbers participated in checklist and number of items used). We calculated the mean compliance rate for each section of the checklist based on the sum of the number of items in each section divided by the total number of items (ie, behaviours/actions actually observed) in the checklist. We used inferential analyses using t-tests to examine mean differences across time points relative to the WHO SSC completion rates (sign-in, time-out, sign-out). Inter-rater reliability was calculated using interclass correlation coefficient (ICC) for observational data. A p value of <0.05 was considered significant and 95% CIs were used.

Ethics

Ethics approval was given by the ethics committees of the hospital (HREC/13/QGC/154) and the university (NRS/06/14/HREC). Participants gave written consent prior to observations. We advised participants of their rights to discontinue participation anytime during the data collection period. Following ethics approval, we sought permission to obtain deidentified clinical incident data from the Director-General, Queensland Health, as required by the Public Health Act (2005).

Structured observations

Table 1 presents descriptive results of team and case-related characteristics across the three time points. In total, 77 surgical teams including anaesthetic and surgical consultants and registrars, residents and nurses were observed over the study period using the WHO SSC.

Table 2 displays descriptive results specifically in relation to checklist initiation and participation, and the proportion of item completion across its three checkpoints. The majority of teams observed across all phases of the study had five to six coreⁱ members32 present for each case. Across study time points, checklist compliance ranged from 75.9% to 82.8%, respectively. Significant improvements were noted in item completion rates across each checkpoint, particularly, in relation to sign-out where completion rates ranged from 79.3% to 94.5% (p<0.0001) following implementation of the PTB programme. Inter-rater reliability between observers across phases was adequate (ICC 0.65, p<0.001).

Clinical incident data

Table 3 presents the results of the clinical incident audit. During the preaudit and postaudit periods, 33 019 surgical procedures were performed, with an increase of 491 cases in the post-test period. A total of 75 incidents among 72 patients were sampled consecutively across the 2-year audit period (ie, at baseline there were 20 patients who had clinical incidents but some had more than one so there were 22 incidents). After implementation, 40 patients had 42 incidents, again some having more than one incident. However, only 64 incidents among 60 patients were included in this analysis as some were recorded during implementation of the PTB programme and 12 months after implementation (November 2016). We performed a descriptive analysis on the subsample of 64 cases, 22/16 264 (0.13%) before implementation and 42 (42/16 755 (0.25%)) after implementation (table 3), indicating a marginal increase (although non-significant) in the postimplementation period. The number SACⁱⁱ3 clinical incidents rose by 20 following implementation and the most predominant incident was inadequate tissue specimen labelling (23/42, 54.8%). The majority (20/22, 90.9% and 37/42, 88.1% respectively) of incidents across both phases did not result in any harm.

Limitations

Few studies have used prospective approaches to evaluate the implementation of practice change interventions.39 This is one of the first longitudinal studies to prospectively and retrospectively evaluate a patient safety programme designed to improve surgical teams’ participation in, and use of, the SSC. Nonetheless, we acknowledge some limitations. First, the use of a single hospital site and sampling methods of prospective data collection may limit generalisability and increase selection bias. Second, we assessed surgical teams’ use of the SSC with direct observations, with results showing moderate consistency between raters, likely due to some variation in interpretation of events. To minimise this, we developed a data dictionary, and the two observers were experienced OR nurses trained in observational research methods. Third, the observational nature of the study may have given rise to the Hawthorne effect. Yet it is likely that this diminished over time as participants became accustomed to the presence of the observers. Fourth, at the time of the implementation, the PTB programme was not mandated, therefore, participation was essentially voluntary, potentially leading to self-selection. However, since its introduction, the PTB programme has been included in the health services policy, and has been subsequently introduced at two similar hospital sites across the health services district. Fifth, clinical incident data are secondary, self-reported and represent a non-random sample of errors from an unknown universe of errors. As such, these data cannot generate incidence rates or capture the entire universe (ie, denominator) of errors. Finally, there may have been some misclassification in the coding of harm and other variables. Despite these limitations, our study has considerable strengths in terms of its longitudinal nature and use of data triangulation. These results offer unique insights into SSC use and the types of clinical incidents identified in surgery. We anticipate these results will drive further improvements in checklist implementation and use beyond the facility where the PTB programme was developed and implemented.