Article Text
PDF
Abstract
Background The recurrence of sentinel events (SEs) is a persistent problem worldwide, despite repeated analyses and recommendations formulated to prevent recurrence. Research suggests this is partly attributable to the quality of the recommendations, and determining if a recommendation will be effective is not yet covered by an adequate guideline. Our objectives were to (1) develop and validate criteria for high-quality recommendations, and (2) evaluate recommendations using the criteria developed.
Methods (1) Criteria were developed by experts using the bowtie method. Medical doctors then determined if the recommendations of Dutch in-hospital SE analysis reports met the criteria, after which interobserver variability was tested. (2) Researchers determined which recommendations of Dutch perioperative SE analysis reports produced from 2017 to 2018 met the criteria.
Results The criteria were: (1) a recommendation needs to be well defined and clear, (2) it needs to specifically describe the intended changes, and (3) it needs to describe how it will reduce the risk or limit the consequences of a similar SE. Validation of criteria showed substantial interobserver agreement. The SE analysis reports (n=115) contained 442 recommendations, of which 64% failed to meet all criteria, and 28% of reports did not contain a single recommendation that met the criteria.
Conclusion We developed and validated criteria for high-quality recommendations. The majority of recommendations did not meet our criteria. It was disconcerting to find that over a quarter of the investigations did not produce a single recommendation that met the criteria, not even in SEs with a fatal outcome. Healthcare providers have an obligation to prevent SEs, and certainly their recurrence. We anticipate that using these criteria to determine the potential of recommendations will aid in this endeavour.
- healthcare quality improvement
- incident reporting
- never events
- patient safety
- root cause analysis
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
The WHO is urging healthcare providers to improve patient safety globally.1 Unintended healthcare events that cause death or significant injury, so-called sentinel events (SEs), have a major impact on patients, relatives and healthcare workers and result in the loss of 64 million disability-adjusted life years per year worldwide.2 3 The majority of reported in-hospital SEs are associated with surgical procedures.4 Wrong-site surgery (WSS), for example—defined as invasive procedures on the wrong patient, at the wrong site or using the wrong procedure—is a persistent problem in most countries.5–8 WSS and unintended retention of a foreign body have both been listed as obvious unacceptable errors or ‘never events’. These never events are deemed preventable and are a major concern to both patients and healthcare workers.9 Many countries have implemented a reporting system for SEs, and in the Netherlands, healthcare organisations are obliged to notify the Dutch Health and Youth Care Inspectorate (DHI) of all SEs, analyse each SE and send the analysis report to the DHI.2 Based on this analysis, healthcare organisations formulate and implement recommendations aimed at preventing recurrence. Despite these and similar efforts, recurrent SEs such as WSS remain a persistent and widespread problem.6–8 A recent survey among Dutch academic hospitals confirmed the recurrence of similar SEs, but also cast doubt on the quality of a significant proportion of the recommendations.10 This does not solely apply to Dutch healthcare, as Hibbert et al have previously shown that only 8% of Australian recommendations are of high-quality.11 However, there is currently no adequate guideline or strategy to determine if a recommendation will be effective, and selecting recommendations for implementation is still based on subjective criteria such as estimated feasibility.10 12
The DHI advises formulating recommendations based on Specific, Measurable, Achievable, Realistic and Timely (SMART) principles.13 However, using SMART does not necessarily result in recommendations that reduce the risk of SE recurrence. For instance, if an SE occurs after a specific protocol was ignored, a frequently encountered recommendation is to improve awareness by sending the specific protocol via email to all healthcare workers within 1 month. While this recommendation may be formulated according to SMART principles, it is not an effective, sustainable intervention. Clearly, the content of the protocol needs to be scrutinised and the reason why it was ignored should be investigated. Merely sending a reminder via email will not resolve the underlying problems that made the SE possible.
In this study, we (1) developed and validated basic criteria to determine the potential effect of recommendations designed to reduce the risk of recurrent SEs, and (2) evaluated the recommendations found in Dutch perioperative SE analysis reports using the newly developed criteria. Although implementation of recommendations and the monitoring of effects are critical steps in the learning process, this was considered beyond the scope of the present study.10 14
Methods
Part 1a: developing criteria
An expert-based consensus meeting was organised that comprised three medical doctors specialised in the handling of SEs in healthcare, together with two experts specialised in dealing with the ‘human factor’ during incident analysis in other safety-critical industries. The ‘human factor’ field involves the study of humans interacting with their environment.15 The goal of the meeting was to develop simple and easily applicable criteria to determine the potential effect of recommendations. The starting point for developing criteria was the bowtie method as used in barrier management. Barriers are physical or non-physical methods implemented by an organisation to prevent or mitigate SEs caused by hazards.16
The bowtie method, shown in figure 1, distinguishes barriers that aim to ‘prevent’, thus reducing the risk of a similar SE, or ‘mitigate’, representing an attempt to limit the consequences of an SE. Ideally, recommendations should interact with these barriers, an example of which might be that a patient dies as a result of a fire in an operating theatre due to a malfunctioning apparatus. Replacing the malfunctioning machine would be a preventive action, as it reduces the risk of a similar SE and thus belongs on the left side of the bowtie. Installing better fire detection equipment would be a corrective action, as it limits the consequences of an SE and therefore belongs on the right side of the bowtie. Components of recommendations which appear to enhance the effectiveness of recommendations in preventing SEs, the ‘core component’ or ‘active component’, are viewed as another basic condition.17 Modifying a protocol, for example, is a meaningless recommendation without elaborating on the content. In order to distinguish an active component, the seven main questions comprising the steps ‘searching for earlier effective recommendations’ and ‘clarifying the recommendation’ (such as ‘what is the core component of an earlier effective recommendation’ and ‘how can that component be integrated into the organisation’) need to be answered.18 The criteria were reviewed by an academic cognitive psychologist specialised in Safety Science, a senior DHI inspector, a former chief medical officer now specialised in SEs in healthcare and a healthcare implementation fellow. The criteria were adjusted based on their suggestions.
Bowtie method.16
Part 1b: validating criteria
Validation of criteria facilitates appropriate use when selecting recommendations with the potential to reduce the risk of recurrent SEs. All Dutch healthcare organisations are obliged to report SEs to the DHI within 3 days after the occurrence or discovery of the event.2 All correspondence between healthcare organisations and the DHI regarding the reported SE, including the results of the analysis and recommendations, is collected in the DHI database and the DHI assesses the quality of the analysis reports.2 Ten complete analysis reports of general in-hospital SEs, assessed and approved by the DHI between 1 October 2017 and 1 January 2018, were randomly selected from the database for use in validating the criteria. Three medical doctors individually determined if the recommendations met the criteria, after which interobserver variability was tested.
Part 2: evaluating recommendations
In the DHI database all perioperative SEs that occurred between 1 July 2017 and 1 July 2018 were selected. A perioperative SE was defined as an SE that occurred prior, during or after an intervention executed in the operating theatre performed by a medical doctor in collaboration with an anaesthesiologist. All recommendations found in these SE analysis reports were noted. In addition, type of hospital, analysis method, analysis period and outcome of the SE were recorded. Two researchers (KB and A-FT (Acknowledgements)) determined, in consensus, which recommendations met the criteria.
Statistical analysis
Interobserver agreement was tested by calculating Fleiss’ kappa (κ) and 95% CIs. Values ≤0 were interpreted as indicating no agreement, 0.01–0.20 as none to slight agreement, 0.21–0.40 as fair agreement, 0.41–0.60 as moderate agreement, 0.61–0.80 as substantial agreement and 0.81–1.00 as almost perfect agreement.19 Descriptive data were reported as numbers with percentages (%) or median with IQR. The χ2 test or Fisher’s exact test was used when appropriate to compare proportions. The Mann-Whitney U test and Kruskal-Wallis test were used to compare continuous variables between the groups. P<0.05 was considered statistically significant. IBM SPSS Statistics for Windows (V.26.0, IBM) was used to perform statistical analysis.
Patient and public involvement
Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.
Results
Part 1a: developing criteria
The following three criteria for recommendations were developed:
Based on the active component17 → the recommendation needs to be well defined and clear.
Based on the relevant questions18 → the recommendation needs to specifically describe what will be modified in the future.
Based on the bowtie method16 → the recommendation needs to describe how it will reduce the risk or limit the consequences of a similar SE.
Part 1b: validating criteria
The 10 analysis reports on general in-hospital SEs randomly selected to validate the criteria yielded 38 recommendations. The three observers individually determined the same outcome for 29 of the 38 recommendations after applying the criteria. Fleiss’ kappa (κ) was 0.67 (95% CI 0.48 to 0.85), resulting in substantial interobserver agreement.19
Part 2: evaluating recommendations
Between 1 July 2017 and 1 July 2018, a total of 115 perioperative SEs were reported to the DHI. Seventy-six (66%) SEs resulted in major injury and 39 (34%) in death. Eighty-five (74%) SEs occurred in the operating theatre, including 18 (21%) WSS cases and 16 (19%) cases of unintended retention of a foreign body.
The 115 perioperative SE analysis reports contained 442 recommendations in total. The median number of recommendations per SE was 4 (IQR 2–5) and the median number of recommendations per SE which met the criteria was 1 (IQR 0–2). Hospital type, analysis method, analysis period or the outcome of the SE did not significantly influence the mean number, as shown in online supplemental file 1. In total, 281 (64%) recommendations did not meet the three criteria.
Supplemental material
Of the 115 SE analysis reports, 32 (28%) did not result in any recommendation that met the criteria. Eighty-three (72%) SE analysis reports produced at least one recommendation that met the criteria, as shown in online supplemental file 2. Of the 442 recommendations, five did not meet any of the three criteria. Half of the recommendations (244) met one of the three criteria, 32 recommendations met two of the three criteria and 161 met all three criteria. Almost every recommendation (437) had a well-defined and clear goal. However, more than half of the recommendations did not specifically describe what needed to be modified (260) or how it might reduce the risk or limit the consequences of a similar SE (270) in the future.
Supplemental material
Discussion
In this study, a basic set of criteria was developed and validated that can be applied to determine the potential effect of recommendations to reduce the risk of recurrent SEs. Thereafter, an evaluation of the recommendations of Dutch perioperative SE analysis reports using the criteria developed was performed. The three criteria were easy to use and showed substantial interobserver agreement. Evaluation of the recommendations found in SE analysis reports indicated that most do not meet the three criteria and over a quarter of all investigations did not include a single recommendation that met the criteria.
Critical appraisal of our findings
Over the period 2014–2016 there were 100 reported cases in the Netherlands of WSS or unintended retention of a foreign body.7 In this study, 34 of these cases were found within the selected year. Assuming that an approximately equal number of surgical procedures are performed each year, this number is consistent with expectations. The majority of recommendations did not meet our three criteria, which suggests that current recommendations are either not specifically aimed at reducing SEs or are not sufficiently well designed to reduce the risk of recurring never events.9
The number of reported SEs alone is not an adequate measure of the quality of delivered healthcare. The occurrence of an SE is influenced by a variety of factors, including awareness of patient safety and SEs, and the learning potential in dealing with SEs. However, the mere fact of a recurrence of a similar SE is in our view a reflection on the quality of delivered healthcare and an indication of how and whether we can learn and improve. Criteria that provide insight regarding the fundamentals of a recommendation could aid in decreasing the number of similar SEs and thus improve patient safety.
The quality of recommendations is only one step in the process of learning from SEs. A failure to draw appropriate conclusions from an SE is most likely multifactorial in nature, as the learning potential consists of many steps and each step is crucial. However, if recommendations fall short, subsequent steps in the learning process will be negatively affected and the ultimate goal of improvement will not be achieved.10 14 Recent studies concluded that high-quality recommendations represent only a small proportion of the total and suggested this might be the reason why similar SEs persist.10 11
In the Netherlands, recommendations in relation to the analysis of SEs must be formulated according to SMART.13 However, using SMART does not necessarily result in recommendations aimed at reducing the risk of recurrent SEs, as SE analysis might highlight shortcomings in a healthcare system that did not directly contribute to that particular SE. For example, a patient dies following unintended retention of a foreign body. During the SE analysis, the analysis team notices technical shortcomings of the operating table and formulates a recommendation to ensure correct functioning of the operating table so that the table does not present a risk in future operations. While this recommendation is clearly aimed at improving patient safety, it is unlikely to have any effect on reducing the risk of an SE due to unintended retention of a foreign body.
To improve objectivity, members of the analysis team are frequently drawn from departments other than the department in which the SE occurred. This means that the home team (professionals working within the involved department) is not included in formulating the recommendations, and extensive information regarding specific medical knowledge or workflow might be taken less into account. As a result, formulating adequate recommendations becomes more challenging, since the exact circumstances in the department are likely to be unfamiliar to SE analysis team members. This could consequently result in recommendations that lack detail and have less impact on patient safety than intended. If our goal is to decrease the number of similar SEs—the reason why we actually analyse SEs—we must be objective but also take this possible reduced impact of the recommendations formulated by ‘outsiders’ into account. The developed criteria can support in formulating high-quality recommendations without the knowledge of the home team.
Our criteria were derived using the bowtie method. The bowtie method helps distinguish barriers that may eliminate or reduce the risk of a similar SE,16 and an ideal recommendation should therefore interact with these barriers. When evaluating currently formulated recommendations, the majority fail to meet these criteria and as such show little potential to reduce the risks underlying the SE. It is not a prerequisite to be familiar with the bowtie method, we deconstructed this method and used it in our criteria. Keeping the criteria in mind can help maintain focus, and as such a departmental head or the executive board could use the criteria to select recommendations, and the analysis team could formulate risk-reducing recommendations based on the criteria.
Limitations
A limitation of our study is the fact that the criteria were not prospectively tested. Therefore, the impact and sustainability of the criteria on the quality of recommendations, the number of similar SEs and ultimately patient safety has yet to be proven in future prospective research. However, the criteria are based on available literature and validation resulted in substantial interobserver agreement. As there is currently no adequate guideline or strategy to determine if a recommendation will be effective, our easily applicable criteria could aid in filling this gap.
The bowtie method used here was developed in safety-critical environments other than healthcare. The number of studies within the medical field concerning the basic conditions of a recommendation is therefore limited, which might call into question the relevance of literature from other safety-critical industries to the medical field. However, the petrochemical industry, for example, played a prominent role in developing the Dutch reporting system that is used to this day.20
For practical reasons, the consensus meeting included a small panel of experts. Organising a broader team might have yielded different results, all team members included though were knowledgeable and experienced in dealing with SEs.
During the validation process we noticed that SE context is important, and in order to apply the criteria detailed information on the SE is essential. We expect this will be less of a challenge to a team analysing an SE than to the medical doctors involved in the validation process of our study, as the former will have access to detailed information on the SE, the department and the organisation where the SE took place.
In this study, we focused on recommendations found in Dutch perioperative SE analysis reports, and although we subsequently validated the criteria using recommendations from general in-hospital SE analysis reports, this might still limit the relevance in other healthcare systems. However, we are confident that the quality of recommendations is a universal problem and that our observations are potentially relevant to other healthcare systems and other safety-critical industries.11
Conclusion and impact
The goal of SE analysis is to provide recommendations that result in the improvement of patient safety. Nonetheless, similar SEs do recur and research suggests that the quality of recommendations might be partly responsible. In this study, we developed and validated three basic criteria that can be used to determine the potential effect of recommendations on reducing the risk of recurrent SEs, and we subsequently evaluated recommendations contained in 115 Dutch perioperative SE analysis reports. The majority of recommendations did not meet the three formulated criteria, and it was especially disconcerting to find that over a quarter of all investigations did not lead to a single recommendation that met the criteria. In other words, despite the fact that a patient had suffered serious harm or had even died, one out of four investigations did not effectively address the causes of the SE. It must be recognised that it is very well possible that the highest quality recommendations cannot be implemented due to valid practical reasons, such as financial constraints or lack of other resources. However, healthcare providers have an obligation to both their patients and colleagues to prevent SEs, and certainly their recurrence. We expect that using the three criteria formulated in this study will help determine the value of safety recommendations and thus aid in this endeavour.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Acknowledgments
The authors thank Gert-Jan Kamps and Alfred van Wincoop, consultant and engineer at Intergo International Centre for Safety, Ergonomics and Human Factors (Amersfoort, the Netherlands); Maurice Vlemminx, senior inspector of the Dutch Health and Youth Care Inspectorate (Utrecht, the Netherlands); Gerjan Heuver, former chief medical officer at Gelre Hospitals (Apeldoorn and Zutphen, the Netherlands); Gera Welker, implementation expert at University Medical Center Groningen (Groningen, the Netherlands); Anne-Fleur The, medical intern at University Medical Center Groningen (Groningen, the Netherlands) and the Dutch Health and Youth Care Inspectorate (Utrecht, the Netherlands) for their contribution to this study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Study conception and design: all authors. Acquisition of data: KB. Analysis and interpretation of data: KB and A-FT (Acknowledgements). Drafting the manuscript: KB. Revising it critically for important intellectual content: all authors. All authors approved the final version of the manuscript.
Funding This work was supported by the Quality-based Governance programme and was funded by the Citrien Foundation. This foundation supports the development of sustainable and broadly applicable solutions in healthcare and is made possible by ZonMw.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.