Article Text
PDF
Abstract
Purpose The purpose of this systematic review was to identify an appropriate method—a user-friendly and validated method—that prioritises recommendations following analyses of adverse events (AEs) based on objective features.
Data sources The electronic databases PubMed/MEDLINE, Embase (Ovid), Cochrane Library, PsycINFO (Ovid) and ERIC (Ovid) were searched.
Study selection Studies were considered eligible when reporting on methods to prioritise recommendations.
Data extraction Two teams of reviewers performed the data extraction which was defined prior to this phase.
Results of data synthesis Eleven methods were identified that are designed to prioritise recommendations. After completing the data extraction, none of the methods met all the predefined criteria. Nine methods were considered user-friendly. One study validated the developed method. Five methods prioritised recommendations based on objective features, not affected by personal opinion or knowledge and expected to be reproducible by different users.
Conclusion There are several methods available to prioritise recommendations following analyses of AEs. All these methods can be used to discuss and select recommendations for implementation. None of the methods is a user-friendly and validated method that prioritises recommendations based on objective features. Although there are possibilities to further improve their features, the ‘Typology of safety functions’ by de Dianous and Fiévez, and the ‘Hierarchy of hazard controls’ by McCaughan have the most potential to select high-quality recommendations as they have only a few clearly defined categories in a well-arranged ordinal sequence.
- adverse events
- epidemiology and detection
- incident reporting
- patient safety
- quality improvement
- root cause analysis
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
- adverse events
- epidemiology and detection
- incident reporting
- patient safety
- quality improvement
- root cause analysis
Introduction
Adverse events (AEs)—defined as unexpected occurrences involving death or serious physical or psychological injury—affect numerous patients in healthcare organisations worldwide.1 Solely in the Netherlands, 1272 AEs were reported by hospitals, private clinics and rehabilitation centres in 2016.2 Many countries have developed a system to register and analyse these AEs in an attempt to prevent recurrence and improve patient safety.3 However, this has not yet resulted in a decrease in the number of AEs.4 Although this suggests that learning from AEs is insufficient, the number of AEs alone is not a measure for the learning effect. An increase in the number of AEs does not necessarily mean healthcare has become less safe. AEs could be better recognised and, therefore, reported more frequently. Recurrence of similar AEs is a better measure for the effect of learning from AEs.
In the Netherlands, 60 cases of wrong-site surgery were reported to the Dutch Healthcare Inspectorate between 2014 and 2016.5 Between April 2014 and March 2015, 124 cases of wrong-site surgery were reported in the UK; and in the USA these events occur approximately 1300 to 2700 times annually.6 7 Despite previous analyses of these type of AEs, they still recur on a daily basis worldwide. Recurrence of similar AEs strongly suggests learning from AEs is complex and unsatisfactory. This might be partly due to the quality of recommendations following analyses of AEs. In Australia, the quality of recommendations following analyses of AEs was recently assessed to investigate their effectiveness and sustainability. Of all the 1137 recommendations evaluated, only 8% were of high-quality.8 In order to achieve a potential reduction in (similar) AEs in healthcare, it seems plausible to focus on implementing the high-quality recommendations.
Insight in the basic conditions of a high-quality recommendation will improve the quality of the recommendations and ultimately improve patient safety. Furthermore, it might aid in directing time and resources when selecting recommendations for implementation. Therefore, the purpose of this systematic review was to identify an appropriate method—a user-friendly, validated method—that prioritises recommendations following analyses of AEs in healthcare based on objective features.
Methods
This review protocol was registered in PROSPERO, the international prospective register of systematic reviews (registration number CRD42018092002) and reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement.9 10
Search strategy
The electronic databases PubMed/MEDLINE, Embase (Ovid), Cochrane Library, PsycINFO (Ovid) and ERIC (Ovid) were searched for published studies on recommendations following analyses of AEs with the assistance of a clinical librarian on 9 March 2018. Search terms included: healthcare, hospital, quality improvement, safety management, recommendation, safety intervention, remedial action, improvement tool, usefulness and usability. The detailed search strategies are presented in online supplemental file 1. No restrictions regarding language, study design or publication date were applied. The reference lists of eligible studies were manually screened to identify additional relevant studies. Through a human factor consultant and engineer at Intergo Human Factors and Ergonomics (Acknowledgements), both specialised in dealing with AEs in safety-critical industries, eligible studies from other industries than healthcare were obtained.
Supplemental material
Study selection
Data was processed using the Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia; available at www.covidence.org). Prior to the study selection, duplicates were removed. Studies were considered eligible when reporting on methods to prioritise recommendations following analyses of AEs. The methods must be intended for or applicable to healthcare. Title and abstract screening was performed by two teams of reviewers (KB/DAD or KB/MJvdL). In case of disagreement, consensus was reached through discussion within the teams, and when necessary, the opinion of the third reviewer was obtained. If the full-text study could not be extracted, the corresponding author of the concerning study was contacted and the full text was requested. Of all selected studies, the full text was analysed by both teams (KB/DAD or KB/MJvdL) and all reviewers agreed on the final selection of studies.
Data extraction
Two teams of reviewers (KB/DAD or KB/MJvdL) performed the data extraction that was defined by the authors prior to this phase. In case of different outcomes, consensus was reached through discussion.
Definition of outcomes
Predefined criteria to assess the quality of the method prioritising recommendations were used to extract the data. The primary outcomes were:
‘user-friendly’: easy to understand, not time consuming and with simple calculations;
‘validation’: the method is tested and results are considered to be reproducible when used different times by the same user and/or by different users;
‘prioritisation of recommendations only’: recommendations can be prioritised without taking other factors such as implementation or other steps of incident analysis into consideration;
‘prioritisation based on objective features’: features or scores were considered to be objective when personal opinion or knowledge was not expected to affect the scoring process, and the scoring could be expected to be similar when performed by different users.
Other outcomes were the ‘description of the method’, ‘categories or scores used for prioritisation’ and ‘development of the method’. This included what the method was based on, which experts were involved in the development and ‘description of the validation’. ‘Year’, ‘country’ and ‘industry’ of publication and ‘applicability to healthcare’ were also extracted. Since the data was not appropriate for quantitative synthesis, no additional statistical analyses were performed.
Quality assessment
Unfortunately, no appropriate quality assessment tool was available for this type of research. The available quality assessment tools for qualitative research do not apply to all types of qualitative research especially when no experimental data, cohorts or interventions were evaluated.
Patient and public Involvement
Patients were not involved in this study.
Results
Included studies
The systematic search identified a total of 1297 studies. After screening of title and abstract, 49 studies were considered eligible. After reviewing the full-text paper, 10 studies were included. One study was added to the included studies after screening the references of the included studies. More detailed information regarding the study selection process is presented in figure 1.
PRISMA flow diagram. PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Study characteristics
The 11 included studies were published between 1990 and 2017.11–21 The studies originated from the USA, Norway, Spain, Ireland, the UK, Turkey, France and Belgium. The methods of eight studies were intended for healthcare.11 12 14 16 17 19–21 Three studies were intended for safety-critical industries in general13 15 18 and two of these were applicable to healthcare as stated by the authors.15 18 The study by de Dianous and Fiévez did not explicitly state the applicability to healthcare.13 However, due to the generalisability of the described methodology, the reviewers decided the described method was applicable to healthcare.
Method specifics
All the 11 studies described a method to prioritise recommendations following analyses of AEs, therefore differentiating between high-quality and low-quality recommendations. Four studies used categories13 16–18 and seven studies used ranking numbers to accomplish this.11 12 14 15 19–21 Nine methods were reviewed as user-friendly by the reviewers.11–19 The two methods not considered user-friendly were those that used extensive calculations in order to prioritise the recommendations.20 21 Specifics regarding the methods and their categories can be found in table 1.
Method characteristics and prioritisation of recommendations per method
Method development
Three methods were based on existing methods like the bowtie method or the failure mode, effects and criticality analysis methodology.13 18 21 Two studies did not describe where the development of the method was based on.17 19 Five methods were based on expert opinion,11 12 14–16 and in four studies, the experts involved in the development were described.11 12 14 16 The number of experts involved ranged from 3 to 57 per study. One study validated the developed method.11 More detailed information regarding the development of the methods can be found in table 2. A narrative summary of the methods is presented in online supplemental file 2.
Supplemental material
Development of the methods for prioritising recommendations
Predefined criteria
After completing the data extraction, none of the 11 found methods met all the predefined criteria as shown in table 3.
Predefined criteria met per method prioritising recommendations
User-friendly
Nine methods were considered user-friendly.11–19 The methods described by Rodriguez-Gonzalez et al and Testik et al were assessed as being not user-friendly.20 21 For both methods, comprehensive calculations were necessary, and the method by Testik et al cannot be used without extensive mathematical knowledge.
Validation
The Change Process and Outcome (CPO) scale was the only validated method.11 Validation was performed in multiple stages. For the final 20-item CPO scale, the inter-rater agreement ranged among the six pairs of reviewers from 0.53 (moderate) to 0.75 (strong), median 0.59 (moderate). The test-retest statistic on a sample of four of the single projects was 0.82 (near-complete agreement).
Prioritisation of recommendations only
Ten methods were developed to solely prioritise recommendations. The outcome of the CPO scale focussed mainly on the results of the improvement project, and less on the quality of recommendations.11
Prioritisation based on objective features
Five methods were identified that prioritised recommendations based on objective features, meaning the categories or scores were not affected by personal opinion or knowledge and the scoring could be expected to be similar when performed by different users.13 15–18
Although prioritisation of recommendations was not based on objective features, some methods have tried to minimise subjectivity; Flottorp et al used a 1 to 5 Likert scale to prioritise recommendations and suggested that at least two people need to assess the recommendations independently and discuss the outcomes afterwards.14 Rodriguez-Gonzalez et al classified the severity of the potential effect for the patient, the likelihood of occurrence for each failure mode and the likelihood of detecting failure on a 1 to 10 scale. All estimated failure modes were obtained by consensual discussions between team members in order to calculate the risk priority numbers.20
Discussion
Learning and improving healthcare based on the analysis of AEs is a multifactorial process in which every step affects the outcome.22 A grading system for recommendations following the AE analysis could have a major impact on the changes for repetition of the AE and on patient safety. It might support prioritisation of the implementation process and clinical effectiveness. The purpose of this review was to identify an appropriate—user-friendly and validated—method that prioritises recommendations following analyses of AEs in healthcare based on objective features. This systematic review identified 11 methods that are all designed to prioritise recommendations. None of the 11 methods met all the predefined criteria. The predefined criteria are essential features to improve and learn from analysing AEs. A broadly used and accepted grading system must be easy to use, objective and preferably validated.
User-friendly
Out of the 11 methods, nine were user-friendly, meaning they were easy to understand, without time consuming or complex calculations.11–19 The methods by Rodriguez-Gonzalez and Testik et al were assessed as not being user-friendly as comprehensive calculations were necessary for both methods and the method by Testik et al cannot be used without extensive mathematical knowledge.20 21
Validation
The CPO scale was the only validated method in this systematic review.11 Validation of a method should identify, and ideally eventually eliminate, the inter-user variability. It is valuable in extrapolation to other centres or fields and will aid in the implementation. For example, the method developed by Geller et al has 24 different approaches to change behaviour in a table with scores to calculate the effects.15 Furthermore, they also stated an intervention usually consists of a number of behaviour change techniques and therefore the relevant scores need to be added. This results in abundant combinations of scores. All items separately can be considered objective features, but considering the number of possible combinations, the method is prone to significant inter-user variability. Without a validation process of this method, it might be less valuable and difficult to implement. The methods by de Dianous and Fiévez, McCaughan et al and McLeod et al had a few and clearly defined categories in a well-arranged ordinal sequence.13 17 18 Even though these methods were not validated, the scoring system consists of only a few categories based on objective features, which makes these methods less prone to inter-user variability.
Prioritisation of recommendations only
The CPO scale developed by Brandrud et al focussed mainly on the results and quality of the improvement project, and less on the quality of recommendations itself, limiting its use in the prioritisation of recommendations.11
Prioritisation based on objective features
In six methods, the performance of the method was judged to be user-dependent and subjective.11 12 14 19–21 For example, some of these methods rated each intervention on a 1 to 5 Likert scale and another assessed understandability, feasibility and usefulness of each recommendation on a 0 to 10 scale.12 19 The user is supposed to rate the quality and effectiveness of recommendations using a number or grade based on their own opinion and experience. The outcome of this evaluation may vary significantly depending on the user. For example, a recommendation rated as a 4 out of 10 for user 1, might be rated a 7 out of 10 by user 2 based on their own personal experiences. Therefore, these methods are considered less appropriate to prioritise recommendations in clinical practice and for prioritisation. We believe that prioritising recommendations using categories—for example, of the bowtie method—is more objective than using a ranking by a Likert scale.
Four methods had clear categories.13 16–18 Two of these four methods were based on the bowtie method and barrier management.13 18 Although this might require a certain knowledge of these existing methods, it is a more objective way of prioritising recommendations. It will result in a more transparent prioritisation. Also, the existing methods are already widely used and have proven to be effective in other industries.
Hettinger et al created a method which places each recommendation in one of 13 predefined categories.16 Each category was assigned a specific value and was placed in a two-dimensional framework showing the prioritisation of the recommendations. This method might be a promising method for prioritising recommendations as the categories are clear and objective. A drawback of this method is that the values assigned to the categories were assigned in a subjective manner. Through interviews with front-line staff in which the staff was asked to rate each category, the value of each category was determined. Without validation, the method might therefore be less suitable for extrapolation to other circumstances or other settings.
Methods best suitable for quality improvement in clinical practice
Five methods meet three out of the four predefined criteria and might therefore be considered the best suitable methods for quality improvement in clinical practice.13 15–18 However, as stated previously, the ‘Taxonomy of behaviour change strategies to guide intervention development and evaluation’ by Geller et al might be more prone to inter-user variability without being validated, and the ‘Model of sustainability and effectiveness in root cause analysis solutions’ described by Hettinger et al is less objective than the other methods.15 16 Although the ‘Summary of the relationships between components of a barrier system’ by McLeod et al has clear categories for the prioritisation of recommendations, these categories must also meet some other criteria, making the method more complicated than the ‘Typology of safety functions’ described by de Dianous and Fiévez, and the ‘Hierarchy of hazard controls’ by McCaughan.13 17 18 The methods by de Dianous and Fiévez, and McCaughan have the most potential to select high-quality recommendations. They have few and clearly defined objective categories in a well-arranged ordinal sequence, which makes them user-friendly. Furthermore, the inter-user variability is expected to be limited.
Limitations
A limitation of this study was the fact there are many different terms that are used for this specific topic. There is little validated research and the lack of MeSH-terms makes it difficult to identify studies in general. Documents and guidelines available to healthcare workers involved in incident analysis, which also address recommendations and which are often only available nationally or even regionally, might not have been retrieved. Even though, our literature search was extensive and in all relevant databases.
No appropriate quality assessment tool was available for this type of research. There are only a few available quality assessment tools for qualitative research. Unfortunately, none of these would be appropriate for all studies included in this review as they are original research papers as well as derived from a guideline and a textbook.17 18 In addition, we believe that the quality assessment does not necessarily reflect the quality of the method for prioritising recommendations.
Although successful implementation of recommendations is essential, this was considered beyond the scope of this review. Studies solely regarding implementation were therefore excluded. An appropriate method facilitates selecting high-quality recommendations for implementation. Implementation of high-quality recommendations will probably decrease the number of (similar) AEs, serving the goal of improving the quality of healthcare.
Learning from AEs is only a limited part of improving healthcare as a whole. AEs often result from the daily variations in our processes. Understanding this variation is an important factor in evaluating AEs. A proactive approach and learning from best practices are at least as important. It remains important to evaluate and repair safety gaps in our system if possible. For the repair of these gaps, the completion of the process of learning is essential: reporting AEs, analysing AEs, formulating recommendations, implementing recommendations and evaluating the effect of recommendations.22 Underperformance or omitting one of the steps will render the complete process useless.
Future perspective
The difficulty to translate the knowledge and experience of experts to quantitative data is reflected in the diverse methods used in the different studies. Ideally, there would be a validated, user-friendly method to prioritise recommendations objectively. Combining the experience of healthcare workers and the knowledge of experts specialised in analyses of AEs in other safety-critical industries, holds great potential in creating a solid method that facilitates prioritisation of recommendations in healthcare. A method that would enable users to select the high-quality recommendations for implementation and give insight in factors determining the quality of recommendations following analyses of AEs, would make repetition of AEs in healthcare impossible or less likely.
Conclusion
There are several methods available to prioritise recommendations following analyses of AEs. All these methods can be used to discuss and select recommendations for implementation. None of the methods is a user-friendly and validated method that prioritises recommendations based on objective features, despite this being an essential feature to improve and learn from analysing AEs. Although there are possibilities to further improve their features, the ‘Typology of safety functions’ by de Dianous and Fiévez, and the ‘Hierarchy of hazard controls’ by McCaughan et al have the most potential to select high-quality recommendations as they have only a few clearly defined categories in a well-arranged ordinal sequence.13 17 Ultimately, selecting high-quality recommendations for implementation might lead to a decrease in the number of (recurrent) AEs, serving the goal of improving the quality of healthcare.
Acknowledgments
The authors thank Faridi van Etten-Jamaludin, medical information specialist, for her assistance in the literature search strategy, and Gert-Jan Kamps and Alfred van Wincoop, human factors consultant and engineer at Intergo Human Factors and Ergonomics (Utrecht, the Netherlands), for their assistance in delivering eligible studies from other high-risk industries.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Study conception and design: KB, MJvdL and DAD. Acquisition of data: KB, MJvdL and DAD. Analysis and interpretation of data: KB, MJvdL and DAD. Drafting manuscript: KB. Revising it critically for important intellectual content: KB, MJvdL and DAD. All authors approved the final version of the manuscript.
Funding This work was supported by the Quality-based Governance programme and is funded by the Citrien foundation. This foundation supports the development of sustainable and broadly applicable solutions in healthcare and is made possible by ZonMw.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer-reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.