Article Text
PDF
Statistics from Altmetric.com
- human factors
- adverse events
- epidemiology and detection
- safety management
- root cause analysis
- patient safety
Introduction
Adverse events in healthcare represent a major source of morbidity and mortality and result in substantial societal costs.1–4 Hospitals in numerous countries, such as the Netherlands, the USA, Canada, Australia and the UK, aim to learn from previous events by thoroughly investigating these. In particular, hospitals invest in studying the events that have caused temporary or permanent disability, death or prolonged hospital stay, also known as sentinel events or serious adverse events (SAEs).5–10 After hospitals investigate a SAE, they write a detailed report. This report starts with a comprehensive reconstruction of the event and concludes with a root cause analysis (RCA), in which hospitals search for root causes and formulate recommendations to prevent recurrence.11 SAE investigations and the associated reports offer great potential for hospitals to learn from each other and improve patient safety. However, SAE investigations and analyses are being criticised for oversimplification,12 resulting in weak solutions,10 11 13 and overall ineffectiveness.11 13–15 Two important shortcomings impede hospitals in particular in their ambition to learn from SAEs.
First, SAE investigations often neglect the complexity of healthcare.13 15 Current methods applied for SAE analysis, focus on finding one linear root cause, even though SAEs are more likely to arise from interactions and combinations of causes and contributing factors in the complex healthcare system.10 13 14 SAE analysis thus needs a shift from focusing on finding the ‘one’ linear root cause towards searching for the combination of interrelated contributing factors and causes. The incorporation of human factors thinking in SAE analysis is assumed to induce this shift.16–20 Human factors is concerned with the understanding of interactions and interdependencies between humans and other elements of a work system.21–23 It searches for opportunities to design healthcare systems that have a greater tolerance of faults and thus might improve the resilience of the system.24–26 One model that integrated principles of human factors into the healthcare domain is the Systems Engineering Initiative for Patient Safety (SEIPS).27–32 This theoretical model is used to study patient safety hazards and adverse events in various healthcare settings.16 30–35 SEIPS is based on the well-known structure–process–outcome model for healthcare quality.36 It therefore is assumed to be familiar to audiences working in healthcare quality and patient safety.31 Although the potential of SEIPS to improve patient safety is proven,31 adoption of the model in retrospective SAE analysis is still limited and should be refined and accelerated.16 30
A second shortcoming of current SAE investigations is that hospitals typically focus on single events within their own organisation,13 14 even though attempts to learn from aggregate analysis of multiple events across hospitals, as already performed in Australia37 and the USA,38 39 are believed to improve learning.38–41 It helps to discover combinations of recurring, underlying, patterns of causes and contributing factors42 and may improve the formulation of more effective, system-aimed recommendations.14 38 However, methods for analysing SAEs vary substantially,10 which complicates aggregate cross-hospitals analysis of SAEs. For example, in the Netherlands, where aggregate analysis of SAEs is scarce, hospitals use three differing methods for analysing SAEs41 43 44: Prevention and Recovery Information System for Monitoring and Analysis (PRISMA-medical),45 Tripod Beta46 and Systemic Incident Reconstruction and Evaluation (SIRE).47 These methods are useful for a structured analysis of SAEs, yet based on their description in scientific articles,41 44 48 49 anecdotal evidence43 46 47 and experience, each method has its own approach, focus and limitations. PRISMA-medical, for example, is highly analytical and categorises latent (technical and organisational) as well as active (human) failures.45 However, the absence of organisational or technical barriers that could prevent SAEs are not considered a root cause in this method.50 Tripod Beta, however, thoroughly investigates system failures and organisational barriers that could prevent SAEs. However, human errors are also explained as organisational or system failures,46 49 even though evidence indicates that flaws in the system and cognitive factors play an important role in the emergence of human error.51–53 SIRE, in turn, emphasises particularly on the primary process and provides a comprehensive narrative description of the event.47 54 To ease aggregate cross-hospital analysis of SAEs, Smits et al43 made a primary attempt to integrate the structures and foci of PRISMA-medical, SIRE and Tripod Beta into a generic framework. Anecdotal evidence of the benefits of this framework to perform an aggregate cross-hospital analysis was presented, but the study emphasised the importance of further development and evaluation.43
Addressing the complexity of healthcare13 and using a consistent method10 41 could thus improve learning from SAEs. This could help to formulate more effective recommendations to enhance patient safety. An alternative approach to study SAEs is therefore necessary. This study aims to develop a novel generic analysis method (GAM) that integrates the SEIPS model31 32 and the PRISMA-medical, SIRE and Tripod Beta framework43 to: (1) facilitate a more holistic analysis using a human factors perspective and (2) ease aggregate analysis across hospitals.
Methods
We followed a structured process (figure 1) of developing, evaluating, testing and refining prototypes of a GAM.
Overview of the steps taken in developing the Generic Analysis Method. PRISMA, Prevention and Recovery Information System for Monitoring and Analysis; SIRE, Systemic Incident Reconstruction and Evaluation; SEIPS, Systems Engineering Initiative for Patient Safety.
Description of the prototype
A first prototype of the GAM was developed based on the theoretical SEIPS model31 32 and the framework43 combining PRISMA-medical, SIRE and Tripod Beta. This prototype included three main pillars. The first pillar addresses the gathering of basic SAE information. Second, each sociotechnical domain of the SEIPS model is evaluated. All relevant characteristics should be collected about the: (1) persons involved, (2) tasks performed, (3) technologies used, (4) organisational factors affecting the work, (5) the physical environment, and (6) the external environment. In this evaluation, we also adopted some important elements of PRISMA-medical, SIRE and Tripod Beta as accumulated by Smits et al43 related to these six domains. The third pillar focuses on the outcomes of the SAE for the involved patients and/or their family, healthcare professionals and the organisation. This part also includes the evaluation of the recommendations formulated in response to the event.
Evaluating and testing the prototype
We started with consulting two theoretical experts (A), both academics in the field of patient safety, to evaluate the first version of the prototype and ask for potential additional theoretical concepts that should be embedded.
We then used the prototype for a pretest session (B), in which the author (MCB) analysed six SAE reports covering the variation of RCA methods used by Dutch hospitals (PRISMA-medical, SIRE and Tripod Beta). These reports were based on thorough SAE investigations performed by independent committees of various hospitals in 2018 or 2019. The multidisciplinary hospital committees investigating the event, typically consisted of a combination of clinicians and quality and safety officers. Committees combined multiple sources for studying the event. The most important source being in-depth interviews with involved healthcare personnel and patients and/or family. Additionally, the committees reviewed all relevant information such as patient records, test results (eg, medical images, laboratory findings, etc) and documentation (eg, guidelines, work instructions, etc). The combination of interviews with the persons involved and reviewing all relevant documentation resulted in an extensive and detailed reconstruction of the event. After the event description, the hospital committees performed an RCA and formulated recommendations to prevent recurrence. The comprehensive data in the SAE reports was suited for a retrospective (re)analysis of the event and, hence, a first test of the prototype’s content and face validity. By including a sample covering all three RCA methods, we could test whether our prototype was applicable irrespective of the initial RCA method used. This step contributed to improving the usability of the prototype, for example, the prototype could be aligned with the working methods of the hospitals.
Next, a panel with experts from practice (C) was consulted for a thorough evaluation of the face and content validity of the prototype during a face-to-face meeting. This panel consisted of a mix of quality and safety officers (n=8) and medical specialists with direct experience in patient safety (n=5). All of them were currently practising in various Dutch general hospitals (n=12). Minutes were taken during the meeting, and the transcribed notes were sent for verification to the participants afterwards.
Subsequently, two potential users, both quality and safety officers, tested the prototype (D). Just as in the pretest session, information-rich SAE reports were used. The users analysed two distinct reports, written by two general hospitals in 2018. This test session was meant to further evaluate content validity of the method as experienced by potential users. It also helped to find flaws and practical barriers and provided the possibility to observe users’ interactions with the prototype. Commentaries from the respondents were used to refine the usability. In addition, the author (MCB) analysed the same two reports. This allowed a rough assessment of the agreement in analyses between the potential users and the researcher. As the most important questions were open ended, a qualitative approach on assessing the agreement between raters was performed. Overall agreement on all questions was reviewed by eye assessment, and differences between the interpretation of raters for the important questions were reflected on (eg, by considering: did raters give different answers? Why did raters not answer that question the same way? Were there random outliers, or did it show a systematic difference in interpretation?).
Patient and public involvement
Since the method is specifically intended for use by healthcare professionals to learn from SAEs across hospitals, professionals (potential users) were consulted at multiple stages during the study. Patients were not involved in developing the method as they currently have no formal role in performing (aggregate) analysis of SAEs. Nevertheless, it is important to involve patients and their family in the SAE investigation by including their vision on the event through interviews and giving them insight into the findings.
Results
Evaluation and test results
In the first step of evaluation, theoretical experts (A) stressed the importance to assess the element of human error in SAEs and therefore referred to the model of unsafe acts.52 This model helps to differentiate between various types of human errors (eg, violations, mistakes, slips and lapses). This could provide a more in-depth understanding of these errors. Another concept the experts recommended to include were the efficiency–thoroughness trade-off principles. Evaluation of the trade-offs between working efficiently and working thoroughly, in this context safely, might enhance our understanding of safety critical decisions being made in complex healthcare situations.55 As these trade-offs reflect situations in which there is a conflict between, for example, production pressure and providing safe and high quality care, they may raise awareness to more system-related issues.
After pretesting the prototype (B), questions and answer categories were reconsidered and rewritten in order to make it more compatible with the information gathered in SAE investigations. Test cases also gave a useful overview of the SAEs and helped to identify relevant characteristics, contributing factors and root causes related to each of the sociotechnical work system domains. Examples of these findings are listed in table 1.
The discussion with experts from practice (C) resulted in a few additional topics that were particularly relevant and therefore needed to be explicitly adopted (eg, the identification and classification of fragility among elderly patients). Also, more practical issues were put forward such as the description and categorisation of clinical areas and age groups. The boundaries of external factors, such as financial restrictions or labour shortages, were argued and established as well.
The final test phase by potential users (D) brought up some important modifications of the questionnaire (eg, textual revision of questions, adjustment or addition of answer categories and merging some of the questions). This test session also presented important misinterpretations of a few questions and answer categories. Overall agreement between the potential users and researcher was acceptable, with exemption of the misinterpreted questions. We decided better instructions were needed and composed an explanation file and example case in which the ambiguities were explained to help future users in interpreting the questions and answer categories as intended.
Analysis of the event reconstructions using the prototype helped in systematically considering all aspects of the sociotechnical work system. This resulted in a holistic analysis of the event. Mapping contributing factors and causes according to the prototype assisted in evaluating interactions and interdependencies between factors of various domains, addressing the complexity.
Description of the final GAM
The described process of developing, evaluating, testing and refining prototypes resulted in a final version of the GAM, consisting of a framework (figure 2) and an affiliated questionnaire (online supplemental appendix 1). Both follow a basic structure for analysing SAEs.
Supplemental material
The first step of the model only intents to gather basic information on what happened. A suggestion of the most important elements that could be included is presented in the upper part of figure 2.
The method then evaluates the six domains of the work systems according to the SEIPS model. Each domain includes a description of relevant characteristics and potential contributory factors. Interactions and interdependencies between domains and factors can be mapped to address and visualise complexity. Online supplemental appendix 2 provides an example of a case for which the work system was analysed using the GAM. In the tasks element, we included a question to get insight in unsafe human acts52 contributing to the event. For the tasks and organisational elements, trade-offs are evaluated between efficiency and thoroughness. Table 1 specifies the six domains of the work system with a more detailed description and provides some examples, based on the test sessions and previous case studies.
Supplemental material
After analysing the work systems and processes, the consequences of the SAE must be considered. It is important to assess the effects on patients and family, as well as caregivers, other professionals and the healthcare facility. Outcomes of SAEs can be multidimensional and may affect the physical, psychological, juridical, social and financial situation of all those involved. When evaluating the impact of SAEs, both immediate as well as distant outcomes must be considered. Formulated recommendations are incorporated in this part as well, so evaluation of the type(s) of recommendations as suggested by the hospital in response to the SAE is possible.
Discussion
Previous studies highlighted issues that hamper hospitals in their ambition to learn from SAEs. Methods used for analysing events, for example, have a strong focus on finding a linear root cause, thereby neglecting the complexity of healthcare.13 14 Hospitals also often study single events within their organisation, instead of analysing multiple events across organisations.13 Substantial variety in methods applied might complicate attempts to perform such aggregate cross-hospital analysis. Although potential solutions such as integrating human factors thinking in SAE analysis and using a more consistent analysis method have been proposed,10 18 20 41 there remains a need for methods embracing these possible improvements. This study aims to develop a novel GAM that stimulates a holistic analysis using a human factors perspective and eases aggregate analysis across hospitals.
A GAM prototype was developed based on a human factors model and integrating important elements of three currently practised methods for analysing SAEs. The paper describes the multiple steps of carefully evaluating, testing and refining prototypes. Suggestions of theoretical experts to complement the prototype were incorporated. Pretesting, evaluation by experts from practice and user test sessions, helped to improve the usability and face and content validity. User test sessions also allowed us to roughly assess inter-rater agreement and think of manners to improve future reliability. These steps resulted in a final version of the GAM, consisting of a framework and affiliated questionnaire. These assist in gathering structured information about the event, evaluate all domains of the sociotechnical work system, map interactions between factors from various domains and assess outcomes of the SAE.
The SEIPS model on which the GAM is based helps to focus beyond hierarchical, linear and causal relations and stimulates evaluation of complexity. To successfully unravel the complexity of healthcare work systems, it is essential to understand the interactions and interdependencies between the persons involved and the other domains of the sociotechnical work system (eg, technology, tasks, organisation and environment). As described by Holden et al,31 any analysis of the work systems should, therefore, include three important concepts: (1) the sociotechnical work systems must be viewed as dynamic, hierarchical and interactive, (2) all individuals engaged in a process or work system should be mapped systematically to reveal social structures and (3) analysis must depict how dynamic systems evolve over time in intended and unintended ways. As exemplified in online supplemental appendix 2, the GAM helps to evaluate the dynamics and interactions within a work system. By mapping all domains of the work system including the persons involved in a structured manner, it might be possible to depict the complex properties of SAEs. Whenever more SAEs are systematically mapped this way, underlying patterns of recurring interactions and structures might emerge. The inclusion of the trade-offs between efficiency and thoroughness,55 as adopted in the organisational and task elements of the GAM, could provide an opening to compare actual (work as done) and expected work practices (work as imagined). This could help to understand how work systems have evolved over time in intended and unintended ways.
In addition to incorporating human factors thinking, GAM combines the narrative approach of SIRE47 with the more analytical approach of PRISMA-medical45 and Tripod beta46 on both organisational barriers and human-related issues. Integrating important elements of these methods helps to perform a holistic analysis. At the same time, the combination of existing methods in one generic method might ease aggregate analysis of SAEs across hospitals.
As GAM is structured according to the widely used SEIPS model30 31 56 and built on methods that are currently applied by hospitals, we believe GAM may be easily adopted. Other promising methods that aim to improve learning from RCA such as the Human Factors Analysis and Classification System,57 AcciMap58 and Functional Resonance Analysis Method59 were not specifically build on the well-known SEIPS model and existing methods. Other methods may also need a specific practice to investigate SAEs or tend to focus specifically on a small and delineated process. This might make them less easy to adopt for SAE analysis. Existing initiatives for aggregate analysis, such as those performed in the USA and Australia37–39 and the preliminary attempts in the Netherlands,41 43 lack the important human factors perspective to guide analysis. Aggregate analysis using the GAM might therefore lead to better understanding of SAEs and could result in more effective system-aimed solutions. Besides retrospective analysis of patient safety events, methods such as the Health Care Failure Mode and Effect Analysis60 61 and Tripod Delta HC62 intent to prospectively identify (latent) risks for patient safety. These methods remain important and can be used complementary to retrospective aggregate analysis of SAEs using the GAM. Outcomes of retrospective GAM analysis might help to identify risks for such prospective analysis.
In its most basic application, GAM can help to perform a structured, holistic analysis of SAE reports from various hospitals, irrespective of the initial RCA method used. The method is thus suited to study the existing ‘treasure’ of SAE reports that is stored in the archives of hospitals and unleash its potential to jointly learn from SAEs. Considering GAM was developed in the Netherlands and integrates methods Dutch hospitals use for SAE analysis, the generic method is particularly suited for application in the Dutch context. However, PRISMA-medical, SIRE and Tripod are not solely used by Dutch hospitals. In particular PRISMA-medical is applied by hospitals in various other countries as it has been accepted by the World Alliance for Patient Safety of the WHO.63–65 At the same time, we believe GAM is also appropriate to stimulate and improve aggregate analysis of SAEs for hospitals currently using other methods, such as cause and effect diagrams, five whys and fishbone diagrams. These methods are after all also based on the principle of investigating single SAEs and a search for a root cause. Application of the GAM might in this case be more time consuming though.
Limitations
A limitation of the study was the modest user test session, as it was limited to two cases. The process of testing was highly time consuming for potential users. Although we performed other testing and evaluation steps, including a pilot phase in which six more reports were analysed by the author, the number of cases in the user test session can be considered as a weakness. More extensive testing is therefore needed to further assess reliability and validity. Further testing might also provide leads to lean the method and improve its usability.
Another issue we should address is that we tested the GAM by applying it to reanalyse information gathered in SAE reports. Although the reports provided a thorough description of the event, the investigation did not specifically study the event according to the GAM. This might have made it difficult to subtract new or other relevant factors from the reports than those already found in the initial RCA. The test sessions, however, indicated that adopting the GAM perspective to reanalyse information as included in the event reconstruction of the SAE reports can still lead to new insights. Directly studying an event using the GAM enables investigators to ask specific additional questions during the interviews with those involved on subjects that are only limitedly covered in traditional SAE investigations and analyses. Based on this study, we cannot draw strong conclusions on the usability of GAM for directly studying a SAE, but given its value in the analysis of SAE reports, the method seems promising for this purpose as well.
Conclusion
A novel GAM was carefully developed. On balance, the method is a promising tool for taking a next step in learning from SAEs. It facilitates a holistic analysis of the SAE and addresses the complex sociotechnical work system in which SAEs in healthcare originate. GAM could also ease aggregate analysis of SAE across hospitals. Using GAM might empower hospitals in their ambition to learn from SAEs and could help to formulate more effective system-aimed recommendations to prevent recurrence of events. The method can now be used to analyse a large sample of events to further explore its potential to improve patient safety.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. Not applicable.
Ethics statements
Patient consent for publication
Ethics approval
Evaluated by the Medical Ethics Review Committee of VU University Medical Center (FWA00017598) and considered as not subject to the Medical Research Involving Human Subjects Act (WMO).
Acknowledgments
The authors would like to thank the professionals that participated in evaluating and testing the prototype. We would also like to thank the hospitals that shared their anonymised serious adverse event reports for test sessions.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors MCB, SMVS and CW jointly conceived the study concept and design. A first prototype of the analysis method was developed by MCB. SMVS and CW provided feedback on this prototype. MCB coordinated and conducted the data gathering process of evaluating, testing and refining prototypes, supported and supervised by SMVS and CW. MCB presented the prototype to the theoretical experts and performed the pretest session. During the meeting with the expert panel, MCB presented the prototype and discussed issues of face and content validity. CW participated in the discussion session. The double analysis in the user test session was performed by MCB, as well as the qualitative assessment of the inter-rater agreement. In all stages of the study, authors discussed findings and potential adaptations to the prototype. MCB wrote a first draft of the manuscript with input of SMVS. SMVS and CW gave feedback multiple times after critical revision of the manuscript. MCB is the guarantor and accepts responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding The study was funded by the Dutch Ministry of Health, Welfare and Sport. No grant/award number.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.