Article Text
PDF
Abstract
Background There is limited engagement in healthcare with the kinds of proactive approaches to risk assessment used in other industries. Bowtie analysis (BTA) has previously been shown to have potential as a straightforward approach to proactively assessing risk in healthcare. The visual nature of BTA diagrams can aid communication of the essential elements of a complex risk management system. The aim of this small case report study was to investigate the training and support likely to be needed for existing healthcare professionals to conduct BTA in compliance with recognised industry best-practice.
Method Of 17 volunteers who attended training, 3 completed an analysis of significant healthcare risks in the study period: misadministration of gentamicin; unknown development of acute kidney injury and disposal of medical devices containing patient identifiable information (PII). Subjective assessments of the quality of the analyses were made against indicators of BTA best-practice.
Results Use of the BTA method led to a deeper understanding of the issues and a more thorough understanding of the risks and what was needed to control them than would have been the case if ‘normal practice’ had been followed. Classroom-based training supported by written guidance; however, do not appear adequate to support development of competence to carry out a quality BTA in a healthcare setting.
Conclusions BTA seems to have potential though further evaluation of its application and utility is necessary. The most cost-effective and productive approach is likely to be to train a small number of people to develop deeper skills and experience in BTA. In addition to training and user guidance, the opportunity to facilitate at least one analysis, with some specialist/trainer support, appears to be essential in developing BTA competence.
- risk management
- safety management
- human factors
- patient safety
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. No datasets were generated in the study,
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
The reported levels of preventable patient harm in healthcare,1 reflects a lack of both systems-based thinking and analysis,2–6 as well as the kind of formal scrutiny of the controls relied on to protect against major risk that are routinely applied in most other safety-critical industries. Despite growing interest in prospectively identifying and managing risk to patient safety,7 there remains limited engagement with proactive risk assessment approaches in healthcare. Evidence of the utility of available methods is also lacking.
Bowtie analysis (BTA) is widely used in many high hazard industries as a means of identifying and understanding how risks of major adverse events are managed and controlled.8 9 Its popularity derives largely from the visual representation used to summarise the results of an analysis. The visual nature of the diagrams can be powerful in aiding communication of the essential elements of a complex risk management system.
BTA is based on the concept that adverse events occur when the controls relied on to manage energies or situations with the potential for harm fail to do what is expected of them. The analysis involves identifying the controls relied on to prevent adverse events and the potential consequences in terms of actual harm or loss (figure 1). The quality and effectiveness of those controls is assessed and factors capable of defeating or degrading them are explored. Some authors (eg, Levison10) have criticised BTA for relying on a linear model of accident causation, which maybe unsuited or have limited impact in complex sociotechnical systems like healthcare. In reality, no assumptions need to be made about the underlying failure mechanisms and related complex interactions.8 11
Conceptual summary of elements in a bowtie analysis (for definition of terminology see Chartered Institute of Ergonomics and Human Factors9).
An exploratory study,11 based on a primary care ‘never event’, concluded that BTA could provide a straightforward approach to engaging frontline healthcare practitioners and managers in proactively assessing risk. A variety of other studies have reached similar conclusions.12–17 Though other than Mullins et al,17 few have applied the method rigorously.11 It is not clear however if BTA is appropriate for use in a healthcare context by non-experts; in particular, concerns remain over the level of training, support and resources required to carry out an analysis. An essential requirement for any proactive risk analysis method expected to be used widely is that an analysis can be conducted reliably, to an acceptable standard, without excessive demand on time and resources and without having to rely on external facilitators.
Based on feedback and learning from,11 draft guidance was prepared on how to conduct BTA in a healthcare setting.18 The guidance is intended for use by anyone tasked to lead a BTA (the ‘BTA Lead’). No expectations are made about their clinical, managerial or technical background, competence or experience. Though the individual does need to possess good non-technical abilities, such as being organised, able to think analytically and having good facilitation skills. The 5-stage process set out in the NES guidance (see figure 2) is summarised in NHS Education for Scotland.18
Summary of five stages in NES guide to conducting bowtie analysis (BTA) (NHS Education for Scotland18). (Note: ‘PA’ in Stage means ‘primary adviser’).
Objective
The study was designed to investigate the amount of training and support needed for existing healthcare professionals to conduct BTA in compliance with recognised industry best-practice,8 9 using the NES process as guidance, together with email and telephone support from a BTA specialist (RM).
Method
Recruitment and training
Volunteers to take part in the study were recruited from an established NHS Scotland community of practice network (http://www.knowledge.scot.nhs.uk/hfe.aspx). Seventeen individuals, representing eleven organisations attended a 4-hour face-to-face training session after which they were invited to initiate an analysis within their own work setting. Of the 17, 7 initiated an analysis, though only 3 actually completed an analysis within the study period.
Interim reviews
Approximately 6 weeks after initiating each of the analyses, a 1-hour face-to-face progress meeting was held between the NES BTA advisor (RM) and each of the BTA Leads. The purpose was to review progress and ensure the BTA concepts and terminology were being applied correctly.
Use of Bowtie software
A variety of commercial software tools are used to support BTAs in industrial settings. Concerns have however been raised that ‘… organisations frequently constrain their thinking, and the depth and breadth of their analysis, around what can conveniently be represented on current computer monitors. This is a practical constraint based on convenience and can have little to do with the reality of the risks involved …’.9
To try to overcome this constraint, as well as a widespread tendency to draw diagrams too early in an analysis. the NES BTA guidance,18 includes a series of tables to encourage analysts to focus on understanding the threats, consequences and controls before attempting to summarise how risks are controlled in the form of a bowtie diagram.
Recognising that the process of drawing, manipulating and using bowtie diagrams can be prohibitively time-consuming, study participants were provided access to evaluation copies of the commercial software package BowtieXP,19 from the time of their interim review. Functionality used was limited to the ability to create and manipulate the diagrams.
Quality evaluation
On completion of the case reports, assessments were made of the quality of the completed analyses. Quality assessments were made by the BTA advisor (RM) reflecting the extent that each of the criteria shown in table 1 were considered to have been met. Each rating used a 5-point scale with the following meanings:
Quality criteria used to assess bowtie analysis (BTA) studies
0: ‘satisfied that the criterion was not met at all’.
1: ‘inclined to be dissatisfied that the criterion was met’.
2: ‘undecided’.
3: ‘inclined to be satisfied that the criterion was met’.
4: ‘satisfied that the criterion was met in full’.
No attempt was made to apply differential weightings to the relative importance of each of the criteria shown in table 1.
Results
Study participants
Three analyses were progressed to the point where they were considered ‘complete’ by the end of the allotted study period. The following adverse event topics were the subject of these BTAs:
Misadministration of the antibiotic gentamicin.
Unknown development of acute kidney injury (AKI).
Disposal of medical devices containing patient identifiable information (PII).
Table 2 summarises details of the three individuals assigned as BTA Leads for each of these analyses.
Details of bowtie analysis (BTA) Leads assigned to each of the studies
Case reports
A: misadministration of gentamicin
Gentamicin is potentially toxic, while misadministration can impact on both efficacy and toxicity.20 Risk of error in the case report hospital seemed to arise from two main sources:
Unlike most other medications used in adults where fixed dosing regimens exist, gentamicin requires both individual dosing adjustment and level monitoring. Dosing and administration intervals are calculated using the patients’ accurate weight and blood tests of renal function.
An electronic prescribing system is used for the majority of medication management. By contrast, gentamicin not only requires data to be manually input to an online electronic calculator for initial dosing, but then requires transcription of the dose and interval (as well as subsequent levels and dosing plans) onto a paper prescription chart. Additionally, the plan then needs to be manually recorded onto the electronic system. The majority of medication administrations rely on prompts from the electronic system, though the varying processes expose risk of transcription errors as well as missing information.
Although it is complex and there are not infrequent incidents, when correctly administered gentamicin is very effective: it is not practical simply to withdraw it from use due to the associated risks.
Analysis process
Case report A was led by a medical consultant with 15 years experience working in an acute unscheduled care setting in a large hospital. In total, the consultant spent around 10 hours working on the analysis over a period of 3–4 months.
Although experienced in incident reviews, the consultant had no formal training or experience using structured approaches to risk analysis beyond using risk assessment matrices to prioritise risks. The consultant was well informed about the risks and difficulties associated with gentamicin and had opinions on how current practice needed to change to reduce those risks.
At the time of the interim review, the analysis had not followed the five-stage process recommended in the BTA guidance and had not made use of the tables provided to structure thinking and evaluation. Following the review, the consultant followed the guidance more closely. The amended analysis was then used as the basis for creating diagrams in BowtieXP.
Results
Figure 3 summarises part of the analysis of the misadministration of gentamicin. It shows the hazard, adverse event and potential consequences, as well as the specific threat analysed. The figure also indicates the nature of the five controls relied on to prevent the risk of incorrect administration of gentamicin (see Chartered Institute of Ergonomics and Human Factors9 for details of these control types):
Two full barriers.
One barrier requiring two elements to deliver the required capability.
Two key safeguards.
One safeguard.
Extract from case report A: degradation factors and safeguards identified for the barrier element ‘Nursing process for medication administration’ (for definition of terminology, see Chartered Institute of Ergonomics and Human Factors9). (Note: “HEPMA” means “Hospital Electronic Prescribing and Medicines Administration”).
Figure 3 also shows examples of degradation factors that could defeat the barrier element ‘Nursing process for medication administration’ as well as safeguards expected to protect that element from degradation.
Although the analysis only dealt with the left-hand side of the bowtie, the consultant felt the analysis of the risks associated with gentamicin was sufficiently thorough for the hospital’s needs. While there was scope to conduct a more comprehensive analysis, she believed they had achieved an adequate appreciation of the scope of risks and controls involved, how they work, as well as how they could be made more robust.
Evaluation
Assessment of the quality of the BTA for gentamicin administration gave an assessed quality score of 78%.
B: unknown development of AKI
Acute Kidney Injury (AKI) has an extremely high mortality rate: it has been estimated that more than 20% of patients with AKI will die during hospital admission, rising to >35% in those with AKI stage 3,21 and is understood to cost the NHS in the order of £620 m per annum.22 Up to 30% of cases are thought to be preventable.23 Combined with an external infection, factors such as increasing age and frailty, polypharmacy and history of AKI are all risk factors.
It is however rare for a patient to go into hospital with suspected AKI: up to 60% of cases are believed to start before the patient is admitted to hospital. A patient will typically be admitted with something else, and only later be diagnosed with AKI.
Analysis process
The analysis was conducted in a general practice surgery with the practice business manager in the role of BTA Lead, and a practice general practitioner (GP) as clinical advisor. Neither had attended the arranged BTA training, although both had attended face-to-face training with the BTA advisor prior to initiating the analysis. Representatives from a cluster of six local GP practices (supporting a community of c50 000 patients) provided data and supporting information.
The business manager spent approximately 50 hours working on the analysis over a period of 4 months. The clinical advisor spent up to 30 hours. As with the gentamicin study, the initial analysis did not initially strictly follow the recommended five-stage process and did not make use of the tables provided to structure thinking and evaluation. Following the interim review, the BTA Lead used the tables included in the NES guidance to structure and evaluate the information gathered. The amended analysis was then used as the basis for creating a diagram in BowtieXP.19
Results
On completion of the study, the BTA Lead felt the analysis was sufficient for the cluster’s needs (although the analysis of the right-hand side of the bowtie had not been completed). Figure 4 shows the left-hand side of the top level of the analysis, comprising the hazard, adverse event and two specific threats as well as the controls relied on to block the threats from leading to the adverse event (see Chartered Institute of Ergonomics and Human Factors9 for definitions).
Extract from case report B: controls identified to block two threats from leading to AKI (for definition of terminology see Chartered Institute of Ergonomics and Human Factors9). AKI, acute kidney injury. (Note: “IDL” means “Immediate Discharge Letter”; “EDL” means “Electronic Document Transfer”).
All but one of the controls were assessed as meeting the criteria to be full barriers.
One barrier was identified as comprising two elements.
One control was identified as a ‘key safeguard’.
Evaluation
Assessment of the quality of the BTA for AKI gave a score of 88%.
C: disposal of medical devices containing PII
Decommissioning medical equipment containing information identifiable as belonging to specific patients (Patient Identifiable Information, PII) would represent a breach of the UK General Data Protection Regulations (GDPR). Recent changes in the GDPR include potential for large financial penalties in the event of the regulations being breached.
Medical devices have become more complex and more software-driven over the last decade or so. They can now store a range of information about a patient’s medical history, including the patient’s name, imagery, test results and so on. This PII is covered by GDPR and should never get into the public domain. Prior to decommissioning, the medical physics department are responsible for ensuring the data storage on every device is clean of all PII.
In the previous year, the department had experienced two incidents of data breaches where medical devices were disposed while still containing PII. This had led to a need to amend the processes and to understand why the department was failing to identify all equipment containing PII before it was decommissioned.
Analysis process
Case study C was carried out in the Diagnostics in Healthcare Sciences department of a large healthboard. The analysis was led by a health physics technical manager with 18 years experience, supported by a second technical manager and the department’s quality manager. The BTA Lead had been trained on risk-based maintenance and had awareness of the use of risk assessment matrices to proactively assess risk.
The analysis was conducted over a period of 4 months. In total, the technical manager spent around 29 hours working on the analysis. The two supporting managers spent about 17.5 hours each.
As with both of the previous case studies, the interim review identified a degree of initial misapplication of some of the BTA concepts and terminology. Following the review, the BTA Lead reverted to using the tables included in the NES guide to structure and evaluate the information. The amended analysis was then used as the basis for creating a diagram in BowtieXP.
Results
Figure 5 shows the adverse event and three identified threats that could lead to its’ occurrence. The figure shows the controls relied on to block each of the threats from leading to a failure to identify PII at the point when equipment is being prepared for decommissioning (see Chartered Institute of Ergonomics and Human Factors9 for details of these control types). Figure 5 also shows two degradation factors and their safeguards.
Extract from case report C: controls, degradation factors and safeguards protecting against threats leading to failure to identify patient identifiable information (PII) on medical equipment prior to decommissioning (for definition of terminology see Chartered Institute of Ergonomics and Human Factors9). (Note: “SOP” means “Standard Operating Procedure”).
Note that in this analysis, none of the controls were considered sufficiently robust to be treated as either barriers or key safeguards according to the criteria defined in Chartered Institute of Ergonomics and Human Factors9: all were considered safeguards.
Evaluation
Assessment of the quality of the BTA for the disposal of medical equipment containing PII gave an assessed quality score of 82%.
Discussion
This small study appears to have been the first to provide training and support in BTA specifically targeted at healthcare professionals, and then to evaluate how effectively the method was applied in the clinical workplace to address specific organisational risk. The three case reports partially achieved the study objectives. They provide preliminary evidence of the potential for BTA to be applied by healthcare professionals with minimal training, though with need for some ongoing support.
The reported BTAs were judged to have been conducted in compliance with recognised good-practice (as defined in Chartered Institute of Ergonomics and Human Factors9) and to a reasonable standard, suggesting BTA is potentially a feasible method for proactively analysing specific risks of harm and loss in different care settings. However, the results demonstrated that the training and guidance provided were not in themselves sufficient to produce a quality BTA: additional support and guidance from a specialist was needed. This may suggest improvements are needed in the nature of the training provided.
A number of other observations can be made from consideration of the three case reports;
In practice, the most cost-effective and productive approach is likely to be to invest in training a small number of people to develop deeper skills and experience in BTA—for example, advisers in clinical risk, quality improvement and patient safety. Expecting full-time clinicians or healthcare managers to facilitate a BTA on their own may not be an effective use of their time.
All three of the BTA Leads expressed the view that the BTA method led to a deeper understanding of the scope of issues, including systemic and human factors issues, and achieved a more thorough understanding of the risks and what was needed to control them than would have been the case if ‘normal practice’ in each organisation had been followed (ie, informal risk assessment without adopting any particular structured approach).
The visual nature of the final BTA diagrams was highly valued. In particular, it supported communication and sharing understanding of the scope of issues involved in controlling the risks.
The rank order of assessed quality ratings of the three studies (88% and 82% and 78%) matched the order of the estimated effort spent by the BTA Leads (50, 29 and 10 hours). However, given the very different contexts in which the three analyses were conducted, as well as the subjective nature of the effort and quality estimates, it is not possible to conclude that the additional effort itself led to higher quality outcomes.
The effort needed to conduct the analyses appeared reasonable and justified given the significance of the adverse events analysed. All three studies dealt with significant current concerns of the organisations involved: effort would have had to be spent on the issues whether or not a BTA was conducted. However, that effort may not have been justified for less serious risks.
Access to a customised software package is essential to creating the BTA diagrams and communicating the results of the analysis. Without such a package the effort needed to create the diagrams would be unrealistic. (Though note that the additional data management functionality provided by most commercial packages supporting BTA is not necessarily required to perform the analysis.)
Strengths and limitations
The training provided was not in itself sufficient to overcome trainees perception of the conceptual simplicity and graphical nature of the method. For example, despite the emphasis given both in the training and in the written guidance not to try to draw BTA diagrams before quality information was available, all three studies did exactly that. It was only after the interim reviews, when each of the BTA Leads were advised to use the tables provided in the NES guidance,18 to capture, clarify, organise and evaluate their thinking, and to then use that information to draw the diagrams, that the analyses became manageable, focused and fully productive. Trainees did not appear to appreciate the importance of using the staged approach, applying the concepts rigorously, and capturing the data needed before attempting to draw diagrams until they had run into difficulties.
The interim reviews by the NES BTA advisor early in the analysis process therefore appears to have been a critical element of the learning process. These reviews identified and corrected misunderstandings or confusions that were not identified in the training. Most importantly, the reviews led to reframing the adverse events in a way that made subsequent analysis more straightforward. For example,
In case report B, the analysis had initially been framed around the risk of patient death. During the interim review, the team realised that, while it is the ultimate concern, death only occurs after patients with unknown AKI get to hospital, and is therefore outside the direct control of the GP. As a GP surgery, they needed to focus the BTA on issues under their direct control: that is, identification and management of the risk before the patient goes to hospital. The adverse event was therefore reframed in terms of a progression towards AKI that is unknown to the GP surgery when the patient goes into hospital (which is within the scope and control of the surgery) rather than the death (which is outside of their control). Specifying the adverse event in a way that was more specific and clearly within the control of the GPs role greatly simplified the analysis.
In case report C, the adverse event was initially framed as the occurrence of a data breach. This confused the adverse event with the consequence: the occurrence of a data breach is the actual loss they sought to prevent. This confusion made it difficult for the team to think clearly about controls. The adverse event was therefore reframed as the technical team initiating the decommissioning process without being aware that the equipment contains PII.
Classroom-based training alone, supported by written guidance, does not therefore appear to be adequate to support development of competence to carry out a quality BTA in a healthcare setting. The opportunity to facilitate at least one analysis, with some specialist/trainer support, appears to be essential.
A significant limitation was the high attrition rate after the initial training intervention. Of the 17 individuals who attended training, 7 initiated an analysis, though only 3 actually completed an analysis within the study period. The key factors behind this high attrition rate seemed to be:
Most attended training out of interest in learning about BTA, rather than having a motivation to conduct their own analysis.
Individuals volunteered to take part through personal interest, so no time or resource was allocated by their organisation to carry out an analysis.
Following training, individuals were either reassigned to other work or had a change in priorities.
Lack of resource to complete an analysis within the time available to the study.
Next steps
The next planned steps in testing the utility of BTA is to target key workforce groups who may benefit from BTA training more than clinicians and managers with full-time and pressured sharp-end jobs. For example, most healthcare organisations employ clinical governance, risk, safety and improvement specialists who advise and support care teams in these domain areas. Application of BTA as a prospective risk analysis method could add value to their roles.
Conclusions
Barrier management refers to the process both of ensuring that sufficient, and sufficiently effective, controls are in place to protect against major risk, as well as assuring that those controls are actually in place and will function as expected when needed.9 Although widely used in many industries, BTA is a novel prospective risk analysis method in healthcare settings. To-date it has had some, though limited, traction. There remains a significant gap in knowledge and routine application of systems analysis tools capable of providing comprehensive understanding of barrier management issues affecting the safety of highly complex sociotechnical care systems. This gap needs to be closed to better inform how to strengthen care system design and so reduce the risks of serious events, including so-called ‘never events’, from occurring and reoccurring. BTA seems to have potential, though further research and evaluation of its application and utility is necessary.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. No datasets were generated in the study,
Ethics statements
Footnotes
Twitter @diggledom, @pbnes
Contributors The work was planned and managed, and the paper written by the first author (RM). WR, MS and MP each led and reported on analyses in their workplaces. PB sponsored the work on behalf of NHS Education for Scotland and acted as technical reviewer.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.