Article Text
PDF
Abstract
Importance Adequate situational awareness in patient care increases patient safety and quality of care. To improve situational awareness, an innovative, low-fidelity simulation method referred to as Room of Improvement, has proven effective in various clinical settings.
Objective To investigate the impact after 3 months of Room of Improvement training on the ability to detect patient safety hazards during an intensive care unit shift handover, based on critical incident reporting system (CIRS) cases reported in the same hospital.
Methods In this educational intervention, 130 healthcare professionals observed safety hazards in a Room of Improvement in a 2 (time 1 vs time 2)×2 (alone vs in a team) factorial design. The hazards were divided into immediately critical and non-critical.
Results The results of 130 participants were included in the analysis. At time 1, no statistically significant differences were found between individuals and teams, either overall or for non-critical errors. At time 2, there was an increase in the detection rate of all implemented errors for teams compared with time 1, but not for individuals. The detection rate for critical errors was higher than for non-critical errors at both time points, with individual and group results at time 2 not significantly different from those at time 1. An increase in the perception of safety culture was found in the pre-post test for the questions whether the handling of errors is open and professional and whether errors are discussed in the team.
Discussion Our results indicate a sustained learning effect after 12 weeks, with collaboration in teams leading to a significantly better outcome. The training improved the actual error detection rates, and participants reported improved handling and discussion of errors in their daily work. This indicates a subjectively improved safety culture among healthcare workers as a result of the situational awareness training in the Room of Improvement. As this method promotes a culture of safety, it is a promising tool for a well-functioning CIRS that closes the loop.
- patient safety
- hospital medicine
- human factors
- incident reporting
- medical education
Data availability statement
Data are available on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
An innovative low-fidelity and widely accepted learning approach to strengthening situational awareness is the Room of Improvement.
WHAT THIS STUDY ADDS
By using a facility’s critical incident reporting system (CIRS) reports to select hazards for a Room of Improvement, situational awareness of hospital-specific hazards can be trained.
The combination of the quality assurance instruments CIRS and Room of Improvement additionally presents a distinct advantage of achieving a closed Plan-Do-Check-Act cycle.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The Room of Improvement can be an effective tool for increasing situational awareness, which may lead to better handover and handling of critical situations in the intensive care unit.
This method is also a promising tool to promote a safety culture and facilitate a well-functioning closing-the-loop CIRS.
The use of hospital-specific CIRS cases could increase motivation and the learning effect.
Introduction
Hospitalisation poses many iatrogenic risks for patients, including improper hygiene measures and the risk of experiencing a diagnostic or medication error.1 In particular, the highly complex environment of an intensive care unit (ICU) presents an area prone to patient safety hazards.2 To mitigate these risks, healthcare professionals require knowledge in patient safety and situational awareness, that is, the ability to maintain an appropriate internal representation of the environmental state in complex and dynamic settings where sudden changes in conditions occur.3
An innovative low-fidelity and widely accepted learning approach to strengthening situational awareness is the ‘Room of Improvement’ or ‘Room of Horrors’, where errors caused by incorrect medical management are hidden in a specifically prepared room and must be detected, thus providing a hands-on experience to train observation, critical thinking and situational awareness.4 5 By using a facility’s critical incident reporting system (CIRS) reports to select hazards for a Room of Improvement, situational awareness of hospital-specific hazards can be trained.6 The combination of the quality assurance instruments CIRS and Room of Improvement additionally presents a distinct advantage of achieving a closed Plan-Do-Check-Act cycle.7
This study aimed to test the feasibility of such a CIRS-Room of Improvement combination and to evaluate the long-term effects of Room of Improvement training on situational awareness, including the ability to identify patient safety hazards during a shift handover in the ICU 3 months after the training, as well as the handling and discussion of errors in everyday work. The inclusion of a post-test with a second Room of Improvement enabled us to observe the learning effects directly, rather than relying solely on participants’ subjective perceptions, as is often the case. Furthermore, we investigated the effect of the mode of training (alone or in a team) as previous research suggests that team training can enhance the learning effect of Room of Improvement.8
Methods
Design and setting
Our study had a 2 (time 1 vs time 2)×2 (alone vs in a team) factorial design. The study took place in a tertiary care hospital in Switzerland during normal shifts in an ICU.
Material
For each time point, we created one Room of Improvement in the ICU, where we placed a mannequin surrounded by 24–28 safety hazards based on CIRS cases from the same hospital.9 The two rooms contained similar, non-overlapping sets of safety hazards, most of them related to (1) technical medical devices, (2) incorrect administration of medication and (3) prescription errors (online supplemental table 1).
Supplemental material
We classified the hazards into (1) ‘critical’ errors that require immediate attention (eg, blood transfusion of wrong blood group) and (2) ‘uncritical’ errors that pose a risk to the patient but are unlikely to be fatal in the short term (eg, blood transfusion via central venous catheter [CVC]). Each room was also equipped with a written patient case (brief medical record, medications, prescriptions, etc), an instruction sheet, a tablet for digital documentation and a solution sheet (online supplemental figure 1).
Procedure
We conducted two sessions with the same cohort of participants 12 weeks apart. At the start of each session, participants were instructed to conduct a patient takeover in our Room of Improvement, either individually or in teams. They were given a QR code that led to (1) a documentation sheet describing the observed errors and (2) a questionnaire that recorded their professional group, individual or team participation, relevance of the identified errors in daily routine, difficulties in identifying errors and the impact of the Room of Improvement on learning and patient safety perception. Additionally, they were emailed (3) baseline (before time 1) and post-test (after time 2) questionnaires to assess their perception of how errors were handled and discussed within their teams (online supplemental table 2).
Participants had 15 min to complete each session and then compared their documented errors with a solution sheet. An expert with clinical and research experience conducted a brief debriefing after each session to enhance learning and emphasised that hazards are hospital-specific CIRS cases. All data collected were anonymised.
Participants
All ICU medical and nursing staff of all educational degrees were invited to participate in the sessions voluntarily. They were free to manage the sessions individually or in monoprofessional and interprofessional teams.
Statistical analyses
The primary end points of this study were the number of correctly detected errors at the two different time points, overall and by category (critical vs non-critical) for individuals and teams. Prior to performing statistical analyses, each dataset per time point was assessed for normality distribution using the D’Agostino and Pearson omnibus normality test, as well as for homogeneity of variances using Levene’s test. For normally distributed data (with Welch’s correction in case of unequal variances), a Student’s t-test was performed, while non-normally distributed data were analysed using the Mann-Whitney U test.
To analyse the pre-post questionnaire regarding handling of errors and discussion of errors within the team, we used the Fisher’s exact test.
Statistical analyses were performed using IBM SPSS Statistics (V.24.0.01, IBM SPSS, Chicago, Illinois, USA) and GraphPad Prism (V.9.4.0 for Windows, GraphPad Software, San Diego, California, USA). Significance was assumed at p<0.05 (two-sided for t-tests and exact for Mann-Whitney U test).
Results
Of 111 eligible healthcare professionals, 85 (77%) participated at time 1, with 23.5% participating alone and 76.5% participating in teams of 2–5 individuals. At time 2, 45 healthcare professionals took part, 44.4% of whom participated alone. At both time points, most teams (52.3%) consisted of two people. Most participants (86.2%) were qualified nurses, 8.5% were physicians, 3.1% were registered nurses and 0.8% had a different professional background. Due to the small number of physicians compared with nurses, it was not possible to statistically analyse the interprofessional teams.
How good were the healthcare professionals at hazard recognition at time 1?
At time 1, individuals detected an average of 43.5% (SD 14.7%) of all implemented errors and teams detected 48.9% (SD 15.9%). Thereby, individuals detected 69.0% (SD 22.9%) of critical errors and 36.8% (SD 16.0%) of non-critical errors, while teams detected 72.6% (SD 27.3%) of critical errors and 42.7% (SD 16.4%) of uncritical errors. No statistically meaningful differences were found between individuals and teams, either overall or for non-critical errors (figure 1A–C).
Distribution of patient safety errors identified by individual and group participants. (A–C) Comparative statistics between individual and group participants in relation to their first and second run for common, non-critical and critical errors. (D) Comparative statistics between the detection rate of critical and non-critical errors. *P<0.05, **p<0.01, ***p<0.001, ****p<0.0001; ns, not significant.
Was there a learning effect?
At time 2, individuals detected an average of 47.0% (SD 12.0%) and teams 57.9% (SD 10.4%) of all implemented errors. For teams, this represented an increase in the detection rate compared with time 1 (Δ8.9%, 95% CI 3.2% to 14.7%; p=0.003), but not for individuals (Δ3.4%, 95% CI −5.1% to 12%; p>0.05).
For non-critical errors, individuals detected an average of 40.4% (SD 12.9%) and teams 53.6% (SD 10.9%) at time 2. For teams, this represented a significant increase in the detection rate, (Δ11%, 95% CI 4.8% to 17.2%; p<0.001), but not for individuals (Δ3.6%, 95% CI −5.7% to 12.9%; p>0.05) (figure 1B). Looking at critical errors, individuals detected an average of 77.0% (SD 23.6%) and teams 77.6% (SD 24.7%) at time 2. Neither rate was significantly different from time 1 (Δ0%, 95% CI 0% to 20%; p>0.05 for individuals and Δ0%, 95% CI 0% to 20%; p>0.05 for teams) (figure 1C).
Note that the detection rate of critical errors was higher than that of non-critical errors at both time points, which indicates a certain level of diligence in critical matters (Δ37.9%, 95% CI 22.1% to 44.2%, p<0.001 for individuals at time 1; Δ36.8%, 95% CI 27.4% to 43.1%, p<0.001 for teams at time 1; Δ36.6%, 95% CI 24.4% to 48.9%%, p<0.001 for individuals at time 2 and Δ30.4%, 95% CI 16.5% to 36.5%, p<0.001 for teams at time 2, respectively) (figure 1D) (online supplemental figure 2).
Changes in perception of safety culture
At baseline (before time 1), 53.5% of participants described the handling as open and professional. This number increased to 75.0% at post-test, p=0.023. Between baseline and post-test, there was an improvement in the behaviour of discussing errors in the team and/or with the person who made the error (39.2%–85.8%, p>0.001). When asked whether errors were discussed in the team or with the person who caused the error, 42.2% of the participants agreed at baseline that this was usually the case. This number increased to 88.8% at post-test, p<0.001. Safety-related incidents such as frequency, safety in documentation and drug prescribing showed a significant improvement (p<0.001).
Discussion
Does a 15 min Room of Improvement training session based on hospital-specific CIRS cases of a patient handover in the ICU have a long-term learning effect? Our results indicate a sustained learning effect even after 12 weeks, at least, when considering the teams at time 2. These findings align with those of Clay et al,8 who suggest conducting simulations in teams and repeating the sessions to reinforce the positive effects of situational awareness training in a Room of Improvement. The training improved the actual error detection rate, and participants reported an improved way of handling and discussing errors in their daily work. This indicates a subjectively improved safety culture among healthcare workers following the situational awareness training provided in the Room of Improvement intervention. Consequently, this emphasises the need for situational awareness training for recognising hazards in medical management and for promoting a safety culture. This observation is consistent with previous research highlighting the importance of recognising and reporting medical errors in the workplace as key components of a hospital’s overall safety culture.10 In light of all these findings, it is advisable to incorporate a Room of Improvement as an educational tool into everyday medical practice and, at best, to repeat it every 3–6 months. CIRS-specific cases provide the best starting point for designing errors, as each medical unit faces its own individual patient hazards, as shown in an ICU.
Limitations and open questions
This study has several limitations in terms of the generalisability of the findings to other professions, training modes and content. First, being a single-centre study involving primarily nurses, the results are not necessarily representative of other healthcare facilities or professional groups. Second, due to our study design, we were unable to completely rule out the possibility of an order effect, despite our efforts to create similar sets of patient safety hazards for both rooms. Third, our setting allowed for voluntary participation, either individually or in teams at the participants’ own disposal, and we ensured completely anonymous data collection. While this approach increased acceptance and resulted in a high participation rate, it leaves the question of the specific conditions for successful Room of Improvement trainings open for future research. The independent decision of the participants as to whether they approach the simulation training alone or in a team is an important limitation. At time 1, there were 20 people alone and 65 in groups, a ratio of 1:4; at time 2, there were 20 alone and 25 in groups, a ratio of almost 1:1. It should be mentioned that the participants were asked not to share the errors they found with each other. Based on the previous experience of the participants in CRM training courses, it can be assumed that most of them adhered to this. This is also indicated by the many ‘additional errors found’ that were not actually present. The extent to which the use of hospital-specific CIRS cases positively influenced the learning outcome would have to be investigated in a double-blind study. Similarly, future research may investigate whether the use of hospital-specific CIRS cases positively increases motivation to participate and the transferability of the learning effect to everyday work.
Conclusion
In conclusion, the Room of Improvement can be an effective tool for increasing situational awareness, which may lead to better handover and handling of critical situations in the ICU. This method is also a promising tool to promote a safety culture and facilitate a well-functioning closing-the-loop CIRS. The use of hospital-specific CIRS cases could increase motivation and the learning effect.
Data availability statement
Data are available on reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study is not subject to the Swiss Human Research Act and therefore does not require a permit.
Acknowledgments
We acknowledge Bölli-Tomic, who served as an English language editor for this manuscript.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors CG and UP wrote the main manuscript text. CAR, WH and UP collected and analysed data. CAR did the statistical analysis and prepared figure 1. JK and UP supervised the process of creating this paper. MF and JK contributed substantially to conception and discussion. All authors revised the manuscript critically for important intellectual content and approved the final manuscript. UP accepts full responsibility for the work and the conduct of the study, had access to the data, and controlled the decision to publish.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.