Article Text
PDF
Abstract
Background Incident reporting systems (IRS) can improve care quality and patient safety, yet their impact is limited by clinician engagement. Our objective was to assess barriers to reporting in a hospital-wide IRS and use data to inform ongoing improvement of a specialty-specific IRS embedded in the electronic health record targeting anaesthesiologists.
Methods This quality improvement (QI) evaluation used mixed methods, including qualitative interviews, faculty surveys and user data from the specialty-specific IRS. We conducted 24 semi-structured interviews from January to May 2023 in a large academic health system in Northern California. Participants included adult and paediatric anaesthesiologists, operating room nurses, surgeons and QI operators, recruited through convenience and snowball sampling. We identified key themes and factors influencing engagement, which were classified using the Systems Engineering Initiative for Patient Safety framework. We surveyed hospital anaesthesiologists in January and May 2023, and characterised the quantity and type of reports submitted to the new system.
Results Participants shared organisation and technology-related barriers to engagement in traditional system-wide IRSs, many of which the specialty-specific IRS addressed-specifically those related to technological access to the system. Barriers related to building psychological safety for those who report remain. Survey results showed that most barriers to reporting improved following the specialty-specific IRS launch, but limited time remained an ongoing barrier (25 respondents out of 44, 56.8%). A total of 964 reports with quality/safety concerns were submitted over the first 8 months of implementation; 47–76 unique anaesthesiologists engaged per month. The top safety quality categories of concern were equipment and technology (25.9%), clinical complications (25.3%) and communication and scheduling (19.9%).
Conclusions These findings suggest that a specialty-specific IRS can facilitate increased physician engagement in quality and safety reporting and complement existing system-wide IRSs.
- Health services research
- Electronic Health Records
- Quality improvement
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Traditional system-wide incident reporting systems (IRS) could improve care quality and patient safety, yet their impact is determined by the degree of clinician engagement.
WHAT THIS STUDY ADDS
We used a combination of interviews, surveys, and user data to determine whether a specialty-specific IRS increased clinician engagement in reporting and identified key themes and factors affecting engagement.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Our findings regarding the need for psychological safety and other considerations of factors affecting an institution’s culture of safety should be considered when relying on an IRS to capture quality or safety related concerns.
Introduction
Adverse events in healthcare settings are a preventable cause of morbidity and mortality,1 particularly in the inpatient setting.2 Incident reporting systems (IRSs) are one-way to identify risks in order to implement interventions.3 Hospitals are required by the Joint Commission to maintain a confidential IRS.4 5 The scope of IRSs has expanded beyond harm prevention into quality, including appropriate and efficient resource utilisation to achieve the best possible patient outcomes.1 6
However, there are mixed findings as to the success of IRSs in making healthcare safer and higher quality.3 7 8 The success of an IRS is dependent on the quality and number of reports submitted. Reporting rates are dependent on evidence that reports are being used appropriately, feedback given to reporters and an existing overall ‘culture of safety’.2 7 9 10 A recent meta-analysis indicated that higher healthcare staff engagement was correlated with patient safety outcomes.11 Yet, engaging physicians in reporting is challenging due to concerns about adverse professional consequences, time limitations and the nature of the incident itself, particularly if the incident did not result in patient harm.12 13 Physicians can view IRSs as an infringement on their autonomy and professional judgement.14–16 Further, IRSs may also reflect bias with one study finding fewer reports on men and white clinicians as compared with their colleagues.17
Evidence-based IRS best practices include having clear roles and responsibilities for events, greater engagement from clinicians of diverse professions and fostering of shared experiences from reports with visible action.18 Publicly rewarding high-impact reports,19 fostering physician-specific spaces for discussion of safety concerns20 and involving managerial non-clinician staff21 can increase engagement. Most research to date focuses on hospital-wide IRSs, but a greater understanding of the complementary role of physician-specific and specialty-specific IRS is needed.
In this mixed methods evaluation, we sought to identify barriers to quality and safety reporting among healthcare professionals working in surgical settings in a large academic health system to inform the ongoing improvement of a specialty-specific IRS embedded in the electronic health record (EHR) targeting anaesthesiologists. The Systems Engineering Initiative for Patient Safety (SEIPS) V.2.0 framework was selected for analysis to highlight the complexity of interactions between people and systems involved in submitting an IRS report and the work factors that may drive engagement or create additional barriers.22
Methods
Setting
This evaluation took place in the Department of Anesthesiology, Perioperative and Pain Medicine at Stanford Health Care (Palo Alto, California, USA), a quaternary academic medical centre that performs approximately 120 000 surgeries on predominantly adult patients, requiring approximately 72 000 anaesthetics annually.
Existing hospital-wide incident reporting system
The hospital-wide IRS allows any employee to report on potential and realised patient safety events, with the option to submit reports anonymously (figure 1, Pathway 1A). Reports are reviewed by a hospital-level safety team consisting of nursing and operational quality experts. Depending on incident severity, this team can carry out its own critical event review or forward the report to another relevant leadership team or nursing or physician leader for a response.
Workflow for all IRSs. Depending on the category selected the reporter could be prompted to submit a report to the hospital-wide IRS instead. EHR, electronic health record; IRS, incident reporting systems; QI, quality improvement.
A second reporting system, also hosted on the hospital intranet, allows reporting of professionalism-related events (figure 1, Pathway 1B). We will refer to this system as the IRS for professionalism. Until 2022 both professionalism and safety events were reported to the same IRS. In figure 1, we describe these two systems.
Specialty-specific incident reporting system
Anaesthesiologists informally recognised the limitations of the hospital-wide IRS, including the cumbersome form, reporting language using terminology nurses are more likely to find familiar and opaque follow-up process and impact. Anaesthesiology quality improvement (QI) leaders developed a complementary IRS, specific to their specialty (figure 1, Pathway 2). The new IRS was adapted from the anaesthesia-specific IRS at Massachusetts General Hospital23 and the EHR-integrated IRS at Lucile Packard Children’s Hospital.24 The IRS was developed as part of a hospital-wide QI programme.25
The point-of-care, anaesthesiology-specific IRS was embedded in the EHR with mandatory (as of October 2022) comment on closure of the patient’s intraoperative record. Using language reflective of the perioperative environment, anaesthesiologists characterised reports as a ‘Quality concern/Notable event’ or ‘Kudos’ which recognised a positive event within the operative encounter. Per incident, a checklist of 50 quality or safety event categories is provided for reporters to review and check off (online supplemental appendix A). At the time of this work, the specialty-specific IRS was available to anaesthesiologists, with the goal of expanding to certified registered nurse anaesthetists (CRNAs) and trainees.
Supplemental material
To ensure integration between the two IRSs, anaesthesiologists are directed to submit reports to the hospital-wide IRS for a subset of events including Equipment/Technology, Clinical Complication or Communication and Scheduling; efforts to automate this step are ongoing. Physician QI leadership also forward reports to hospital-wide IRS as appropriate.
Reports are extracted from the EHR into a secure spreadsheet monthly. One of two departmental physician QI leads and faculty volunteers review reports, sorting them into eight categories (discuss with committee; follow-up; follow-up and forward; forward; not to discuss; project; project and forward; track) (online supplemental appendix A). A subset of reports with high acuity are selected by reviewers and brought before the Quality Reporting Subcommittee of the Departmental Quality Council for discussion. This subcommittee is composed of departmental QI leadership and a rotating group of 20 faculty volunteers. The next steps are determined for each report and are recorded in the spreadsheet.
Reporting trends and the downstream impact of reports are presented at monthly faculty meetings and emailed to the department every month. Efforts to provide individual report/reporter feedback are ongoing.
Data collection and analysis
Qualitative interviews, faculty surveys and user data from the specialty-specific IRS were collected in a convergent mixed methods design,26 and results were interpreted and presented through the SEIPS V.2.0 framework to highlight work factors addressed by the specialty-specific IRS and limitations.22 This project was deemed not human subjects research by the Stanford Institutional Review Board as quality improvement (Protocol ID #68776). Individual interview participants gave oral consent prior to interviews and meeting participants were informed that notes would be systematically collected by the evaluation team.
Survey of faculty anaesthesiologists
Attending anaesthesiologists were asked to complete an anonymous online survey (Google Forms, Mountain View, California, USA) during monthly department meetings in January and May 2023, and the survey link was emailed to allow absent individuals to participate. Frequencies and percentages were calculated for each question. The denominators for reported percentages are the number of surveys completed in January and May 2023, respectively. Further details can be found in online supplemental appendix B. Major survey concepts included: a number of reports filed in the hospital-wide and specialty-specific systems, reasons for being less likely to file a report and confidence that the issue prompting the report will be addressed.
Engagement with specialty-specific IRS
Engagement in the specialty-specific IRS was assessed using the Department’s Quality Council spreadsheet which captured all IRS reports from 1 October 2022 to 31 May 2023. Four outcomes were used to describe engagement: (1) the number of ‘kudos’' reports (total and per month); (2) the number of quality and safety concern reports (total and per month); (3) the number of unique reporting clinicians relative to the faculty exposed to the intervention (total and per month); and (4) the number of unique patient encounters with a report (total). Percentages of the type of concern (total and per month) and assessment (total) were also calculated. Additional details can be found in online supplemental appendix A. All survey and engagement analyses were performed using SAS (V.9.4; SAS Institute, Cary, North Carolina, USA).
Qualitative interviews
Adult and paediatric anaesthesiologists, surgeons, operating room registered nurses, CRNAs and anaesthesiologist physician trainees were recruited using convenience and snowball sampling and invited to participate in 30 min semi-structured interviews via teleconferencing. All outreach ceased on reaching thematic saturation.
The interview guide (ASL, CBJ, SV), was meant to capture experiences with the hospital-wide IRS and the specialty-specific IRS, types of safety events reported, psychological safety to report and feedback on reports (online supplemental appendix C). Interview recordings were transcribed verbatim (REV; San Francisco, California, USA), de-identified and imported into qualitative software for analysis (QSR International’s NVivo 2020). Interviews were analysed thematically using a deductive and inductive approach in which a priori codes were drawn from the interview protocol and emergent themes were identified during analysis. A subset of interviews were coded by three authors (ASL, SV and AP) to inform consensus discussions,27 and the remaining interviews were coded by a single author (ASL) (online supplemental appendix E). Data were analysed by individual themes and by the interviewee’s profession to identify patterns. Observational notes from monthly Anaesthesia QI Reporting Review Committee Meetings provided additional context.
Ethics statement
The Stanford University Institutional Review approved this project (Protocol # 68776) as a quality improvement project for the purpose of improving clinical care.
Patient and public involvement
Patients were not involved in this evaluation.
Results
Our mixed-methods results pertained to: (1) engagement and reporting in hospital-wide and specialty-specific IRSs, (2) barriers to IRS reporting, including improvement recommendations for specialty-specific IRS.
Engagement and reporting in hospital-wide and specialty-specific IRSs
The first faculty survey in January 2023 was completed by 65 anaesthesiology faculty who reported limited engagement with hospital-wide IRS: only 53.8% (35/65) of anaesthesiologists self-reported that they completed at least one report in the last year; 46.2% (30/65) completed no reports.
Anaesthesiologist engagement with the specialty-specific IRS was more substantial; a total of 178 clinicians submitted 1059 reports to the specialty-specific IRS for 1057 encounters from 1 October 2022 to 31 May 2023. On average, 61±9 clinicians submitted a report per month with a range of 47–76 each month. Of the 1059 submitted reports, 95 were positive ‘kudos’ reports from 47 clinicians, averaging 11.9±9.5 ‘kudos’ per month. Most reports (n=964) were for quality and safety concerns. The total number of quality and safety reports was highest in the first month of implementation for the specialty-specific IRS (n=168), decreasing to a relatively stable average of 114±14 reports per month in subsequent months (figure 2A). More faculty anaesthesiologists reported experiencing the measured barriers when referring to the hospital-wide IRS than the specialty-specific IRS; the number of faculty reporting each barrier in January and May surveys is shown in table 1.
Engagement with the specialty-specific incident reporting system from October 2022 through May 2023. (A) Shows the number of quality and safety reports and number of unique physician reporters per month. (B) Shows the percentage (%) of quality and safety concern types out of the total number of quality and safety concerns for each month.
Responses of faculty anaesthesiologists to survey questions capturing barriers to submitting reports through hospital-wide system (January 2023 survey) and specialty-specific (January 2023 and May 2023 surveys) incident reporting systems (IRS)
Physicians categorised concerns in the specialty-specific reporting system; figure 2B shows the percentage of reports within the six categories of concerns. Across all periods, the most common concerns were equipment and technology issues (25.9%), clinical complications (25.3%) and communication and scheduling concerns (19.9%).
The departmental quality and safety leadership reviewed all reports and summarised the results of committee meetings at monthly faculty meetings in part to increase physicians’ confidence that concerns will be addressed (January survey 61.5% (40/65) reported being somewhat to completely confident). Indeed, by May 2023, this percentage had risen to 70.5% (31/65).
Barriers for IRS reporting
For the specialty-specific IRS, 44 anaesthesiology faculty completing the May 2023 survey reported that barriers to engagement in specialty-specific IRS decreased relative to the hospital-wide IRS (table 1). However, limited time (25/44, 56.8%) and not knowing what to report (8/44, 18%) remained ongoing barriers.
Qualitative interviews took place with 24 clinicians and QI team members (of 46 invited), lasting 24 min on average between January and April 2023. For the participant’s clinical role, please refer to online supplemental appendix D. Barriers are organised according to the work system factors of the SEIPS V.2.0 framework in table 2A,B, with the rightmost column elaborating on how the specialty-specific IRS addresses these barriers.
(A) SEIPS V.2.0 person, task and tools and technology factors influencing clinician engagement in hospital-wide IRS and whether the specialty-specific IRS addresses these factors. (B) SEIPS V.2.0 organisation and environmental factors influencing clinician engagement in hospital-wide IRS and whether the specialty-specific IRS addresses these factors
Interviewees identified barriers at each SIEPS V.2.0 level. Most factors related to engagement were categorised as organisational challenges with emphasis on fear of negative professional consequences, desire to receive feedback and processes to capture trends and act on them in a timely fashion.
Person level
Interviewees reported limited engagement in the hospital-wide IRS based on not knowing how to file reports and perceiving events as not being reportable. These events tended to not result in negative outcomes, such as lack of equipment or near-misses, being a QI opportunity.
The specialty-specific IRS addresses these barriers by being embedded in the EHR, and therefore being easy to locate, as well as lowering the threshold for reporting events given that the convenience of quickly noting opportunities for improvement.
Task, tools and technology factors
Task factors were largely related to the time to report events, as well as when reporting occurs. Interviewees noted that filling out the report does make closing a patient’s record a longer process and that if the event being reported was an emotionally-draining one, adding more requirements is not ideal. Other barriers related to timing include that events can happen even after the record is closed.
Barriers related to tools and technology were poor accessibility and too many required fields in the hospital-wide IRS report. The specialty-specific IRS largely addressed barriers related to tools and technology by having few fields to fill out and streamlining the reporting experience but did not address task-related barriers—especially those related to clinicians having protected time to report events—although it did allow reporters to include positive events.
Organisational factors
Most factors noted were organisational, largely related to negative experiences with hospital-wide IRS reports, not seeing reports being acted on, not perceiving department-specific improvements efforts resulting from the hospital-wide IRS, lack of psychological safety and certain roles being more likely to submit reports. Interviewees tended not to differentiate between the hospital-wide IRS for professionalism and the one for safety, suggesting that professionalism reports are often submitted to the hospital-wide IRS system and that reporters assume both are treated equally. Some participants noted negative experiences from professionalism reports, such as being asked to justify clinical decision-making or reports about unprofessional behaviour submitted when acting appropriately.
Individuals noted that getting feedback from a supervisor regarding inappropriate professionalism reports hampered willingness to submit future reports. They also commented that actions taken on safety reports seemed to be of little benefit to patient safety. While one individual acknowledged that professionalism issues could impact patient safety, the majority indicated that safety and professional concerns should be handled separately. Physicians wanted more direct oral feedback at the time of the incident regarding behaviour interpreted by their colleagues as unprofessional. However, some interviewees noted other concerns regarding professionalism reports, such as bias in who is reported being unprofessional, with more non-white and female individuals being reported on.
Some organisational factors were partially addressed by the specialty-specific IRS, such as having department-specific oversight over reports and efforts, with more feedback than the hospital-wide IRS, but more challenging organisation-wide barriers were not addressed.
Environmental factors (internal and external)
Environmental barriers noted by interviewees included limited access to a computer for filing reports, being motivated to report due to fear of litigation and in settings with limited personnel-being identified when filing a report with the risk of retaliation. A surgeon noted not spending much time in front of a computer. Another clinician commented on their motivation for filing reports to avoid possible litigation, which does not align with the IRS’s purpose to improve patient safety. Others shared in settings such as in an operating room- if someone files a report it is easy to identify who could have done so which allows for professional retaliation. The specialty-specific IRS does not address these barriers.
Discussion
This mixed methods evaluation found numerous barriers to physician engagement in quality and safety IRSs, many which were addressed by a specialty-specific IRS embedded into the EHR. Physician engagement in reporting was specifically supported by organisational-level facilitators: point-of-care accessibility via the EHR, clear separation between safety/quality concerns and professionalism issues and regular feedback on actions due to IRS reports. Our findings validated previous research on the role of feedback, bias and a culture of safety in quality reporting. We demonstrated the importance of feedback to reporters for motivating further engagement,28 29 and showed that the lack of a culture of safety can be a barrier to engagement.30 31 We also observed that the manner in which professionalism issues are addressed by leadership can affect confidence in the system as a whole and preferences in feedback to the reporter (eg, not feedback on one’s clinical judgement, but whether a submitted report is appropriate or if there are any improvement efforts stemming from it).
Integrating the IRS into the EHR and making interaction mandatory facilitated access and use of the specialty-specific system. Notably, reducing this access barrier through technology (integration with EHR) and organisation (making it a forced choice) resulted in 408/964 (42.3%) reports issues delegated to other units and 46/964 (4.8%) new QI projects. This volume of new projects is almost twice what is reported for other specialties as ongoing projects (n=25), suggesting increased access has identified more areas for improvement than other methods.32
Despite attempts of the IRSs to separate professionalism concerns from quality/safety concerns, these were at the forefront of many conversations and related to the role of anonymity. With a separate track for professionalism concerns, our findings suggest anonymity of reports may not be desirable or feasible for quality and safety reporting. Previous work in the context of a learning health system has identified facilitative factors: confidentiality, not being required and easily accessible. Here, the hospital-wide IRS relied on predominantly anonymous reports, while the specialty-specific IRS required identification. Physicians believed the latter system led to improving the number of high-impact issues reported (vs low-impact, interpersonal issues).31 33 34 To protect a confidential, but not anonymous, reporting system from reluctance to report, previous reviews suggest providing legal immunity to reporters, highlighting positive quality changes resulting from reporting and attending to any fear of embarrassment that might result from reporting.35
Our findings highlight the importance of having a convenient reporting system, but also the need to build psychological safety across all roles. Interviewees stressed that feedback on improvement efforts is important to motivate continued reporting and support a culture of safety. Feedback could address power differentials that negatively impact perceptions of safety culture. One study demonstrated that more hierarchical settings impeded safety culture36; it is an open question as to whether transparency could serve to create more level power structures, thereby encouraging safety culture and reporting.
Given the reporting volume and limited resources devoted to the specialty-specific IRS, it is challenging to provide personalised feedback even though it could translate to higher engagement. While ideally, everyone would receive feedback on the appropriateness of reports, that is not feasible due to initial volumes.
Limitations for this evaluation included the anonymity of the faculty providing survey data, which could not be linked to engagement data. Also, although the faculty meeting was an opportunity to reach a large number of anaesthesiologists, it is highly likely not all anaesthesiologists attended due to timing conflicts. Hospital-wide IRS engagement data were also largely anonymous, not linked to the profession and were therefore not used; however, accessing such data may be an area for future work. Finally, data collection took place at the beginning of the intervention because of resource limitations; only the latter portion reflects a ‘steady-state’ behaviour.
Conclusion
These findings suggest that a specialty-specific IRS can facilitate increased physician engagement in quality and safety reporting and complement existing system-wide IRSs. Future improvements include organisational efforts to build greater psychological safety when reporting incidents, perhaps through greater information sharing on safety issues and initiatives.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
Stanford Medicine Institutional Review Board exempted this study. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors would like to thank Dr Brian Bateman of the Department of Anesthesia and Dr Marcy Winget of the Evaluation Sciences Unit for their support of the project.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
ASL and SV are joint first authors.
Contributors ASL: Contributed to the conception of the study, data collection, data analysis and writing this manuscript. She will act as guarantor for this manuscript. SV: Contributed to the conception of the study, data collection, data analysis and writing this manuscript. SK: Contributed to the conception of the study, data analysis and writing this manuscript. CB-J: Contributed to the conception of the study, data analysis and writing this manuscript. MA: Contributed to the conception of the study, commented/reviewed this manuscript. DV: Contributed to the conception of the study, commented/reviewed this manuscript. DG: Contributed to the conception of the study, commented/reviewed this manuscript. WJG: Contributed to the conception of the study, commented/reviewed this manuscript. AP: Contributed to the conception of the study, data analysis and writing this manuscript.
Funding This project was supported by Stanford Health Care as part of the Improvement Capability Development Program.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.