Article Text
PDF
Abstract
Background The COVID-19 pandemic necessitated increased synchronous distance education (SDE) in graduate medical education, presenting challenges for Quality Improvement and Patient Safety (QIPS) best practices, which call for integration with daily clinical care and investigation of real patient safety events.
Objective To evaluate educational outcomes for QIPS training after conversion of a mature, in-person curriculum to SDE.
Methods 68 postgraduate year (PGY)-1 residents were surveyed before and after the SDE Culture of Patient Safety training in June 2020, and 59 PGY-2s were administered the Quality Improvement Knowledge Application Tool-Revised (QIKAT-R) before and after the SDE QIPS seminar series in July–August 2020. Values before and after training were compared using sign tests for matched pairs (PGY-1) and Wilcoxon signed-rank tests (PGY-2).
Results 100% (68 of 68) of PGY-1s and 46% (27 of 59) of PGY-2s completed precourse and postcourse surveys. Before the course, 55 PGY-1s (81%) strongly agreed that submitting patient safety event reports are a physician’s responsibility, and 63 (93%) did so after (15% increase, p=0.004). For PGY-2s, the median composite QIKAT-R score was 17 (IQR 14.5–20) before and 22.5 (IQR 20–24.5) after the seminars, with a median difference of 4.5 (IQR 1.5–7), a 32% increase in QIPS competency (p=0.001).
Conclusions Patient safety attitudes and quality improvement knowledge increased after SDE QIPS training at comparable levels to previously published results for in-person training, supporting SDE use in future hybrid curricula to optimise educational value and reach.
- Medical education
- Graduate medical education
- Patient safety
- Quality improvement
Data availability statement
Data are available on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Best practices in quality and safety training in graduate medical education call for close integration with clinical care, including investigation of real patient safety events. Efficacy of synchronous distance education for such training, necessitated by the COVID-19 pandemic, has not been described in the literature.
WHAT THIS STUDY ADDS
Quality and safety attitudes and knowledge increased after a synchronous distance education curriculum, with the magnitude of changes similar to previously published reports for in-person training.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
There are benefits of in-person quality and safety training (better understanding of the clinical environment) and virtual training (expanded educational reach, improved learner psychological safety). Given similar improvements in attitudes and knowledge transfer, educators may choose to create hybrid curricula to blend the best of both approaches.
Introduction
The COVID-19 pandemic and need for physical distancing led to increased synchronous distance education (SDE). Best practices for SDE are described in the literature.1 2 However, prepandemic best practices for Quality Improvement and Patient Safety (QIPS) training in Graduate Medical Education call for close integration with clinical care, including evaluation of real patient safety events (PSE).3 4 The multidisciplinary, interactive nature of QIPS work and the need to understand the clinical environment present obstacles for SDE.
A review of distance-based QIPS training concluded that such programmes can be effective.5 However, established courses frequently use asynchronous online modules and do not incorporate real PSE, deviating from QIPS best practices. Furthermore, prepandemic courses were implemented to address a lack of institutional QIPS educators, and do not address the translation of in-person QIPS best practices into SDE. QIPS educators must understand the impact of SDE curricula to inform future efforts and potentially create high-impact blended models.
We sought to evaluate changes in knowledge and attitudes after transitioning from a mature in-person QIPS curriculum to SDE.
Methods
Setting and participants
The curriculum was implemented at an urban institution that spans a quaternary/academic care centre, academic county and trauma referral centre, and Veterans Affairs medical centre. The graduated QIPS curriculum focuses on the culture of patient safety during postgraduate year (PGY)−1 and PSE case investigation during PGY-2. A faculty director and two chief residents for QIPS lead the curriculum. Data are from the 2020–2021 academic year, in which PGY-1s included internal medicine, neurology and ophthalmology, and PGY-2s included internal and family medicine.
Interventions
SDE best practices include familiarity with videoconferencing and educational software, focused small group exploration and effective facilitation.1 2 The in-person curriculum was adapted accordingly as below.
PGY-1: culture of patient safety seminar
Prior to the pandemic, PGY-1s participated in a half-day Culture of Patient Safety Seminar centred around patient/family speakers sharing nuanced personal narratives about tragedies as well as PSE successes where resident involvement led to meaningful improvements. Thereafter, chief residents led PGY-1s through interactive exercises to introduce high-reliability organisation principles (eg, proactively identifying possible future harms), demonstrating their ability to contribute to PS efforts, culminating in a report-out to the patient/family speakers.4 The SDE curriculum followed the same format, with speakers delivering personal narratives through an online videoconferencing platform (Zoom). Chief residents led small group sessions (5–7 learners) using breakout rooms, slide templates and the Zoom annotation feature to replace physical posterboards and sticky notes and maintain interactivity.
PGY-2: QIPS seminar series
Prior to the pandemic, PGY2s participated in a 4-week longitudinal seminar (half-day per week). The seminars blended large-group didactics with mentored small-group investigation of selected PSE cases with educational value and operational importance to the health system. Weeks 1–2 focused on case investigation while weeks 3–4 focused on QI methodology, concluding with residents presenting findings and recommendations to medical centre leadership.4 The SDE seminars followed the same format and were conducted using an institutional, HIPAA-compliant Zoom account given the investigation of real PSE. Chief residents taught didactics while faculty moderated the chat and added QIPS pearls. In facilitated small groups, residents used online diagramming software (LucidChart) to synchronously assemble and display timelines, cause maps, process maps, impact/effort matrices, design measures, project charters and plan-do-study-act (PDSA) cycles. Course directors trained facilitators on LucidChart and created templates for all activities in advance. The overall time balance was shifted towards small group investigation to foster engagement. Stakeholder interviews were conducted via telephone. Unlike prepandemic years, a Gemba walk was not conducted due to infection control precautions.
Outcomes measures
We conducted presurveys and postsurveys (online supplemental information) using a 5-level Likert scale of agreement to assess PGY-1 attitudes towards their role in advancing the culture of patient safety and entering PSE reports.
Supplemental material
We used the validated Quality Improvement Knowledge Application Tool-Revised (QIKAT-R) to evaluate PGY-2 preknowledge and postknowledge. Scores are reported as a composite of three domains (Aims, Measures and Change; each domain scored separately and worth 0–9 points).6
Analysis of the outcomes
We summarised survey responses and knowledge scores using descriptive statistics. For the PGY-1 course, we compared levels of agreement with survey questions using sign tests for matched pairs. For the PGY-2 course, we compared composite QIKAT-R scores using Wilcoxon signed-rank tests for paired data. Statistical analyses were performed using Stata V.15.0 (StataCorp).
Patient and public involvement
Patients and public were not involved in the design, conduct, reporting or dissemination plans of this educational research.
Results
Figure 1 displays the survey results for PGY-1s. Complete paired predata and post data were available for all 68 participants. Before the seminar, 55 respondents (81%) strongly agreed that submitting PSE reports are a physician’s responsibility, compared with 63 (93%) after (15% increase, two-sided sign test, p=0.004).
Survey results of postgraduate year 1 residents before and after completing the synchronous distance education Culture of Patient Safety Seminar, 2020–2021 academic year. Survey respondents selected from 5-level Likert scale (strongly disagree, disagree, neutral, agree and strongly agree. No respondents selected strongly disagree or disagree). All 68 participants completed surveys both before and after the orientation. PSE, patient safety event.
Figure 2 shows the QIKAT-R scores for PGY-2s who completed the precourse and postcourse knowledge assessments (N=51 and 34, respectively, of 59 participants). Of the subset who completed both (N=27), the median composite score was 17 (IQR 14.5–20) before and 22.5 (IQR 20–24.5) after the seminar series, with a median difference of 4.5 (IQR 1.5–7), a 32% increase in QIPS knowledge (Wilcoxon signed-rank test, p=0.001).
Scores on the quality improvement knowledge application tool revised before and after postgraduate year 2 residents completed the virtual 4-week Quality Improvement and Patient Safety Seminar Series, 2020–2021 academic year. (A): Scores by domain. (B): Total scores. The shaded boxes denote the second through third quartiles of scores. The horizontal lines inside the boxes denote the median values. The Xs inside the boxes denote the mean values. The whiskers denote the upper and lower values. The small circles denote outlier values. Total number of respondents before seminar: 51. Total number of respondents after seminar: 34.
Discussion
Our results demonstrate that SDE QIPS curriculum can improve patient safety attitudes and quality improvement knowledge, with similar magnitude of effect to our previously reported findings for in-person training. Twenty-one per centof PGY-1s more strongly agreed that PSE reports are a physician’s responsibility after the in-person Culture of Patient Safety Seminar compared with 15% after SDE.4 There was a 32% increase in median composite QIKAT-R scores for both in-person and SDE.4
To our knowledge, this is the first evaluation of attitudes and knowledge transfer after transitioning to an SDE QIPS curriculum. We believe our foundation of local patient cases and well prepared, effective small group facilitation enabled the curriculum’s success. The PGY1 Culture of Patient Safety Seminar hinges on compelling, nuanced narratives delivered by real patient/family speakers. The presentation as SDE did not diminish the emotional gravity, as many learners attested to in words and physical expression during subsequent small group sessions. The PGY2 QIPS Seminar Series hinges on effective small group facilitation of recent PSE cases. The chief residents have extensive experience with software tools, enabling a smooth transition to SDE. This ability to lead a younger generation of near-peer learners through virtual platforms has been identified as a need in the literature.7 The most notable downside to the SDE curriculum was the loss of the unique and potent insights afforded by a Gemba walk to better understand the clinical environment in which PSE events occurred.
Limitations include short duration and small sample size from a single institution. The lack of a control cohort, given in-person training was not possible during the height of the pandemic, significantly limits our ability to draw firm conclusions regarding the comparative effectiveness of in-person and SDE curricula. However, the stand-alone, pre–post analysis of the SDE curriculum does provide some evidence to support SDE, and the historical, semiquantitative comparison to a recent cohort from the same institution using the same curricular framework allows for hypothesis generation for future studies of SDE compared directly to in-person training. Additional limitations include the use of only attitudes and short-term knowledge transfer as outcome measures; per prior application of the Kirkpatrick framework to QIPS training, evaluation of behaviour change (eg, entering PSE reports; engaging in QI work) and patient outcomes as higher-level outcome measures should subsequently be evaluated.8 Given the potential for virtual tools (online diagramming software, virtual meetings) to facilitate QI project work, this will be especially important for future study.
Our evaluation supports the notion that in-person QIPS best practices can be successfully adapted to SDE, with similar gains in attitude and knowledge. SDE also has distinct benefits, including easier attendance (especially in multiple-hospital systems, programmes with rural tracks or for away rotations) and the ability to schedule more stakeholder interviews for PSE case investigations. QIPS educators also remain a limited resource, and SDE may facilitate easier expansion, helping integrate horizontally across specialties and disciplines and vertically across levels of training. Educators thus may wish to incorporate hybrid components in future iterations of QIPS curricula (eg, virtual didactics but on-site Gemba walks). Previous reports also support increased learner satisfaction and psychological safety with blended virtual curricula.9–11 Future evaluation of blended curricula, and specific components of SDE versus in-person training, can help identify the optimal dose and use of virtual training tools.
Conclusion
SDE QIPS training improved patient safety attitudes and quality improvement knowledge, with similar effect size compared with previously published reports evaluating in-person training. As in-person instruction returns, educators may wish to blend SDE and in-person training into hybrid curricula, with subsequent evaluation to identify optimal dose and use of virtual training tools.
Data availability statement
Data are available on reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
Our institutional Human Subjects Division determined that this human subject research qualifies for exempt status (Category 1) on 6 July 2020, IRB ID STUDY00010684. Informed consent was not obtained for this exempt study.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors AC contributed to conception and design, data acquisition and manuscript drafting, gave final approval for publication and is the guarantor accountable for all aspects of this work. NK contributed to conception and design and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work. LKVV contributed to data analysis and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work. MN contributed to data acquisition and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work. LS contributed to data acquisition and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work. TA contributed to conception and design and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work. CW contributed to conception and design, data acquisition and manuscript drafting, gave final approval for publication and agrees to accountability for all aspects of the work.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.