Article Text
PDF
Abstract
Background Psychiatry has not prioritised quality improvement and patient safety (QIPS) to the same degree as other medical specialties. Professional capacity building in QIPS through the education of residents is essential to improving the quality and safety of mental healthcare delivery.
Local problem The University of Toronto postgraduate psychiatry program is the largest psychiatry training program in North America. Training in QIPS was introduced in 2006. In 2019, a curricular review found that few trainees acquired competence in QIPS.
Methods Curricular change was undertaken using Kern’s Six-Step Approach to curricular design. We used a continuous quality improvement framework to inform the evaluation with data collection using an online educational application. We aimed to improve competence in QIPS as demonstrated by assessment of the quality of individual quality improvement projects (IQIP) on an 11-item rubric. We used a family of quality improvement measures to iteratively improve the curriculum over 3 years.
Interventions We restructured the QIPS curriculum into four case-based seminars for third year psychiatry residents. The curriculum included: clear learning objectives, multimodal instructional methods, and an IQIP.
Results The mean score on preintervention project evaluations was 5.3/11 (49% (18)), which increased to 9.2/11 (84% (11.5)) with the revised curriculum (t=8.80, two tail, p<0.001; Cohen’s ds 2.63). In the first two cohorts of residents to complete the IQIPs, 67/72 (93%) completed at least one Plan-Do-Study-Act cycle, compared with 11/23 (48%) in the 2 years before the new curriculum.
Conclusions To ensure our trainees were attaining the educational goal of competence in QIPS, we introduced a revised QIPS curriculum and embedded an evaluation rooted in improvement science. This study adds to the limited literature which uses continuous quality improvement to enhance QIPS education, which is particularly needed in mental health.
- Quality improvement
- Graduate medical education
- Mental health
Data availability statement
Data are available upon reasonable request. Data are available upon request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Psychiatry has lagged behind other medical specialties in the uptake and practice of quality improvement methodologies. There is a lack of evaluated quality improvement and patient safety (QIPS) curricula in psychiatry.
WHAT THIS STUDY ADDS
A virtually delivered, case-based curriculum in QIPS was effective in teaching key QI competencies using psychiatry-based clinical examples. Trainees completing the programme could be effective participants in QIPS initiatives in the future.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
This study provides evidence for the use of a virtually delivered QIPS curriculum in mental health with modest resources. It is feasible to implement in a range of training programmes and settings.
Introduction
Problem description
Psychiatry has historically lagged behind other medical specialties in adopting quality improvement and patient safety (QIPS) as a clinical and educational priority.1 Without sufficient training and experience in QIPS among psychiatrists, mental healthcare is unlikely to systematically improve, and adverse patient safety events are likely to continue to recur.
Although postgraduate training in QIPS is a requirement of educational accreditation in both Canada and the USA and is included in Core Medical Training in the UK, there are few published curricula specific to mental health.2 Even fewer publications evaluate the quality of the QIPS projects in which trainees participate. For example, a physician–manager curriculum, which included education in QIPS, was introduced in the psychiatry residency programme at the University of Toronto in 2006.3 4 The curriculum included didactic sessions and a group-based QI project (GQIP). A qualitative evaluation focused on learners’ experiences completing the GQIPs, but did not assess the methodological rigour or clinical impact of the projects.5 A subsequent quantitative evaluation of a QI curriculum6 used the Quality Improvement Knowledge Assessment Tool and a QI self-assessment tool to assess psychiatry residents’ knowledge and skills, but did not include a thorough appraisal of the QI project the residents worked on during the curriculum. Similarly, another study from the UK focused on an education intervention to increase engagement of psychiatry residents in QI projects; however, this study also did not systematically evaluate the quality of the projects.7 Due to these gaps in the literature, it remains unclear how effective these curricula are in achieving the goal of psychiatrists competently participating in QIPS projects, with meaningful clinical impact, on graduation.
While other QIPS curricula in psychiatry have recently been described,2 8 none focus on the Institute for Healthcare Improvement’s (IHI) Model for Improvement,9 which has been adopted by many healthcare organisations internationally, including the United Kingdom’s National Health Service. As Bolland10 argues, the Model for Improvement is one of two leading approaches to QI and it is increasingly important for clinicians to not only understand this approach to QI, but also apply it in their clinical contexts. We focused on the IHI model as it is endorsed by our regional health authority and it can be readily applied in the multiple mental healthcare contexts in which our residents train, namely, inpatient units, ambulatory clinics, and emergency services. Further, the IHI model can be implemented by both individual clinicians and teams. As such, we developed a psychiatric QIPS curriculum centred on the Model for Improvement9 to promote competence in QIPS among psychiatric trainees. This article describes an educational improvement study that uses the IHI approach to continuous quality improvement (CQI) to structure the curricular evaluation.
Educational objective
We set an initial educational objective for the revised QIPS curriculum, namely, by June 2019, 90% of trainees will have working knowledge of key QI tools and data analysis as demonstrated by completion of three case studies and progress quizzes. Trainee and educational stakeholder feedback during the pilot year of the case-based curriculum identified that the case studies did not provide learners with experience in primary QIPS data collection and analysis. Further, the QIPS medical education literature emphasises the importance of experiential learning through QI projects to enhance learning through performance in practice.11–13 Thus, an individual quality improvement project (IQIP), in which trainees applied the model for improvement, was added during the second year of the revised curriculum. Group projects were not feasible due to a reduction in the length of clinical rotations in the residency programme. We revised our educational objective to reflect the introduction of the IQIPs, namely, by June 2020, 80% of trainees completing the new curriculum will have working knowledge of key QI tools and data analysis as demonstrated by completion of an IQIP with at least one Plan-Do-Study-Act (PDSA)14 cycle.
Methods
We followed the medical curriculum development and evaluation framework outlined by Thomas et al.15 This six-step model includes a general needs assessment, a targeted needs assessment, setting clear learning objectives, selecting instructional methods to meet the objectives, implementation and evaluation. In designing and implementing the curriculum, we wanted to develop a dynamic and responsive approach to the evaluation that would permit improvements to occur throughout the academic year, in contrast to the usual practice of reviewing students’ evaluations annually and changing the curriculum for subsequent cohorts. Traditional curriculum design and evaluation frameworks often lack the agility needed to allow for the dynamic implementation of new teaching methods and content.15 Furthermore, the COVID-19 pandemic has demonstrated the need for educators to be nimble and capable of changing quickly, while concurrently collecting meaningful data on the effectiveness of the intervention. We also wanted to provide a living example of the QI process to the trainees, making explicit to the trainees that they were part of a process to rapidly improve their learning experience. While the evidence for using a CQI framework in medical education is recommended to promote continuous curricular improvement15 16 and has been widely accepted,17 there has been slow uptake of a rapid CQI approach to curriculum implementation and evaluation. Through a CQI framework, instructors can test curricular changes in each teaching session and decide to adopt or abandon changes. Emerging literature suggests this approach is both feasible and appropriate for educational interventions.16–19
Educational interventions
We maintained the educational structure of five half-days for the QIPS curriculum for third year psychiatry residents, but substantially revised the curricular content of four seminars (table 1). Residents verbally presented their applied projects during the final seminar. The team involved in implementing the revised curriculum included: three psychiatry faculty with advanced certificate training in QIPS, the psychiatry programme director and administrative staff from postgraduate education. The new curriculum included clear learning objectives and a variety of instructional methods which emphasised interaction (table 1). The seminars were modified so they could be delivered in-person, or online using Zoom or a similar video-conferencing software, which is advantageous for bringing learners together in a large, geographically dispersed training programme. The interactive components of the curriculum included: group-based games to simulate data collection, visualisation and analysis for QI; case studies with short answer questions that required applied use of QI methodology; think-pair-share; and small group supervision with faculty to develop IQIPs. The case-study content was developed from QIPS projects in mental health that were previously successfully completed in our department. They included a range of clinical settings (primary care, urgent ambulatory care and inpatient) to expose trainees to the breadth of QIPS opportunities in psychiatry.
Revised quality improvement in psychiatry curriculum framework
Evaluation methodology
During the first year of the new curriculum, we focused our evaluation on group-level quiz performance to collect just enough data to determine if trainees were learning the curricular material. However, as we piloted the new curriculum, we recognised that group-level data would not be sufficient. We then developed a more comprehensive evaluation plan, in the second and third year of the new curriculum, that included quantitative and qualitative feedback from the trainees, instructors and educational stakeholders. We structured the evaluation using the QI concepts of outcome, process and balancing measures.
Outcome measures
Residents’ verbal presentations of their IQIPs were evaluated by faculty using an 11-item rubric. The rubric was derived from previously used QI evaluations developed for team-based QI projects.20 21 It was adapted for individual practice improvement and assesses the following elements: (1) project rationale; (2) identification of at least one quality domain addressed by the project; (3) an aim statement; (4) description of a change idea and rationale for selection of a change idea; (5) applied use of a QI diagnostic tool such as a process map, fishbone diagram or pareto chart; (6) identification of a measurement plan that includes outcome, process and balancing measures; (7) description of a rational and feasible data collection plan; (8) data displayed in graphical format such as a run chart or control chart; (9) one PDSA cycle completed; (10) two or more PDSA cycles completed; (11) generating recommendations for the next step of the project which are supported by data. Each item was scored as 0 (incomplete), 0.5 (partially completed), or 1 (correctly completed) for a maximum total possible score of 11.
To ensure data completeness, we communicated with the postgraduate education office to clarify which residents were granted academic accommodations or leaves by the postgraduate education director. Trainees who were unable to attend the presentation day submitted their presentation electronically for evaluation. To evaluate whether the observed outcomes were attributable to the revised curriculum, the 23 group-based projects completed by 61 residents who participated in the old curriculum (academic years 2016–17 and 2017–18) were evaluated using the same rubric to facilitate comparison with the results from the revised curriculum. Trainees’ overall performance on the IQIPs (mean score on 11 items) was compared with the overall performance (mean score on 11 items) on the team-based QI projects from the old curriculum, using a two-tailed t-test, assuming unequal variance. We also calculated Cohen’s ds (for unequal sample sizes) to assess the effect size. All data analysis was completed using Microsoft Excel. The distribution of the data was visualised. The data on overall performance from the old curriculum were normally distributed, but the data on the overall performance from the revised curriculum had moderate left skew. The t-test was therefore recomputed using the −1/
transformation of the data.
Process measures
At each session, trainees completed a 10–15 item quiz which included multiple choice and short answer questions to test factual knowledge of QIPS concepts and methodology. For example, one question presented a run chart and the trainee needed to interpret the data using run chart rules to detect special cause variation. The quizzes were initially administered on paper. One of the iterative improvements we introduced was a web-based teaching application, Nearpod (https://nearpod.com/), to administer the quizzes electronically which provided trainees with immediate performance feedback on their knowledge acquisition. The quiz answers were also reviewed during the didactic portion of the seminar to promote consolidation of learning. All data from the trainees who voluntarily and anonymously participated in the quizzes were extracted from Nearpod and included in the evaluation of the curriculum. To measure engagement in this aspect of the curriculum, we measured the number of quizzes completed at each session. Quiz data were grouped by the month in which the quiz was completed to track longitudinal changes in performance over the course of the curriculum. The data were visualised to ensure normal distribution and single-factor analysis of variance was used to assess quiz performance over the course of the seminars in each academic year (grouped by the month the quiz was completed). Eta-squared (η2) was calculated to assess effect size. Trainees completed the quizzes anonymously, using a different pseudonym at each session, which prevented use of a repeated measures statistical analysis across individuals.
Balancing measures
The Nearpod App was used to collect real-time qualitative feedback from trainees about the curriculum and their experiences working on their IQIPs. The qualitative feedback from trainees was collated for each session and reviewed by the faculty to inform modifications to the didactic portion of the curriculum and faculty supervision of the development of the IQIPs. We also solicited qualitative feedback from the Psychiatry Postgraduate Program Director.
We generated group and individual-level data on learning outcomes. Trainees received individual performance feedback on the quizzes and verbal feedback on the development of the IQIP from faculty and peers at every learning session. We used a CQI framework that included PDSA cycles for our curricular evaluation to ensure our educational objectives were being achieved. The changes we introduced for each PDSA cycle are outlined below.
PDSA#1 involved piloting the case-based curriculum: learning outcomes were evaluated using three paper-based anonymous quizzes aggregated at the level of the group of trainees.
In PDSA#2, we maintained the case-based curriculum and introduced the IQIP into the curriculum. To prepare their IQIPs, trainees worked on the IHI project charter during the latter half of seminars two to four. They received group-based supervision from the QIPS faculty during the seminar. They verbally presented their IQIPs during the final seminar and their performance was evaluated by faculty.
PDSA#3 involved abandoning paper administration of quizzes and testing out electronic administration using an online education platform, Nearpod. Trainees could voluntarily complete the quizzes anonymously. They received instantaneous feedback on their performance on the quiz. Based on the outcomes from PDSA#1, the number of quizzes was increased to 4 and performance on the quizzes was collated for each student individually. Nearpod was also used to solicit qualitative feedback in real-time during the sessions. The feedback was displayed on a collaborative electronic message board such that trainees could agree with posts made by others. Qualitative feedback was used by the instructors to facilitate real-time adjustments to didactic content and communications with the trainees.
In PDSA#4, all seminars were offered virtually, in real time. The COVID-19 global pandemic began to impact in-person learning at the University of Toronto in March 2020. During the 2020–2021 and 2021–2022 academic years, all postgraduate education seminars in psychiatry were offered virtually using the Zoom videoconferencing platform. Based on the findings from PDSA#3, the frequency of quizzes was increased to 5 with the goal of further consolidating trainees’ knowledge of QIPS concepts and methodology. Use of the Nearpod platform was adopted based on positive feedback from trainees and faculty. Due to qualitative feedback from the trainees, supervision of the IQIPs was modified to include example projects.
Patient and public involvement
Given the focus of the project, input from patients and the public was not obtained.
Results
General needs assessment
The University of Toronto postgraduate psychiatry programme is the largest psychiatry training programme in North America with more than 35 trainees/year and clinical rotations available at more than 20 affiliated hospital/clinic sites. Beginning in 2006, trainees in their third year of the 5-year programme were expected to: (1) attend one half-day workshop on patient safety in first year of residency; (2) attend three half-day workshops in third year of residency which reviewed key concepts in quality improvement and planning quality improvement projects; (3) complete a GQIP with supervision from a faculty member affiliated with a teaching unit/clinic.4 Most GQIPs involved child and adolescent psychiatry, care of individuals with chronic mental illness, or hospital-based emergency psychiatry as these clinical services corresponded to the core clinical rotations of the trainees, or their on-call duties.
In 2020, the Canadian Royal College of Physicians and Surgeons, which accredits postgraduate education in Canada, required a transition to a competency-based approach to medical education. In preparation for this change, in 2018, a program-wide review of the entire residency curriculum at the University of Toronto was undertaken, leading to curricular reform. The clinical rotations were restructuring from 6 months to 1 to 4 months on various clinical teaching units. With the shortened duration of clinical rotations, trainees would no longer be affiliated with clinical teams for a sufficient time to design, implement and collect data for GQIPs.
Targeted needs assessment
The GQIPs completed from 2016 to 2018 were assessed using a standardised rubric to evaluate fidelity to QI methodology. Quantitative assessment of the learning outcomes from the GQIPs revealed that most trainees were not gaining competency in essential QIPS tools and methods (figure 1), and thus were not well positioned to lead or contribute to QIPS interventions on completion of the residency training programme.
Final QI project evaluation scores. (A) 
2016–2018 (n=23). (B) 
2016–2018 (n=23), 
2019–2021 (n=72).
Results for outcome measures
In the first cohort of residents to complete the IQIPs (2019–2020 academic year), 27/31 (87%) completed a project with at least one PDSA cycle. In the second cohort (2020–2021 academic year), 40/41 (98%) completed an IQIP with at least one PDSA cycle. The mean score from the rubric for the IQIP evaluations for the residents who completed the revised curriculum between 2019 and 2021 was significantly greater than the mean GQIP scores for the residents who completed the preintervention curriculum. The standardised mean difference (Cohen’s ds) of the old curriculum compared with the revised curriculum was 2.63 (table 2). The −1/
transformation of the mean project scores also yielded a statistically significant difference in performance on the projects associated with the revised curriculum and the standardised mean difference (Cohen’s ds) was 1.29 (table 2). Figure 1 depicts the performance on the different components of the rubric used to evaluate the GQIPs in the 2 years before the revised curriculum was introduced and in the two cohorts of residents who completed IQIPs with the new curriculum. There was a substantial increase in trainees’ demonstrated ability to describe change ideas, outline a data collection plan, complete QI data analysis and generate recommendations, but limited change in the number of projects in which multiple PDSA cycles were undertaken.
Evaluation of overall performance on QI projects old curriculum compared with the revised curriculum
Results for process and balancing measures
During the first year of the revised curriculum, there was limited improvement in the mean score over three quizzes which were administered in November, February and May (table 3), although there was some improvement in performance on individual items. Voluntary participation in the quizzes declined substantially by the final session and the instructors found the data collection on paper cumbersome to collate and analyse. These findings informed the decision for PDSA#3 to switch to electronic administration of the quizzes, beginning with the second quiz in the second year of the curriculum.
Performance on progress quizzes with the revised curriculum
During the second year of the curriculum, quizzes were administered in September, November, February and March. There was no significant improvement in quiz performance over the academic year and there was attrition in voluntary participation (table 3). The Nearpod application was identified as a beneficial component of the curriculum in qualitative feedback from trainees. They also requested additional specific instructions to develop their QI project charter. This feedback informed the development of PDSA#4.
During the third year of the revised curriculum, performance on the progress quizzes showed significant improvement over the course of the academic year, with an η2 of 0.092, and voluntary participation in the quizzes was more consistent (table 3). Pairwise post hoc t-tests, corrected for multiple comparisons, showed significant improvement with a large effect size on quizzes 4 and 5, respectively, compared with quiz 1.
Discussion
Summary
Using CQI, we implemented and evaluated a revised psychiatry residency QIPS curriculum centred on the model for improvement which included: clear learning objectives, multimodal instructional methods and an applied IQIP. The model for improvement was also used to evaluate the curriculum. The family of measures (outcome, process and balancing) developed for the evaluation indicate that we met our aim of improving trainees’ knowledge, skills and attitudes in QIPS.
We adopted an innovative approach, administering anonymous quizzes through an online educational application, Nearpod. This application not only provided trainees with immediate performance feedback, but also provided the instructors with timely measurement of the trainees’ knowledge (quiz performance) and attitudes (qualitative feedback). Further, the online application was adaptable for both in-person and online instruction, which proved invaluable with the shift to virtual education associated with the COVID-19 pandemic. It provided another means to promote interaction between faculty and trainees in the online environment as the global pandemic limited other opportunities for educational interaction. It also facilitated the collection of balancing measure data which underscored our commitment to learner engagement and promoted rapid, responsive modifications to the curriculum to better meet trainees’ educational needs. To our knowledge, this is the only study to include a comprehensive evaluation of a postgraduate psychiatry QIPS curriculum focused on the model for improvement. This study adds to the limited literature22 23 in which improvement science is employed to enhance QIPS education. Our results demonstrate the utility of the measurement plan associated with the model for improvement for educational evaluations. The data collection and analysis supported timely feedback to both residents and faculty, assisted with stakeholder engagement, and promoted real-time curricular change.
Using the CQI approach to curriculum renewal, we exceeded our goal of 80% of trainees completing an IQIP with at least one PDSA cycle. Our analyses showed a substantial improvement in competence in QIPS, as indicated by the significantly improved project evaluations for residents completing the revised curriculum, compared with the previous curriculum. There was a large effect size, as measured by Cohen’s ds, associated with the revised curriculum. In addition, in the third year of the curriculum, the mean scores on the content quizzes progressively improved over the course of the academic year, with a moderate effect size, and trainees more consistently voluntarily participated in this evaluation strategy.
Our work adds to the growing literature which demonstrates the utility of CQI as a framework for studying educational interventions in QIPS.22 23 The framework facilitated our ability to quickly convert to an online educational model, necessitated by the COVID-19 pandemic, while simultaneously continuing to study our educational interventions despite these profound contextual changes in the educational environment.
Limitations
Our study was limited to psychiatry residents studying at the University of Toronto. As a very large training programme, it was challenging to collect and collate attendance data at each seminar, particularly when the curriculum was running in-person. At some of the early seminars, 50% or fewer of the residents participated in the quizzes on Nearpod. The instructions to log onto Nearpod were provided at the outset of the session, which may have accounted for the lower than expected participation as some trainees arrived late to the seminars. Nevertheless, we were able to collect some process measure data at each seminar which guided concurrent curriculum development. The results for our outcome measures do not suggest this limitation hampered our trainees from achieving curriculum goals. Few residents completed IQIPs with multiple PDSA cycles, and this should be a goal of further improvement to the curriculum. Our evaluation would be enhanced by further longitudinal study of the career outcomes of our trainees, that is, to ascertain if they are more likely to engage in QIPS initiatives, following completion of their residency.
Conclusions
Mental health patients were excluded from many of the initial seminal studies in QIPS.24–26 Psychiatric patients continue to be excluded from many patient safety studies and reports,27 which has contributed to psychiatrists being slower than other medical specialists to incorporate an emphasis on QIPS in education and clinical care. In Canada, there are increasing regulatory requirements to engage in QIPS to maintain specialist certification28 and medical licensure.29 As such, rigorously developed QIPS curricula for psychiatrists are imperative. These learning needs served as an impetus to develop a more robust curriculum for psychiatry residents.
Our residency programme took an early lead in introducing a physician manager curriculum which included training in QIPS. However, a quantitative evaluation of the curriculum was recommended,5 but not undertaken, and most trainees who completed this curriculum failed to demonstrate competence in using QI methods and tools in their applied group projects. In this context, we introduced a revised QIPS curriculum. We embedded an evaluation rooted in improvement science to ensure we were helping our trainees meet their educational goals, while concomitantly promoting the development of a future generation of psychiatrists who will improve the quality and safety of mental healthcare delivery.
We are now in the fifth year of our curriculum evaluation and we have introduced several curricular changes through our PDSA cycles which have supported the sustainability of our work. Given the demonstrated positive educational outcomes of this evaluation, we will aim to spread our curriculum to other residency programmes and pursue an ongoing educational evaluation driven by improvement science.
Data availability statement
Data are available upon reasonable request. Data are available upon request.
Ethics statements
Patient consent for publication
Ethics approval
Our study was exempt from Institutional Research Ethics Board review as it was an evaluation of an existing educational program. Participation in the seminar quizzes was completely voluntary and anonymous. Quiz results did not influence the residency program’s evaluation of the trainees. IQIP evaluations were submitted to the residency program as were the original curriculum’s GQIP evaluations.
Acknowledgments
We gratefully acknowledge the administrative support of the Postgraduate Education Division of the Department of Psychiatry, Temerty Faculty of Medicine, University of Toronto.
References
Footnotes
Contributors TB designed and wrote the study and conducted the analysis. JRP contributed to the data collection and writing of the study. AEW contributed to the data collection and writing of the study. TAB is the author acting as guarantor.
Funding Funding for a portion of this project has been provided by the University of Toronto Department of Psychiatry Postgraduate Innovation Fund (grant/award number: not applicable).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.