Article Text
PDF
Abstract
Introduction Practical Assessment of Clinical Examination Skills (PACES) constitutes the final part of the mandatory Royal College of Physicians exam series for progression to higher specialty training. Pass rates were lower for core medical trainees (CMTs) in Coventry and Warwickshire in comparison to other regions within the West Midlands and nationally.
Objectives Our aim was to improve pass rates in the region through the introduction of a stimulating and supportive teaching framework, designed to enhance the quality and frequency of PACES teaching.
Methods To identify key areas for change a baseline questionnaire, including Likert Scale and free text questions related to PACES teaching, was distributed to all CMTs in the region. Many trainees highlighted concern over lack of PACES-orientated teaching and support, with particular emphasis on: lack of bedside-teaching with feedback; infrequent opportunities for practising communication skills; and difficulty identifying suitable patients in an efficient manner. To address these concerns the following interventions were implemented over two Plan, Do, Study, Act (PDSA) cycles which were analysed at 6 months and 12months: a digital forum to highlight relevant inpatients for examination practice; a peer-to-peer mentoring scheme; a consultant-led bedside-teaching rota; and classroom-based communication skills sessions.
Results Pass rates at Annual Review of Competence Progression improved from baseline to the end of the first year of implementation, 56.3% to 77.3%, respectively. Furthermore, following analysis of questionnaires at each PDSA cycle, we demonstrated a progressive improvement in trainee satisfaction in exposure, quality and relevance of teaching.
Conclusion Our innovative, cost-effective teaching framework for PACES preparation has improved exam outcomes and facilitated swift junior doctor career progression, while raising the profile of the trust. Furthermore, this innovation provides a template for potential adoption in other National Health Service institutions.
- medical education
- team training
- quality improvement
- graduate medical education
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Problem
National Membership of Royal College of Physicians (MRCP) statistics for 2014–2015, indicate that only 67% of West Midlands’ core medical trainees (CMTs) had completed Practical Assessment of Clinical Examination Skills (PACES) by the time of their final Annual Review of Competence Progression (ARCP), in comparison with a national average of 78%.1 Moreover, local data for trainees within Coventry and Warwickshire demonstrated an even lower pass rate of 56.3%.
Through personal and peers’ experience of undertaking the MRCP PACES exam, the authors identified suboptimal support, resources and learning opportunities for PACES exam preparation locally.
Although a robust weekly teaching programme was operational at University Hospital Coventry and Warwickshire (UHCW), designed to cover key aspects of the CMTs' e-portfolio curriculum, there was no PACES-focused teaching framework. The MRCP exam series is a mandatory requirement for progression through core medical training onto specialty trainee registrar (ST) training. Delay in obtaining RCP membership results in delayed career progression, postponing trainees from securing higher specialist training posts, in turn causing variable vacancies in specialist doctors’ positions nationally, potentially affecting quality service provision.
This project aims to identify and remove barriers to effective ward-based learning by introduction of a robust PACES-focused teaching framework including a peer-to-peer mentoring scheme. It is believed that these interventions will improve trainee engagement with the MRCP exam series and improve local examination pass rates, while boosting doctors’ morale.
Background
The MRCP exam series is a prerequisite for all CMTs to progress onto higher-level training. The exam series consists of three parts: two written aspects (parts 1 and 2) and the PACES examination. National statistics demonstrate lower pass rates for trainees in the PACES examination, 53.8%, as compared with the two written examinations (56.8% and 75.8% for parts 1 and 2, respectively).2 Several well established online resources are available to aid preparation for the written exams, in the form of question banks and past papers.
However, given the practical nature of the PACES exam, exposure to appropriate patients in clinical practice and availability of structured feedback from senior colleagues is required. Although there are several national PACES courses offering this they are often expensive, places are quickly filled and dates prior to trainees’ examinations are sometimes unavailable. Furthermore, the majority of these courses are in London and other major cities, compelling a large cohort of candidates to embark on cumbersome travel and stay during the vital pre-examination period.
Preparation for PACES exam requires long-term, consistent trainee engagement with regular practice of examination skills. This helps trainees feel more prepared for the examination, and builds confidence in their role as a senior doctor in day-to-day working.3 This crucial link is often overlooked by trainees, who are superficially focused on passing their exams. This project aims to overcome this mindset, and provide a framework to allow structured preparation well in advance of exams.
Baseline measurement
Final ARCP outcome data for CMTs in Coventry and Warwickshire for 2014–2015 were analysed to assess the proportion of trainees who had successfully completed the MRCP exams. Of a total 16 final year CMTs, only 9 had completed MRCP PACES by their ARCP date giving a pass rate of 56.3%.
Baseline Likert Scale questionnaires (online supplementary file) were distributed to all CMTs in Coventry and Warwickshire to determine the current quality of the local PACES teaching. Likert Scales were used, as they are easy to understand and allow participants to offer positive, negative and neutral opinions. Furthermore, they allow for easy interpretation and graphical representation of quantitative data.4
Supplementary file 1
Forty questionnaires were distributed with 35 completed questionnaires returned. These showed that 23% of CMTs had completed the MRCP exam series. Of these responses, 23 were from trainees who were in core medical training over the preceding 12 months, and included in our analysis. Nine per cent rated the relevance of formal CMTs' teaching to the PACES exam as 'good' or 'very good'. Seventeen per cent of trainees rated their exposure to ward-based PACES teaching as 'good' or 'very good', with 21% of trainees having attended more than four bedside-teaching sessions in the preceding 12 months, and 61% having attended no sessions at all. Furthermore, the quality of these sessions was reported as being 'good' or 'very good' by only 22% of trainees. Forty-eight per cent reported the available sessions as being of 'poor' quality.
Free text suggestions highlighted a lack of support structure in preparation for the PACES exam. Candidates reported a problem with finding suitable patients to practise examination skills in their respective hospitals. Some highlighted the need for formal history and communication skills practice sessions.
See online supplementary file which is an example of the questionnaire sent to trainees.
Design
Based on the initial data, key areas for change were identified and the following interventions were introduced over two Plan, Do, Study, Act (PDSA) cycles. Interventions for cycle 1 were introduced following the baseline measurements. The baseline questionnaire was redistributed and data analysed after 6 months. Cycle 2 interventions were then introduced and the same questionnaire was again used to assess for any improvement at 12 months.
In order to address the need for a general support network, a PACES mentoring scheme was implemented, whereby each trainee was allocated a mentor who had completed the MRCP series. Their role was to offer support and guidance in preparation for the exam in addition to providing one-to-one teaching opportunities on a more regular basis. Local volunteers were recruited (predominantly CMTs and junior registrars) allowing for each candidate to be allocated their own mentor.
This support framework was enhanced by the introduction of a confidential PACES forum on 'WhatsApp', which allowed both mentors and candidates to arrange sessions and share locations of useful practice cases without breaching any confidential patient information. To ensure this was the case, rules regarding sharing of patient information were strongly enforced and candidates were asked to agree to user conditions prior to joining the forum.
Furthermore, to offer candidates more support in practising history and communication skills with feedback, several out of hours, formal, classroom-based, group sessions were arranged. Candidates were offered the opportunity to sign up for these sessions and volunteers were recruited to be actors for the scenarios used.
Finally, consultants leading classroom-based curriculum-targeted CMT teaching were advised to use a focused, strategic teaching5 approach to PACES in the final half an hour of each teaching session.
Strategy
In PDSA cycle 1 the introduction of PACES mentors and the 'WhatsApp' group led to a marked improvement in trainee satisfaction and improved exposure to bedside-teaching. However, disappointing exam outcomes, and therefore pass rates, following the first exam sitting as well as trainee feedback identified the need for more senior-led bedside-teaching opportunities, to simulate PACES exam conditions.
This led to PDSA cycle 2 and the introduction of regular consultant-led bedside-teaching sessions via a weekly bedside teaching rota, to address the identified need for more senior-led teaching with performance feedback. All medical consultants in UHCW were approached to determine their availability and willingness to be involved in teaching. A core team of clinicians including consultants, STs and senior grade academic staff was allocated to an out of hours PACES teaching rota. Spaces were limited to a maximum of five trainees per session in order to optimise teaching delivery.
To assess overall effectiveness of our interventions, Likert Scale graphs were repeated for each aspect of study. Furthermore, the completion rate of MRCP by the time of final ARCP was compared with the previous year.
Results
The pass rates for MRCP PACES improved following our implementations. Of 22 CMTs 17 had passed PACES by their final ARCP date. This equated to a pass rate of 77.3%, which was a clear improvement compared with the baseline rate of 56.3% in the preintervention cohort, 12 months earlier.
These improvements were also reflected by the qualitative measurements of trainee satisfaction with exposure to and quality of bedside-teaching as well as relevance of curriculum-based teaching.
Ratings as ‘good’ or ‘very good’ for exposure to bedside-teaching improved from 17% to 46% after PDSA cycle 1 and to 59% after PDSA cycle 2.
There was an objective measure to verify this with rates of attendance to four of more PACES-oriented bedside-teaching sessions increasing from 21% at baseline to 46% after PDSA cycle 1 and to 55% after PDSA cycle 2.
Ratings as ‘good’ or ‘very good’ for quality of bedside-teaching sessions increased from a baseline of 21.7% to 61% after PDSA cycle 1 and 65% following PDSA cycle 2.
Overall ratings of relevance of curriculum-based teaching rated as ‘good’ or ‘very good’ showed an increase from 9% at baseline to 25% after PDSA cycle 1. This increased further to 34% following reinforcement of the intervention in PDSA cycle 2.
Graphs showing comparison of baseline to results at 6 months and 12 months following interventions are given in figure 1.
Summary of results in graphical format. CMT, core medical trainee, MRCP, Membership of Royal College of Physicians; PACES, Practical Assessment of Clinical Examination Skills.
Lessons and limitations
Despite results showing that this project has been successful in achieving its aims, we have identified some limitations.
Through difficulty in obtaining clean retrospective data regarding the overall pass rate of the PACES exam by ARCP, we have only compared our pass rate to the previous year. Comparing our pass rate to an average pass rate from the previous 2 years or 3 years would have added more significance to our results. Qualitatively and through assessment of raw data, the local college tutor has been able to confirm that pass rates had been lower than average for at least the preceding 2 years prior to our interventions.
Multiple interventions were introduced simultaneously within each PDSA cycle making it difficult to determine which were most useful. It is possible that some interventions will have been more influential on pass rates than others. However, qualitative data between PDSA cycles suggest that introduction of a consultant-led bedside-teaching rota and the WhatsApp group were the most positively received by trainees.
This project has demonstrated that the introduction of a teaching framework improves trainee satisfaction, and increases the frequency of, and engagement with, formal bedside-teaching sessions. It is beneficial for the candidates involved, and has been hugely rewarding for those participating in teaching. The efficiency of the ‘WhatsApp’ group, highlights the increasing role of technology and social media in modern education.
As a result of the first-come-first-serve policy with regard to attendance to bedside-teaching sessions, we identified that often the same trainees were attending multiple sessions, thus taking away opportunities from other, perhaps less motivated, trainees.
Furthermore, we felt that having prearranged bedside-teaching sessions could have potentially taken away the trainees' responsibility and independence for self-directed learning. We therefore felt it was important to highlight to trainees that this framework should be used as an adjunct to their own regular practice and revision. Similarly, we found that in the early stages of the project trainees were often not engaging with all aspects of the framework, namely the 'WhatsApp' group and utilisation of mentors. Engagement improved as exam dates approached. In the future, this could be addressed by the introduction of a recommended revision timeline.
The introduction of a regular consultant-led bedside-teaching rota was well received by candidates. However, its implementation was time-consuming and relied heavily on robust organisation to ensure the rota was adhered to. This could eventually lead to a collapse in the improvement structure when the current organisers withdraw input. To overcome this potential problem, it has been ensured that the framework implemented is easy to follow and can be continued and further improved by successive generations of junior doctors.
In the high-pressure climate of today's National Health Service, consultants often find it difficult to commit to regular bedside-teaching sessions, even in a teaching hospital environment. This problem could potentially be even worse in other hospital settings. It is in everyone’s interest to ensure that trainees have support from seniors in learning, to facilitate timely progression.
Conclusion
This project demonstrated a clear improvement in overall MRCP exam series pass rates by the end of CT2, bringing Coventry and Warwickshire (77.3%) in line with the national average (78%).
Furthermore, trainee feedback on exposure to, and satisfaction with, teaching demonstrated notable improvements following interventions.
Those who have delivered teaching or mentoring have also benefited from this project as they have had the opportunity to hone their leadership skills, keep up to date with knowledge and improve on teaching experience.
We have identified limitations in this quality improvement project and have recommended appropriate solutions. The most difficult limitation to overcome involves identifying which of multiple facets of improvement has been the most beneficial. Feedback from trainees in 'free text' suggests that having a peer-mentor to guide them through the stressful exam process, more senior support and regular bedside-teaching opportunities with performance feedback were most appreciated.
Although we identified that centralising the consultant-led bedside-teaching sessions potentially inhibits self-directed learning, it avoids consultants being overrun with individual requests and facilitates a workable rota for all to attend. This is particularly important given the limited availability of senior doctors owing to other commitments. As previously discussed, the use of a first-come-first-serve rota potentially limits less motivated trainees to fewer sessions. However, when used in conjunction with the peer-mentors through an interactive forum, these trainees can be identified and offered extra support.
Utilisation of a shared forum was key to the success of the framework, in that it enabled trainees to efficiently target their practice and avoid wasting precious revision opportunities between ward commitments. Its success though, relies on the collaborative input of all involved throughout. In our experience this is not always easy to maintain.
Existing literature supports the concept that focused strategic learning improves exam outcomes.5 The majority of CMT classroom-based teaching is blueprinted nationally, in order to ensure a broad foundation of core clinical knowledge. This however potentially limits its usefulness to PACES exam preparation. Negotiating with consultants delivering classroom sessions to include focused strategic learning during their sessions has allowed for the CMT curriculum to be intertwined with the PACES curriculum, thus ensuring dual relevance.
Improving local trainee satisfaction and MRCP PACES pass rates, as well as enhancing teaching opportunities, raises the profile of the trust as a whole. This can ultimately lead to improved recruitment and retention of trainees, while justifying local deanery funding for education.
We feel that our innovative and cost-effective interventions could be used on a wider scale to improve progression through core medical training and onto higher-level training. Furthermore, it encourages teamwork and leadership, both of which are essential skills required throughout a career in medicine. Ultimately our tailor-made teaching framework should not be used alone and should be used in conjunction with proactive self-directed learning, with significant preparation time.
Acknowledgments
We thank Dr Narendra Reddy for reviewing our manuscript and providing us with feedback prior to submission.
Footnotes
Contributors RJ and TR planned and led the quality improvement project and wrote the whole article together as co-first authors. NM supervised the project as a lead consultant and reviewed and edited the article.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.