Article Text
PDF
Abstract
Background MRI interpretation and accurate radiological staging are crucial to the important treatment decisions and a consequent successful patient outcome in rectal cancer.
Aims To investigate the effect of intensive training on rectal cancer MRI staging performance of radiologists and the impact of different course elements on learning outcomes.
Methods In this prospective intervention study, 17 radiology specialists and 1 radiology registrar participated in a training programme including a 6-hour imaging workshop, a 3-hour session of individual feedback and independent MRI readings of primary rectal cancer cases. Their rectal MRI interpretive performance was evaluated through repeated readings of 30 training cases before and after each course element and a time interval with no educational intervention. A proforma template for MRI staging of primary rectal cancer was used and the results were compared with a reference standard of an expert panel. Participants repeatedly reported on confidence scores and self-assessed learning outcome. Outcomes were analysed using mixed-effects models.
Results At baseline the quality of rectal MRI assessment varied significantly, with a higher interpretive performance among participants with shorter radiological experience (10.2 years vs 19.9 years, p=0.02). The ability to perform correct treatment allocation improved from 72% to 82% (adjusted OR=2.36, 95% CI 1.64 to 3.39). The improvement was largely driven by the participants with lower performance at baseline and by prevention of overstaging. Individual feedback had a significant impact on the improved interpretive performance (adjusted OR=1.82, 95% CI 1.27 to 2.63), whereas no significant change was seen after workshop or case readings only. Confidence scores increased significantly during training.
Conclusions Targeted and individualised training improves the rectal cancer MRI interpretive performance essential to successful patient treatment, especially among radiology specialists with lower performance at baseline.
- Continuing education, continuing professional development
- Healthcare quality improvement
- Health professions education
- Quality improvement
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- Continuing education, continuing professional development
- Healthcare quality improvement
- Health professions education
- Quality improvement
WHAT IS ALREADY KNOWN ON THIS TOPIC
Accurate staging of primary rectal cancer on MRI is essential for guiding the multidisciplinary approach to therapy. Understaging as well as overstaging can indirectly lead to increased morbidity and mortality in patients with rectal cancer.
WHAT THIS STUDY ADDS
We demonstrate that rectal cancer MRI staging performance of radiologists can be improved by targeted and individualised training, mainly by prevention of overstaging.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Certification of radiologists and quality assurance of the radiological interpretation in rectal cancer MRI increases the quality of patient care and has the potential to reduce morbidity as well as the personal and national healthcare costs for patients with rectal cancer.
Introduction
When a patient is diagnosed with rectal cancer, careful and systematic MRI interpretation and accurate radiological staging are crucial to the important treatment decisions and a consequent successful patient outcome. This information is shared within the multidisciplinary teams (MDT) of specialists, who are dependent on accurate preoperative staging to provide the optimal therapeutic strategy for each patient, taking into account patient comorbidity and treatment preferences. Radiological assessment is therefore integral to this process, with high-resolution MRI being the mandated imaging tool for local staging and surgical planning of rectal cancer. Trained radiologists can accurately predict the depth of extramural spread, involvement of the anticipated surgical resection margin and vascular invasion, and therefore, help identify patients at high risk of local and distant disease recurrence.1 2
MRI for the preoperative assessment and treatment planning of patients with newly diagnosed rectal cancer has been mandatory in Denmark and other countries for almost 20 years and during this time circumferential resection margin (CRM) positivity rates have fallen. Inaccurate preoperative staging can have serious implications for a patient with rectal cancer. Overstaging may cause overuse of neoadjuvant therapy with inevitable significant added morbidity without any corresponding beneficial effect of the treatment.3–8 Equally, understaging of the tumour may result in a higher risk of local and distant failure. A previous study showed that reporting and technical performance of MR scans in rectal cancer improved after an MDT course with workshop and on-site visits for radiologists.9 However, the study also documented that an improvement in the ability to interpret rectal MRI would require more intensive training efforts.
The aim of this study was to examine the effect of an intensive rectal cancer MRI staging training programme including workshop, individual feedback and repetitive independent case readings, on radiologists’ ability to perform correct treatment allocation. We hypothesised that this aim could only be achieved through a targeted and individualised training effort. The study aimed to evaluate their interpretational performance on key MRI findings in local staging of primary rectal cancer and their self-reported confidence scores and learning outcome, before and after training. Importantly, we measured the impact of different course elements on patient care by measuring the ability to indicate the correct patient treatment according to radiology assessment.
Methods
Course management
The course management consisted of one professor of surgery and one associate professor of radiology with previous experience in running MDT courses in rectal cancer, two staff members from Centre of Competence Development, Central Denmark Region (one head of office with a master in systemic leadership and organisational development and one secretary) and one surgical registrar for setting up a database for reporting and for technical support.
Participants
Consultant radiologists and radiology registrars with special interest in rectal MRI were invited to participate in a rectal cancer MRI staging training programme. The invitation was mailed to the heads of radiology departments all over Denmark or directly to radiology specialists known by the course management in mid-November 2016 and contained information about the purpose, structure and content of the course including periods with independent rectal cancer MRI case readings. The course had a maximum limit of 20 participants. Participants who failed to complete and submit their results from the case readings within the given time frame were excluded.
MDT development course programme
The course was held during the first half of 2017 and consisted of three modules; an introduction, a workshop and a session of individual feedback. It was led by two consultant radiologists with 12–25 years of practical, scientific and educational experience in rectal MRI. Repeated periods of independent rectal MRI case readings were included. The course structure is shown in figure 1.
Flow chart of training programme.
Introduction
At the introduction, the national Danish guidelines for diagnosis, staging and treatment of rectal cancer (preoperative chemoradiotherapy or direct surgery) were outlined including the level of evidence underlying the recommendations.10 The participants received a CD-ROM (compact disc-read only memory) containing 30 MRI cases of primary rectal cancer for independent case readings and received instructions regarding the use of REDCap—an electronic reporting tool designed to support data collection for research studies. Finally, each participant was scheduled to a session of individual feedback, the timing of which was determined by prior randomisation of the participants into two groups (figure 1).
Workshop
The 6-hour imaging workshop included information on techniques to ensure high-quality rectal MRI and instructions in pelvic anatomy and key anatomic landmarks, principles for measuring the distance from the tumour to the anal verge, accurate T staging of rectal tumours based on axial tumour location and tumour morphology, assessment of margins including principles for measuring the distance between the tumour and the mesorectal fascia (MRF)/levator muscles/sphincter complex and evaluation of lymph node disease, extramural vascular invasion (EMVI) and extramural discontinuous vascular deposits. Examples of rectal cancer MRI were used to illustrate key points within rectal cancer MRI staging.
Individual feedback
Feedback was given in an in-person format (physical attendance) between each course participant and one specific course leader during a 3-hour individual session including one-to-one supervision. The session was structured according to John Hattie’s principles of feedback11 and was based on the participant’s results from the first two rounds of case readings. The feedback included information on the level of the participant’s performance, clarification of recurring inaccuracies and personalised training through rectal MRI cases, selected on basis of identified issues to improve the participant’s performance. The session was concluded with a summary of the learning objectives and focus areas for the participant.
Key staging elements with direct impact on patient treatment decisions addressed during workshop and the individual feedback are shown in online supplemental appendix 1.
Supplemental material
Independent MRI case readings
Thirty training cases were extracted from a collection of MRI examinations of primary rectal cancer established during a previous postgraduate multidisciplinary development programme.9 These MRI examinations comprised a representative sample of tumours of varying stages, extent of extramural disease and height in the rectum.9 Examinations were performed according to the protocol by Brown et al,12 and were anonymised in DICOM (Digital Imaging and Communications in Medicine) format on CD-ROMs. These cases were not presented in any other part of this course.
Participants were expected to complete all four rounds of case readings during the 5-month course period; at baseline, after the workshop, after feedback in the first randomisation group and after feedback in the second randomisation group. Approximately 5 weeks were scheduled for each round of case readings, separated by time periods of approximately 2 weeks without case readings. The participants performed the independent case assessments using a PACS (picture archiving and communication system) workstation at their own centre, and received an email with a link to the electronic reporting tool at the beginning of each case reading period. The reporting tool comprised a pro forma template for MRI staging of primary rectal cancer (online supplemental appendix 2) for registration of key MRI findings expected to be included in the MRI case assessments (online supplemental appendix 3). The participants finalised their assessment by concluding whether the patient should be offered neoadjuvant therapy or direct surgery, according to the Danish guidelines.
Assessments were compared with a reference standard based on independent readings with subsequent consensus by an expert panel. The expert panel consisted of three consultant radiologists with 12–25 years of daily practical and scientific experience in rectal MRI.
Participant self-evaluation
Participants rated their self-assessed learning outcome from each course module and from each period of case readings using a 5-point numeric scale (1=poor outcome, 5=excellent outcome). After each period of case readings, they rated their self-assessed inclination to (1) take an active part in the MDT meeting and (2) enter into dialogue with other MDT members regarding the MRI assessment, using a 5-point numeric scale (1 = ‘none’, 5 = ‘very much’).
Statistical analysis
For overall performance level, tumour category, tumour stage, tumour height category, lymph node status, EMVI status, tumour-free MRF, the outcome was agreement with the reference standard or not. Tumour-free MRF was derived from the dichotomisation of the minimum distance from the extramural spread to the mesorectal fascia, at a 5 mm and 1 mm level, and was evaluated for cases where the reference standard was >1 mm and ≥5 mm, respectively. Proportions were calculated from submitted case evaluations only, as indicated by absolute numbers with a varying denominator in parentheses. For tumour location, a clock face centre of the tumour was derived from the clock face interval (eg, tumour centre at 2 o’clock for a tumour located from 10 o’clock to 6 o’clock), and tumour location was evaluated as the distance from that of the reference standard. Binary outcomes were analysed using a mixed-effects logistic model, and continuous responses (tumour location, confidence scores and learning outcomes) were analysed using a mixed-effects linear model, both with crossed random effects by participant and case, and round as a fixed effect. The participants were divided into two equally sized groups based on their baseline assessment of the 30 MRI cases of primary rectal cancer: a group of higher performance (n=9) and a group of lower performance (n=9). The purpose of randomisation according to timing of feedback was to measure the effect of individual feedback compared with a control group. However, by chance the participants’ baseline performance level turned out to be unequally distributed among the two randomisation groups. To compensate for that the educational effect was measured comparing the results of the case readings in rounds 2 and 4 (figure 1). A p<0.05 was considered to represent statistical significance. All statistical analyses were performed using STATA (STATA, release V.16.1 IC, StataCorp).
Patient and public involvement
Patients or the public were not involved in the design, conduct, reporting or dissemination plans of our research.
Results
Characteristics of study participants
Among 20 participants, 17 radiology specialists and one radiology registrar with 4–30 years of radiology experience completed the course programme. Two participants were excluded; one unregistered from the course for personal reasons and one failed to complete the case readings at baseline. Table 1 summarises the characteristics of the 18 remaining participants. Ten participants (56%) had 2 years or less experience in rectal MRI. Two-thirds of the participants regularly attended MDTs.
Characteristics of the participating 17 radiology specialists and one radiology registrar
Participants with lower performance had an average of 19.9 (SD 8.2) years of radiological experience and 3.2 (SD 2.1) years’ experience in rectal MRI, whereas the higher performing group had an average of 10.2 (SD 6.9) years of radiological experience and 1.8 (SD 2.8) years’ experience in rectal MRI (p=0.02 and p=0.24).
Correct treatment allocation
In this training situation, patients would have received an incorrect treatment in 28% of the evaluations (144/522) before the educational programme, primarily due to overstaging (57/144, 40%) and this was more than halved (19/98, 19%) at the end of the course (p=0.001). Overall, the participants’ ability to correctly allocate patients to either preoperative chemoradiotherapy or direct surgery improved significantly from 72% (378/522) to 82% (436/534) of evaluations (adjusted OR=2.36, 95% CI 1.64 to 3.39). The participants with lower performance improved from 61% to 76% (adjusted OR=3.11, 95% CI 1.92 to 5.05), the corresponding results for the higher performing group being 84% to 87% (adjusted OR=1.60, 95% CI 0.90 to 2.84). Wrong treatment allocation due to lack of understanding of guidelines accounted for 33% (48/144) of the erroneous assessments at baseline and 23% (23/98) at the end of the course (p=0.08).
Parameters of importance for correct treatment allocation
The participants’ performance on key staging parameters and prognostic factors are shown in table 2 for all participants and table 3 for participants divided by performance status. The course had a significant effect on the ability to correctly identify the location of the tumour in the rectal wall, to identify proper tumour category, to correctly assess EMVI status and to predict a tumour-free MRF, both within a 5 mm level and a 1 mm level. For tumour location, the effect was observed in both performance groups. Whereas the improved ability to detect EMVI could be attributed solely to the higher performing participants, the significant effect on the remaining improved parameters was largely driven by the lower performing group including an improved ability to categorise tumour height correctly. No effect was observed regarding the ability to correctly distinguish between early and advanced tumours or to detect lymph node involvement.
Radiologists’ performance on key staging parameters and prognostic factors in rectal MRI
Radiologists’ performance on key staging parameters and prognostic factors in rectal MRI, stratified by performance status
Impact of different course modules on correct treatment allocation
An increase in overall performance level was registered after individual feedback (75% (398/529) at round 2 vs 82% (436/534) at round 4; adjusted OR=1.82, 95% CI 1.27 to 2.63), whereas no statistically significant learning outcome was registered after workshop (72% (378/522) at round 1 vs 75% (398/529) at round 2; OR=1.29, 95% CI 0.91 to 1.83) or from case review alone (‘intervention group’ who received feedback between rounds 2 and 3: 82% (193/234) at round 3 vs 82% (192/235) at round 4; OR=0.94, 95% CI 0.53 to 1.68; ‘control group’ who received feedback between rounds 3 and 4: 77% (230/297) at round 2 vs 77% (229/299) at round 3; OR=0.94, 95% CI 0.58 to 1.52).
Participant self-evaluation
At the end of the course, participants felt more qualified to take active part in MDT meetings and to enter into dialogue with other MDT members (table 4).
Radiologists’ self-evaluated confidence scores and learning outcome scores, overall and stratified by performance status
The participants estimated that learning outcomes were greatest after the individual feedback (mean score: 4.4±0.7 after general introduction vs 4.8±0.4 after individual feedback, p=0.02). Their self-evaluated learning outcomes after the workshop did not differ significantly from the learning outcomes after the general introduction (mean score: 4.4±0.7 after general introduction vs 4.6±0.5 after workshop, p=0.22). During the course, the participants reported to gain an increasing learning outcome throughout the rounds of case readings and that they acquired improved routines in rectal MRI assessment (table 4).
Discussion
In modern management of patients with rectal cancer, local staging on rectal MRI has become crucial to the pathway of care. Proper treatment stratification depends on accurate assessment of important prognostic MRI findings. Among these, the distance from the anal verge to the lower border of the tumour, tumour category and tumour’s relationship to the mesorectal fascia/levator muscle are important considerations.13 Previous studies have described the risk of staging failures due to overestimation of T category,14–16 which supports our finding of tumour overestimation being the leading cause of incorrect treatment selection at baseline. We observed many instances of radiologic overstaging that we could overcome through careful explanation of predictable tumour morphology and behaviour.
This study demonstrates that with targeted and individualised training there is a significant impact on correct allocation of patients for their treatments. The improvement from 72% to 82% (adjusted OR=2.36, 95% CI 1.64 to 3.39) was largely brought about by prevention of overstaging. This has implications for reducing morbidity as well as the personal and national healthcare costs for patients. The avoidance of unnecessary radiotherapy is also important for patients who may suffer other pelvic malignancies in future and can thus still receive radiotherapy.
The greatest improvement was achieved within the participants with lower baseline performance. Individual feedback had a significant impact on the improved interpretive performance, whereas no significant change was seen after workshop or case readings only. Correspondingly, the participants scored learning outcome from individual feedback highest compared with the outcome from workshop and case readings, although they did report on a learning outcome from case readings as well. Confidence scores increased during training. The improved confidence scores were only significant in the participants with higher performance, who also had the lowest confidence scores at baseline.
Learning outcomes were different among the participants with lower and higher baseline performance, respectively. The participants with lower performance significantly improved their rectal MRI interpretation with regard to proper treatment stratification and key staging items. Already at baseline, the participants with higher performance had accuracy levels either matching or even outperforming the final accuracy levels of the participants with lower performance, and as the only two image interpretive features, they were significantly more likely to identify the location of the tumour in the rectal wall and to detect EMVI at the end of the course. However, the participants with higher performance had a greater gain in confidence over time, suggesting that the different learning elements of the course have confirmed these participants in the quality of their work and thus strengthened the assurance of their assessment. Lack of confidence and ambiguity among radiologists may result in uncertain interpretation and vague reporting with the risk of adverse clinical implications such as poor function within the MDT, delay in making the correct treatment decisions or even incorrect diagnosis or treatment.17
Surprisingly, the study showed that the participants with higher baseline performance had shorter radiological experience. This finding was unexpected and counterintuitive, and we can only speculate on the explanation for this outcome. Despite their more recent training, interpretation of rectal MRI is still not included in the core curriculum for radiology trainees in Denmark and cannot explain the mismatch between experience and performance at baseline. The inexperienced readers of our study possibly embraced the challenge of the case readings with greater curiosity and had an acceptance of a prolonged time spent per case than the experienced readers. Our findings indicate that long-term radiological practice, potentially including MRI experience within other subspecialty areas, does not necessarily ensure the quality of rectal MRI interpretations.
A positive effect of educational programmes and audits on the surgical quality in rectal and colon cancer has previously been documented.18–22 In the field of MRI of rectal cancer, interventional studies have shown that the technical quality of rectal MRI9 and the completeness of rectal cancer staging reports9 23 can be improved through multidisciplinary development courses, on-site visits and proforma reporting. This is the first study to measure the effect of training on the MRI assessment of rectal cancer and to systematically examine the impact of different course elements on learning outcomes.
Previous studies have demonstrated a positive effect of experience24 and training on MRI interpretive accuracy.25–29 The design of the studies varied with regard to caseload, the extent and nature of the educational intervention and how the effect was measured. Only one study investigated the effect of interactive, face-to-face feedback25 and another study focused on the optimal strategy for learning by examining the effect of different educational elements.26 Rosenkrantz et al compared continual feedback versus self-directed learning in the learning curve for tumour detection on prostate MRI and found no difference.26 We believe that during lectures, the teacher and trainees will often focus on overview, information and memory storage rather than thinking, understanding and translating content and theory into practice. Through the dynamics of individual, face-to-face feedback used in our study, the expert had the opportunity to encourage or maintain the attention, commitment and enthusiasm of each participant with the purpose of strengthening his or her rectal MRI interpretive weaknesses and defining areas of focus for improvement. The improvement in correct treatment stratification after individual feedback in our study may be explained by the more focused and profound nature of this training intervention.
Our study has some limitations. Experienced and inexperienced participants may have had different incentives to accept the course invitation leading to self-selection bias. Experienced participants may be those with a need to raise their performance level and inexperienced participants those with a drive and the personal resources to take on a new modality or type of examination. With a caseload of 120 case readings, our study included a level of self-directed learning comparable to others (70–200 cases),25–29 but unlike these studies, we let our participants review the same cases four times. This way we eliminated the influence of potential differences in case difficulty on our analysis. Since the participants never had access to the reference standard, we find cumulative interpretive advantage due to prior case knowledge unlikely, which is supported by our finding of no improvement in interpretive performance after repeated case readings without any educational intervention. The choice of an expert panel consensus over pathological results as the reference standard is due to the fact that an estimated one-third of Danish patients with primary rectal cancer are treated with neoadjuvant chemotherapy due to particularly locally advanced tumours. We included examples of such tumours in the case set to make sure that the participants would be able to evaluate patients in need of neoadjuvant therapy correctly. The choice of an expert panel consensus over pathological results as the reference standard may possibly explain the high diagnostic accuracy for lymph node staging, as it is known that the diagnostic accuracy for lymph node staging compared with pathology is poor regardless of the choice of radiological modality. Although the framework for the individual feedback sessions was set in advance, providing feedback in a completely standardised form was difficult. Feedback is an interaction between people and curious, enthusiastic and perhaps skilled participants could achieve a greater learning outcome than less motivated and/or less skilled participants. The performance level of the participants was determined by their ability to make correct treatment stratification based on their case reviews, which is not the primary task of the radiologist in MDT meetings, but provided us with a composite measure of clinical relevance to monitor their ability to identify and interpret several relevant imaging features. We believe that knowledge of guidelines is crucial and that it is incumbent on the radiologist to have a clear understanding of the impact that their own interpretations and statements have on patient care. The overall effect of training was measured shortly after the end of the course and it is unclear whether the achieved level of interpretive performance among the participants will be maintained in the long term, which would require longitudinal skills assessment after for example, 3, 6 and 12 months. Finally, our findings may not apply to the radiology profession at large, but given the variation in the participant baseline performances, we believe that the study outcome can be reproduced in a similar population of radiologists with an evident training need or interest in MRI local staging of rectal cancer.
In the assessment of rectal MRI, the radiologist now influences key management decisions, such as deciding on the extent of the surgical intervention or whether the patient would benefit from preoperative chemoradiotherapy or not—decisions that are essential to the oncological result, functional outcome and quality of life of the patient. Therefore, accurate rectal MRI interpretation as well as a confident assessment delivery are crucial in the treatment decision-making process, and radiologists should be trained in both. Our study highlights the need for both quality assurance of the radiological interpretation in MRI of rectal cancer and training of radiologists in the identification of key staging items before assessing rectal MRI in clinical practice. Sufficient resources must be allocated to audits and the certification of radiologists in rectal MRI assessment to improve the performance level of those who need it through training including individual feedback.
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants but according to Danish law, the study classifies as a quality assessment study and need not to be notified to a research ethics committee. Participants gave informed consent to participate in the study before taking part.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @P_Bondeven
Contributors Study concept and design: PB, GB, SL and BGP; data collection: PB and BGP; data analysis and interpretation: SB, TKG, SL, BGP; drafting the article: SB; critical revision of the article: PB, TKG, GB, SL and BGP; final approval: all authors; study guarantor: BGP.
Funding This research was funded by The Danish Cancer Society and Health Research Foundation of Central Denmark Region.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.