Intended for healthcare professionals

Learning In Practice

What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review

BMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7473.1017 (Published 28 October 2004) Cite this as: BMJ 2004;329:1017
  1. Arri Coomarasamy, specialist registrar in obstetrics and gynaecology (arricoomar{at}blueyonder.co.uk)1,
  2. Khalid S Khan, consultant obstetrician and gynaecologist1
  1. 1 Education Resource Centre, Birmingham Women's Hospital, Birmingham, B15 2TG
  1. Correspondence to: A Coomarasamy

    Abstract

    Objective To evaluate the effects of standalone versus clinically integrated teaching in evidence based medicine on various outcomes in postgraduates.

    Design Systematic review of randomised and non-randomised controlled trials and before and after comparison studies.

    Data sources Medline, Embase, ERIC, Cochrane Library, DARE, HTA database, Best Evidence, BEME, and SCI.

    Study selection 23 studies: four randomised trials, seven non-randomised controlled studies, and 12 before and after comparison studies. 18 studies (including two randomised trials) evaluated a standalone teaching method, and five studies (including two randomised trials) evaluated a clinically integrated teaching method.

    Main outcome measures Knowledge, critical appraisal skills, attitudes, and behaviour.

    Results Standalone teaching improved knowledge but not skills, attitudes, or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes, and behaviour.

    Conclusion Teaching of evidence based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes.

    Introduction

    The knowledge and skills needed for critical appraisal of literature and practice of evidence based medicine (EBM) are often taught through standalone courses and workshops in classrooms away from clinical practice. An early (and now out of date) review showed that in these educational interventions, gains in knowledge were poorer among postgraduates than undergraduates.1 Without reinforcement in subsequent practice, even the modest knowledge gains from such courses are likely to deteriorate over time. Postgraduate and continuing education received in this way is unlikely to lead to any meaningful changes in clinical care. In theory, teaching and learning that is integrated into routine practice should bring greater benefits.

    We examined the effects of postgraduate teaching in EBM and explored the effect of the teaching methods (whether standalone or integrated into clinical practice) on various outcomes.

    Methods

    We searched Medline, Embase, ERIC, Cochrane controlled trials register (CCTR), Cochrane database of systematic reviews (CDSR), database of abstracts of reviews of effects (DARE), Health Technology Assessment database (HTA), Best Evidence, Best Evidence Medical Education (BEME), and Science Citation Index (SCI) using the following search terms and their word variants: “evidence”, “critical”, “appraisal” or “journal club” combined with “AND” to “teach$”, “learn$”, “instruct$”, or “education”. We also searched reference lists of known systematic reviews.14 The final electronic search was conducted in April 2004.

    We included studies that evaluated the effects of postgraduate EBM or critical appraisal teaching compared with a control group or baseline before teaching, using a measure of participants' learning achievements or patients' health gains as outcomes. Learning achievement was assessed separately for knowledge, critical appraisal skills, attitudes, and behaviour.

    Knowledge relates to issues such as remembering materials as well as grasping the meaning—for example, defining and understanding the meaning of number needed to treat (NNT). If this knowledge can then be applied accurately to given problems this will be regarded as a gain in critical appraisal skills—for example, the ability to generate a number needed to treat when baseline risks and odds ratios are provided. Spontaneously acknowledging a need for the use of a certain piece of knowledge or skill in practice will be regarded as a change in attitude—for example, recognising without prompting the need for different NNTs for different clinical scenarios and intending to calculate the respective NNTs for different levels of risk. Finally, a change in behaviour occurs when one seeks the necessary information and applies the knowledge and skills to solve the issue in practice—for example, searching the literature, finding relevant baseline risks and odds ratios, and calculating necessary NNTs to guide clinical practice.

    We excluded studies on teaching of EBM in undergraduate education. We graded the quality of the evidence in these articles as either level 1 (randomised controlled trials) or level 2 (non-randomised studies that either had a comparison with a control group or a before and after comparison without a control group). We could not use meta-analysis because of the obvious heterogeneity in features, quality, and assessment tools in individual studies. We weighted our conclusions by quality of methods.

    Results

    The literature search identified 42 potentially useful citations. We examined the full manuscripts of all of these citations and identified 23 articles (including one yet to be published) relevant for inclusion in our review (tables 1 and 2).526 Of the 19 articles that we rejected, 15 examined populations unsuitable for our review (for example, undergraduates or non-medical staff), two examined an unsuitable intervention (for example, the effect of dissemination of EBM guidelines rather than teaching of EBM), and two were reviews of primary studies already included in our review. Of the 23 included studies, four were randomised (level 1) trials (R Taylor, personal communication),5 22 23 and 19 were non-randomised studies (level 2), comprising seven non-randomised controlled studies and 12 before and after comparison studies. Teaching methods included workshops, seminars, and journal clubs alone or in various combinations (tables 1 and 2). Eighteen studies (R Taylor, personal communication),521 including two randomised trials (R Taylor, personal communication)5 (level 1), evaluated a standalone teaching method (table 1), while five studies,2226 including two randomised trials,22 23 evaluated an integrated teaching method (table 2). Integrated teaching focused on training in EBM components (such as question formulation, literature searching, and critical appraisal) in real time clinical ward rounds or basing the EBM teaching sessions on encounters with patients on the wards and in clinics. The outcomes reported were knowledge, skills, attitude, and behaviour. None of the studies assessed patients' health.

    Table 1

    Primary studies of standalone teaching of critical appraisal skills and EBM in postgraduate trainees

    View this table:
    Table 2

    Primary studies of clinical practice based (integrated) EBM teaching among postgraduate trainees

    View this table:

    Does knowledge improve?

    Of the 23 studies, 17 assessed knowledge (fig 1). The weight of evidence, including the evidence from the three randomised trials (R Taylor, personal communication)5 that reported on this outcome, indicated an improvement in knowledge from both teaching methods.

    Fig 1
    Fig 1

    Changes in knowledge, skills, attitude, and behaviour after critical appraisal skills or EBM teaching, grouped by quality of studies. Data presented as 100% stacked bar chart with numbers inside bars indicating number of studies that provided information for a particular outcome (see tables 1 and 2 for details of each study)

    Do critical appraisal skills improve?

    Nine of the studies assessed critical appraisal skills (fig 1). The only randomised trial that reported this outcome in the standalone group did not find an improvement (R Taylor, personal communication). Of the six non-randomised studies that reported this outcome in the standalone group, three found an improvement. On the other hand, both the studies, including a randomised trial,22 which reported skills as an outcome in the integrated teaching group found an improvement. Therefore, on balance, there is weak evidence that standalone courses improve appraisal skills and good evidence, including evidence from a randomised trial,22 that the integrated approach leads to gains in appraisal skills.

    Do attitudes change?

    Six studies assessed change in attitudes, three each in both teaching groups (fig 1). In the standalone teaching group the three studies, including a randomised trial (R Taylor, personal communication), did not find a change in attitudes. In the integrated teaching group, however, all studies, including one randomised trial,22 found an improvement in attitudes. Therefore there is compelling evidence that teaching integrated into clinical practice changes attitudes about the role of EBM or critical literature appraisal in medicine, while a standalone approach does not.

    Does behaviour change?

    Fourteen studies assessed the outcome of behavioural change after EBM or critical appraisal teaching, including four randomised trials, two in each teaching group (fig 1). The two randomised trials (R Taylor, personal communication)5 in the standalone group found no change in behaviour, and both randomised trials in the integrated teaching group observed a change in behaviour.22 23 These findings from the randomised evidence were found to be consistent with the findings of the non-randomised studies, with four of seven studies in the standalone group not showing a change in behaviour and all three non-randomised studies in the integrated teaching group showing benefit. The improvements noted in behaviour included changes in reading habits24 and choice of information resources,25 as well as substantial outcomes such as changes in management of patients23 26 and guidelines.26

    Do patients' health outcomes improve?

    None of the studies evaluated health outcomes. As the integrated teaching approach showed that it was possible to change behaviour, however, this holds the potential for improving health outcomes. The translation of changes in behaviour into complex outcomes such as better care of patients may not be a linear one, as improving care is likely to be affected by many factors, only one of which may be the practice of EBM. Moreover, such improvements in patients' outcomes are likely to occur over a long period of time and among many other changes, making them difficult to identify in studies of evaluation of teaching or practice of EBM and critical appraisal skills.

    Discussion

    To our knowledge, a comparison of the effects of standalone versus integrated teaching in critical appraisal skills and EBM has not been done before. In addition to not making the distinction between standalone and integrated courses,24 several existing reviews have generally considered undergraduates and postgraduates together. There is empirical evidence, however, that the outcomes of teaching EBM markedly differ between undergraduates and postgraduates, with smaller gains in knowledge among the postgraduates.1 Moreover, adult learning theory suggests that the determinants of learning in the two groups are different, with postgraduate learning tending to be driven by self motivation and relevance to clinical practice, whereas undergraduate learning is generally driven by external factors such as curriculum and examinations.27 This suggests that effectiveness of educational interventions in teaching critical appraisal skills and EBM should be evaluated separately for postgraduate and continuing education, which we have done.

    Studies examining the effectiveness of educational interventions may suffer from various weaknesses. Even a randomised controlled study, which is generally regarded as the optimum method for settling questions of effectiveness, is not immune to many of these weaknesses. These weaknesses include difficulty with standardising the educational intervention(s), contamination between the two arms of a study, inability to blind the study participants and the teachers from the educational intervention(s) leading to selective cointervention, and finally difficulty with measuring outcomes due to the lack of valid and reliable assessment tools. Some of these factors make randomised trials unfeasible in educational settings, thus necessitating other designs such as non-randomised controlled and before and after studies. We included all three designs in our review.

    We have shown that while standalone teaching and integrated teaching are both effective in improving the knowledge base, it is clinically integrated teaching of EBM that is likely to bring about changes in skills, attitudes, and behaviour (fig 2). Changes in attitudes are likely to be important in bringing about sustained changes in behaviour, which may ultimately benefit care of patients. It is therefore important that teachers of critical literature appraisal and EBM consciously find ways of integrating and incorporating teaching of critical appraisal into routine clinical practice. Where resources and facilities are available, such teaching can form part of a “real time” ward round with the dual purposes of teaching EBM skills and attempting to improve care with best available evidence.28 29 If the provisions for real time teaching are not available, then even traditional teaching settings, such as a journal club,30 can be adapted to be based on real and current clinical problems, thus illustrating that the process is not merely an academic exercise but that it informs care.

    Fig 2
    Fig 2

    Reasons why integrated teaching may achieve better outcomes than standalone teaching

    The purpose of EBM is to integrate best research evidence with clinical skills and patients' values and preferences.31 Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. This is because the ultimate aim of improving care could not be achieved with changes in knowledge and skills alone—it would also require changes in attitudes and behaviour. Critical appraisal and EBM teaching that is integrated into clinical practice seems more effective in improving such substantial outcomes including behavioural changes. Teachers of critical appraisal and EBM should aim to bring teaching out of classrooms into the clinic, but this will require a greater effort. Future studies should focus not only on substantial outcomes such as behaviour and health outcomes but also on longer term outcomes as there is the potential for decay of learning outcomes over time.

    Summary points

    Critical literature appraisal and evidence based medicine (EBM) can be taught through standalone courses or through instructional methods that incorporate teaching into routine clinical care

    Several randomised and non-randomised studies have evaluated the effects of teaching EBM to postgraduates

    Both standalone courses and integrated teaching improve knowledge

    Improvements in skills, attitudes, and behaviour, however, come about when teaching is integrated into clinical practice; standalone courses bring about no change

    It is important to incorporate EBM teaching into clinical practice, but this would require a sustained effort well beyond standalone instruction

    Footnotes

    • Contributors AC and KSK conceived, conducted, and wrote the review. AC is the guarantor.

    • Funding None.

    • Competing interests AC and KSK have a grant from West Midlands Deanery to teach EBM to specialist registrars in the region, as well as a European Union Grant (LSE031068WM2) to promote EBM among small to medium size enterprises that supply the NHS.

    • Ethical approval Not required.

    References

    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.
    6. 6.
    7. 7.
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16.
    17. 17.
    18. 18.
    19. 19.
    20. 20.
    21. 21.
    22. 22.
    23. 23.
    24. 24.
    25. 25.
    26. 26.
    27. 27.
    28. 28.
    29. 29.
    30. 30.
    31. 31.
    View Abstract