You are here

  • Home
  • Archive
  • Volume 5, Issue 12
  • Achieving change in primary care—effectiveness of strategies for improving implementation of complex interventions: systematic review of reviews

Article Text

Article menu
XML
Health services research
Research
Achieving change in primary care—effectiveness of strategies for improving implementation of complex interventions: systematic review of reviews
  1. Rosa Lau1,
  2. Fiona Stevenson1,
  3. Bie Nio Ong2,
  4. Krysia Dziedzic2,
  5. Shaun Treweek3,
  6. Sandra Eldridge4,
  7. Hazel Everitt5,
  8. Anne Kennedy6,
  9. Nadeem Qureshi7,
  10. Anne Rogers6,
  11. Richard Peacock8,
  12. Elizabeth Murray1
  1. 1eHealth Unit, Department of Primary Care and Population Health, University College London, London, UK
  2. 2Arthritis Research UK Primary Care Centre, Research Institute for Primary Care Sciences and Health Sciences, Keele University, Keele, UK
  3. 3Health Services Research Unit, University of Aberdeen, Scotland, UK
  4. 4Centre for Primary Care and Public Health, Queen Mary University of London, London, UK
  5. 5Primary Care and Population Sciences, Faculty of Medicine, University of Southampton, Southampton,UK
  6. 6Faculty of Health Sciences, NIHR CLAHRC Wessex, University of Southampton, Southampton, UK
  7. 7Division of Primary Care, University of Nottingham, Derby, UK
  8. 8Archway Healthcare Library, London, UK
  1. Correspondence to Rosa Lau; r.lau{at}ucl.ac.uk

Abstract

Objective To identify, summarise and synthesise available literature on the effectiveness of implementation strategies for optimising implementation of complex interventions in primary care.

Design Systematic review of reviews.

Data sources MEDLINE, EMBASE, CINAHL, Cochrane Library and PsychINFO were searched, from first publication until December 2013; the bibliographies of relevant articles were screened for additional reports.

Eligibility criteria for selecting studies Eligible reviews had to (1) examine effectiveness of single or multifaceted implementation strategies, (2) measure health professional practice or process outcomes and (3) include studies from predominantly primary care in developed countries. Two reviewers independently screened titles/abstracts and full-text articles of potentially eligible reviews for inclusion.

Data synthesis Extracted data were synthesised using a narrative approach.

Results 91 reviews were included. The most commonly evaluated strategies were those targeted at the level of individual professionals, rather than those targeting organisations or context. These strategies (eg, audit and feedback, educational meetings, educational outreach, reminders) on their own demonstrated a small to modest improvement (2–9%) in professional practice or behaviour with considerable variability in the observed effects. The effects of multifaceted strategies targeted at professionals were mixed and not necessarily more effective than single strategies alone. There was relatively little review evidence on implementation strategies at the levels of organisation and wider context. Evidence on cost-effectiveness was limited and data on costs of different strategies were scarce and/or of low quality.

Conclusions There is a substantial literature on implementation strategies aimed at changing professional practices or behaviour. It remains unclear which implementation strategies are more likely to be effective than others and under what conditions. Future research should focus on identifying and assessing the effectiveness of strategies targeted at the wider context and organisational levels and examining the costs and cost-effectiveness of implementation strategies.

PROSPERO registration number CRD42014009410.

  • PRIMARY CARE
  • Implementation
  • Systematic review

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.1136/bmjopen-2015-009993

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • To the best of our knowledge, this is the most comprehensive systematic review of reviews to examine the evidence on the effectiveness of single or multifaceted strategies for improving implementation of any kinds of complex interventions in primary care. As a result, 91 relevant reviews were included.

    • The review addressed a number of questions and was conducted using rigorous and transparent multistep reviewing methods.

    • The review reveals most of the existing research focused on strategies that addressed individual-level barriers. Most of these professional-level strategies were associated with small to modest improvement in professional practice and process outcomes. There is a lack of research on organisational-level strategies and context-level strategies.

    • It is possible that not all relevant primary research studies were captured by included reviews (especially those published recently), so some findings may be missed by concentrating on reviews.

    Introduction

    Internationally the pace of change in healthcare continues to be rapid with a drive to implement more clinically and cost-effective interventions to improve care. The need to reduce the delay in translating evidence-based interventions into every day clinical practice, known as the ‘second translational gap’, is widely acknowledged.1 ,2 Almost all changes to practice in primary care involve ‘complex interventions’, that is, interventions with multiple interacting components. They include changes in individual clinicians’ diagnostic and treatment approaches, in operational systems including information technology, altered divisions of labour between healthcare professionals and organisation of care; and require change at multiple levels.3 For instance, improving hand hygiene practices may appear simple but on closer inspection, it requires change at multiple levels.4 At the individual level, education might change attitudes and acceptance; local opinion leaders might be effective motivators; and reminders could prompt hand hygiene. Organisational structure and culture are important to facilitate change in hand hygiene practices including having adequate resources and infrastructure, for example, changing sink location to improve accessibility and convenience, having a continuous supply of hand wash and paper towels.4 It is also widely recognised that the policy context, professional and organisational context and political economic circumstances impacting on the healthcare environment impact on the design and implementation of complex interventions.5

    Ninety per cent of patient contacts take place in primary care in England. Primary care clinicians are generalists and often manage a set of undifferentiated symptoms or health problems; this requires a combination of wide ranging knowledge, clinical experience and sound judgement.6 Their roles and activities have changed and expanded over the recent years; for example, they are increasingly likely to be involved in care coordination for people with complex problems and areas of ‘specialist’ care (eg, diagnostics and minor surgery), as a result of the development of new medical technologies.7 Furthermore, these clinicians often work as part of a multiprofessional team. In England, primary care has been subject to particularly rapid change since the introduction of the Health and Social Care Act of 2012. All of these make this setting particularly challenging to implement ‘complex interventions’.

    Implementation strategies can be defined as techniques or methods aimed at improving or optimising the uptake and implementation of complex interventions into routine care.8 In this paper, we use this definition of implementation strategies, and use the term ‘strategy’ where we focus on implementation, to differentiate from the term ‘intervention’ which we use for the clinical intervention being implemented. The Cochrane Effective Practice and Organisation of Care (EPOC) group has developed the EPOC taxonomy of interventions designed to improve the delivery, practice and organisation of healthcare services. This taxonomy divides implementation strategies into (1) professional interventions (strategies targeted at professionals), such as printed educational materials, audit and feedback, educational meetings, computerised and non-computerised reminders, educational outreach visits, local opinion leaders; (2) organisational interventions (strategies targeted at the organisation), such as introducing a new role or way of working; (3) financial interventions (strategies targeted at the wider context) such as incentives or changes in reimbursement structure/method; and (4) regulatory interventions (strategies targeted at the wider context) such as introduction of or change in policy or legislation (see online supplementary file 2).9 Strategies may be used alone or in combination and as described in the EPOC taxonomy, may target health professionals, organisations or wider contextual issues.

    A systematic review of reviews was deemed to be the appropriate method to address this complex issue as the literature is substantial and heterogeneous, covering different clinical interventions, populations, clinical domains and outcomes. Existing reviews tend to focus either on a particular type of complex intervention (eg, introduction of new technologies or promoting uptake and use of guidelines) or on a particular health condition (eg, mental health or diabetes). No single review provides researchers, managers, clinicians or policymakers with coherent guidance to which strategies are effective at implementing change in primary care.

    We aimed to identify, summarise and synthesise the available review literature on the effectiveness of implementation strategies for improving uptake of complex interventions in primary care. This review addressed the following questions:

    1. What is the effectiveness of single strategies alone in improving uptake of complex interventions in primary care compared with no strategy or alternative single strategy?

    2. What is the effectiveness of (particular combinations of) multifaceted strategies in improving uptake of complex interventions in primary care, compared with no strategy, alternative single strategy or other combinations?

    3. Are multifaceted strategies more effective than single strategies (or vice versa)?

    4. What are the active components of strategies which appear to be associated with success?

    5. What is the cost-effectiveness of available implementation strategies?

    Methods

    Search strategy

    A comprehensive electronic search was performed in five databases: MEDLINE, EMBASE, Cumulative Index of Nursing and Allied Health (CINAHL), the Cochrane Library and PsycINFO. The search was performed by the primary reviewer (RL), supported by a specialist librarian (RP). The search strategy was developed using both medical subject headings, for example, ‘translational medical research’, ‘evidence-based practice’, ‘general practice’, ‘review’, ‘review literature as topic’ and free-text words, for example, evidence to practice, evidence practice gap, family doctor, implementation, adoption. Articles reported in English and published up to December 2013 were eligible for inclusion in this review. Citation searches were carried out in ISI Web of Science and reference lists of all included articles were screened for additional literature. Details of the search strategy for MEDLINE are provided in online supplementary file 3.

    Eligibility criteria

    Eligibility criteria were defined to enable transparent and reproducible selection of papers for inclusion, using the PICO framework.

    Population: reviews where at least 50% original studies came from primary care in developed countries.

    The Royal College of General Practitioners (RCGP) has defined primary care as “the first level contact with people taking action to improve health in a community”.10 Primary care teams are defined as teams or groups of health professionals that include a primary care physician (ie, general practitioners, family physicians and other generalist physicians working in primary care settings). Reviews exclusively on secondary care, dental practices, pharmacies or developing countries were excluded.

    Intervention: use of single or multifaceted strategies to improve implementation of complex interventions that focus on changing clinical practice (see online supplementary file 2). Studies that aimed to evaluate the efficacy or effectiveness of new models of care (eg, collaborative care model for depression care, case management or other integrated care services) were excluded. In addition, we decided to exclude reviews of clinical decision support systems because the focus of reviews in this area was improvement of clinical outcomes, rather than uptake or use. As this review focused on implementation with the aim of improving healthcare delivery and/or clinical practice, we excluded strategies aimed at directly changing patients’ behaviour.

    Comparator: usual care, no strategy or a different implementation strategy (either single or multifaceted).

    Outcome: degree of implementation measures, such as composite professional outcome (eg, adherence to desired practice), measures of process of care (eg, referral rates) and professionals’ performance (eg, prescribing, adherence to guidelines). Papers that reported outcomes related to patient health status or change in professionals’ knowledge (without any reference to behaviour or performance in practice) only were excluded.

    Study types: systematic reviews (structured search of bibliographic and other databases to identify relevant literature; use of transparent methodological criteria; presentation of rigorous conclusions about outcomes), meta-analyses and narrative reviews (purposive sampling of the literature use of theoretical or topical criteria to include papers on the basis of type, relevance and perceived significance, with the aim of summarising, discussing and critiquing conclusions).11 These reviews were carried out by including quantitative primary studies (eg, randomised controlled trials (RCTs), controlled before and after studies) and they are the appropriate study design to investigate the effectiveness of implementation strategies. Original research studies, meta-syntheses of qualitative research papers, secondary analysis of original data (eg, individual patient data meta-analysis), conference abstracts, editorials and commentary articles were excluded.

    Study selection

    Duplicate references were deleted. The titles and abstracts of all the records obtained from the search were independently double-screened. The primary review author (RL) screened all identified citations (titles and abstracts) for potential inclusion; co-authors acted as the second reviewers. RL obtained the full text of potentially eligible articles which were assessed for eligibility against the prespecified inclusion and exclusion criteria by two reviewers (RL and EM). Any discordance or uncertainty was resolved through discussion between the two reviewers initially and the involvement of a third reviewer as necessary.

    Data management and extraction

    For all eligible full-text articles, data were extracted by a single reviewer (RL) using standardised structured data abstraction forms. The content of the data abstraction forms were reviewed for validity by the co-authors, who have extensive experience in systematic review methodologies and implementation/evaluation of complex interventions, to ensure all key important information from the included reviews were captured. Information about the reviews, including title, aims and objectives, setting, review methodology, number of included primary studies, details of analysis, critical appraisal of included primary studies such as the use of any quality assessment tool, and outcome measures were extracted.

    Owing to the substantial literature relevant to this review, a systematic, transparent and rigorous method was developed and applied, to enable more effective and efficient data management and synthesis. In brief, this method involved the following steps: (1) sorting papers according to the EPOC taxonomy; (2) selection of a benchmark review paper for each category; (3) selection of important outcomes; (4) data extraction. Selection of a benchmark review was based on predetermined criteria, namely: rigour of reviewing methodology (quality associated with methods and analysis undertaken), comprehensiveness (scope and breadth of topic) and year of publication (most recent review usually included the highest number of relevant studies). These criteria were developed by all co-authors through consensus, and then applied by one author (RL) and checked by two other authors independently. For example, Forsetlund et al12 was chosen as the benchmark review paper for continuing medical education because (1) it included the largest number of primary studies covering a number of broad topics, that is, general management of various health conditions such as prescribing behaviour, preventive care, screening; (2) quality appraisal was conducted using appropriate checklists; (3) adjusted median risk difference (RD) and relative percentage change were calculated; and (4) the analysis included only primary studies that were of low/moderate risk of bias. We identified six subsequent reviews that were found to be relevant to continuing medical education, all of which conducted narrative synthesis and did not assess the quality of the included primary studies; one had a relatively limited scope of only focusing on older patients.

    As many benchmark reviews reported large numbers of outcomes of varying relevance, we decided to select at least one and no more than three outcomes based on their generalisability, validity and reliability. We operationalised generalisability as the degree to which a given outcome was likely to apply across different settings, validity as the extent to which the measure accurately reflected a desired outcome (eg, a change in prescribing behaviour was prioritised over a change in knowledge), and reliability as the degree to which the measure was likely to give similar results if repeated under similar circumstances. As many of these judgements were subjective, we aimed to achieve consensus among co-authors using the following process: RL extracted all the outcomes from each benchmark review and circulated them to all co-authors, who applied the above criteria to rank the available outcomes. Where there was disagreement between co-authors, further discussion was held until consensus was reached.

    Finally, data were fully extracted from each selected benchmark, including characteristics of the review (eg, aim/objectives, databases searched, topic/targeted behaviour, selection criteria, outcome measures) and selected outcomes. Data for both dichotomous and continuous outcome measures were extracted. For dichotomous outcomes, the adjusted RD was usually calculated and reported in the reviews. The RD is the difference in outcome between intervention and control group means postintervention minus the difference between groups before the intervention. For continuous outcomes, the percentage change relative to the control mean postintervention was usually calculated. This is the adjusted difference between the intervention and control group means divided by the postintervention control group mean×100%. Median RD or change relative to the control was preferred as the summary estimate is less likely to be driven by possible outlying results (such as large effects from small studies of poor methodological quality). The interquartile ranges (IQRs), as a measure of the spread of the data, were also extracted. The results of the remaining relevant reviews in each EPOC category were summarised and entered into the synthesis table. Some papers conducted subgroup analyses and metaregression on various predetermined features, most commonly level of complexity (low vs high), type of targeted behaviour, format, and presence or absence of tailoring. This information was extracted if provided, in order to explore potential features associated with implementation success.

    Data synthesis

    A narrative approach was employed to synthesise the results of the included reviews using a synthesis table that was structured in accordance with our research questions. The synthesis table allowed comparison of results between benchmark paper and non-benchmark papers for each strategy. An example of this can be found in online supplementary file 5. Results of each non-benchmark paper were summarised (along with effect size if provided) and compared with the results of the benchmark paper. The results were arranged by topic or targeted behaviour ((1) any targeted behaviour; (2) guideline implementation (eg, guideline on asthma, cardiovascular disease); (3) disease management/diagnosis (eg, diabetes, hypertension, dementia); (4) prevention and screening (eg, cervical cancer, breast cancer); (5) prescribing behaviour (eg, antibiotic prescribing for respiratory conditions). Information such as the number or type of included studies and whether quality appraisal of studies was performed, were extracted to help explain potential differences (if applicable) in results between the benchmark and non-benchmark paper. Furthermore, a table (table 2) was developed to record the active components of strategies which appear to be associated with success.

    In addition to reporting the size of effect, to aid interpretation, we categorised the results using the definitions proposed by Grimshaw et al13 for dichotomous outcomes (absolute difference):

    • ‘Small’ to describe effect sizes ≤5%;

    • ‘Modest’ to describe effect sizes >5% and ≤10%;

    • ‘Moderate’ to describe effect sizes >10% and ≤20%;

    • ‘Large’ to describe effect sizes >20%.

    A flow diagram summarising the steps used to undertake this review of review can be found in online supplementary file 4.

    Quality assessment

    A subset of data extraction and synthesis (all benchmark review papers plus two randomly selected subsequent papers for each category) were checked by the co-investigators, using a quality assurance form. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) checklist was used to critically appraise the quality of reporting of the included benchmark review papers. PRISMA is a 27-item checklist consisting of preferred reporting items for systematic reviews and meta-analyses and it is primarily focused on randomised trials and quantitative data.14

    We reported our findings in accordance with the PRISMA guidelines. The full version of the review protocol was published elsewhere.15 This systematic review was part of a National Institute of Health Research (NIHR) School for Primary Care Research (SPCR) funded project (SPCR FR4 project number: 122). The systematic review protocol was registered on the PROSPERO database (CRD42014009410).

    Results

    Identification of relevant reviews

    Searches of the five electronic databases to December 2013 yielded a total of 6164 potentially eligible papers. Following the screening of titles and abstracts and full-text papers, 91 papers were included in the final systematic review of reviews, of which 9 were selected as benchmark reviews. Figure 1 presents the PRISMA flow diagram of study selection.

    Figure 1
    Figure 1

    PRISMA flow diagram of study selection.

    Characteristics of included reviews

    Details of included reviews are presented in online supplementary table S1. The majority of the included reviews (n=64; 70%) reported data on strategies targeted at individual healthcare professionals (ie, professional-level strategies); with 20 reviews (22%) reporting data on audit and feedback,13 ,16–34 18 (20%) on printed educational materials,13 ,21–24 ,26 ,27 ,32 ,35–44 16 (18%) on educational outreach visits,13 ,21 ,22 ,26–28 ,32 ,35 ,37 ,40 ,45–50 26 (29%) on educational meetings,12 ,19 ,21 ,22 ,25 ,31 ,32 ,35 ,38 ,41 ,42 ,47 ,51–64 7 (8%) on local opinion leaders21 ,22 ,24 ,32 ,37 ,40 ,65 and 24 (26%) on physician-based reminders.13 ,19 ,21 ,22 ,25 ,28 ,29 ,37 ,40 ,49 ,58 ,60 ,66–77 Ten reviews (11%) reported data on organisational implementation strategies (including revising professional roles and facilitation).28 ,61 ,69 ,78–84 Eleven reviews (12%) reported data on strategies targeted at the context level; all focused on financial strategies (eg, performance-based payment, fixed fee per patient achieving a specified outcome, single threshold target payment, capitation)20 ,23 ,78 ,80 ,85–91 and we could not identify any reviews on the effectiveness of regulatory strategies. Limited evidence was found on the cost-effectiveness of implementation strategies (economic evaluations, eg, cost-effectiveness, costs benefit analyses were rare).

    The focus of included reviews varied: some focused on a specific strategy (eg, audit and feedback) across multiple topic areas and outcomes; others considered the effectiveness of any or multiple strategies to improve a particular targeted behaviour (eg, cancer screening, guideline adherence); and yet others considered the effectiveness of a specific strategy to improve a particular targeted behaviour (single strategy, single topic area). Seventeen reviews focused on guideline implementation, 13 on quality of care or disease management, 1 on technology implementation, 18 on preventative care, 2 on collaborative working and 4 on prescribing behaviour.

    Fifty reviews (71%) were based exclusively in primary care and the remaining in mixed healthcare settings. Twenty-four reviews (26%) were undertaken in the USA, 12 (13%) in Canada, 17 (19%) in the UK, 6 (7%) in Australia, 14 (15%) in Europe and 9 (10%) elsewhere. The original studies included in the reviews were conducted worldwide, although 21 (23%) reported that the original studies were predominantly conducted in the USA. The number of original studies included in the reviews ranged from 2 to 235.

    Methodological quality of included reviews

    Benchmark reviews

    All nine benchmark reviews12 ,34 ,44 ,50 ,65 ,76 ,82 ,91 ,92 applied a priori criteria for selecting eligible papers and critically appraised the quality of the included primary studies. Five included RCTs only,12 ,34 ,44 ,50 ,92 and four excluded studies that were graded as high risk of bias, or judged to be of poor quality.12 ,34 ,50 ,82 Some benchmark reviews used criteria to select the outcomes reported. Where the primary papers described a primary outcome, this was used; where there were multiple outcomes with no named primary outcome, the median value across multiple outcomes was calculated.12 ,34 ,50 ,65 All outcomes were expressed as compliance with desired practice (composite outcome) which may include outcomes such as adherence to guidelines, screening rates and appropriate referrals, or process improvements. Eight reviews conducted some form of quantitative analysis (eg, meta-analysis, calculations of median RD, metaregression)12 ,34 ,44 ,50 ,65 ,76 ,82 ,92 and one conducted narrative synthesis.91 Quality assessment of all benchmark papers can be found in online supplementary file 6.

    Other (non-benchmark) reviews

    Overall, 79 reviews (96%) reported the use of explicit inclusion/exclusion criteria. Sixteen reviews (20%) included only randomised trials, 59 (72%) included studies with both randomised and non-randomised designs (eg, quasi-experimental, controlled before-after studies, interrupted time series). Eighteen (22%) conducted some form of quantitative analysis (eg, meta-analysis, calculations of median RD, metaregression) and the rest conducted narrative synthesis. Forty-seven reviews (57%) critically appraised their included primary studies using some form of checklist/assessment or described quality issues in the results or discussion. Only one review synthesised data using a theoretical framework.16

    Effects of single strategies

    Strategies directed at individual professionals
    Single strategy alone versus no strategy or usual care

    The most frequently reported comparison was between the effectiveness of a single implementation strategy (eg, educational outreach or audit and feedback) and no strategy (table 1). The majority of these reviews reported dichotomous outcomes (or median improvement, often calculated as median RD) observed small to modest effects, ranging from 2% to 9%. Figure 2 illustrates the median effects and IQRs of single strategies targeted at professionals compared with no strategy or usual care, reported in the benchmark reviews. The lower IQR of educational outreach visits, audit and feedback, educational meetings and computerised reminders were all above zero (the line of no effect). Printed educational materials and local opinion leaders were the least effective single strategies. The IQRs of all strategies overlapped considerably, indicating that no single strategy appeared to be more effective than others.

    Figure 2
    Figure 2

    Graph illustrating median effects of single professional-level strategies alone versus no strategy or usual care. *Trials=inclusion of RCTs and quasi-experimental trial design; studies=inclusion of trials and non-trial design. CME, continuing medical education; RCT, randomised controlled trial.

    Not all benchmark reviews provided results for continuous outcomes. The use of educational outreach visits was associated with the largest median change relative to no strategy (23%, IQR=12–39%), followed by educational meetings and workshops (10%, IQR=8–32%) and audit and feedback (1.3%, IQR=1.3–11%). In general, findings from non-benchmark reviews agreed with those from the benchmark reviews (table 1).

    View this table:
    Table 1

    Summary of the effects of single strategies and multifaceted strategies on adherence to desired practice

    Single strategy versus alternative single strategy

    Only benchmark reviews of audit and feedback, local opinion leaders, printed educational materials and educational meetings reported direct head-to-head comparisons of these single strategies with alternative single strategy; this comparison was not commonly reported in primary studies. For example, only two trials with a moderate risk of bias compared educational meetings to other strategies, namely an educational outreach visit and a facilitated implementation of an office system to improve services. In both trials, educational meetings were associated with a decrease in compliance (adjusted RD of −1.4% and −8.0%), relative to the comparison strategies. Similarly, two trials compared opinion leaders alone to other strategies (standardised lectures and audit and feedback) and found a 14% absolute increase in adherence to desired practice for opinion leaders alone.65 No conclusions could be drawn from the limited evidence.

    Strategies directed at the organisation

    Revising professional roles

    We could not identify a benchmark review in this category. Six reviews examined the effects of revising professional roles, for example, having a nurse with a redefined role to offer support, such as undertaking preventive and follow-up tasks.28 ,61 ,69 ,78 ,83 ,84 In general, these reviews demonstrated an improvement in process of care outcomes.

    Practice facilitation

    Five reviews78–82 examined the effects of practice facilitation, defined as having experienced facilitators, who can be internal or external to an organisation, to work with individual practices in order to facilitate and support a range of processes and activities, such as education, interactive consensus building and goal setting, quality improvement and problem solving. The benchmark review (total n=23 studies; 20 RCTs and 3 controlled clinical trials) reported an overall effect size of 0.56 (95% CI 0.43 to 0.68; p<0.001) which favoured practice facilitation (relative to controls) with non-significant heterogeneity and some indications of publication bias. It also found primary care practices are 2.76 (95% CI 2.18 to 3.43) times more likely to adopt evidence-based guidelines through practice facilitation.82 Similar significant effects were observed in other reviews.79–81 Practice facilitation improved adoption of guidelines in various clinical areas that focused on prevention, system-level improvements and outcomes associated with chronic disease management within practice settings.79

    Changing organisational culture

    One review assessed strategies to change organisational culture to improve professional practice.93 However, the authors were unable to draw conclusions about effective strategies for changing culture as no relevant primary studies fulfilled the methodological criteria for inclusion. There was a lack of reviews that summarised the evidence on organisational-level implementation strategies and little is known about what they might comprise.

    Strategies directed at the wider context (eg, policy)

    Financial strategies

    Eleven reviews examined the effectiveness of financial strategies and the majority of these could not calculate an overall effect estimate due to heterogeneity, including the type of financial payment (eg, performance-based payment, capitation, fee-for-service), the size of payment, outcomes measured, targeted behaviour and the context/setting in which they were implemented. The benchmark review included seven studies and showed that financial strategies had positive but modest and variable effects on a small number of performance and quality of care outcomes.91 Other relevant reviews also reported mixed effectiveness. The majority of primary studies included in these reviews were conducted in the USA, and therefore may have limited applicability to other healthcare systems.

    Effects of multifaceted strategies

    Some reviews hypothesised that multifaceted implementation strategies could be more effective as more barriers could be addressed.60 However, the data suggested the effects of multifaceted strategies were variable and either no more effective or only slightly more effective in changing practice than single strategies.

    All benchmark reviews assessed the effectiveness of their chosen strategy (or strategy of interest, eg, audit and feedback) plus additional strategies (more than one, eg, audit and feedback plus educational outreach visits), compared with no strategy; and the findings of this comparison group were largely similar to the findings of single strategies alone versus no strategy. Evidence from the remaining reviews (in the same category) also presented mixed results.12 ,13 ,17 ,19 ,20 ,27 ,32 ,34 ,36 ,48 ,50 ,51 ,52 ,60 ,61 ,65 ,71 ,76 ,78 ,104 ,105–113 Single strategies could be as effective as multifaceted strategies in improving practice particularly when baseline adherence to desired practice was low.

    Features of implementation strategies associated with success

    Drawing on the literature included in this review of reviews, we identified features of implementation strategies that appeared to be associated with success. These are presented in table 2 and include features such as interactivity, tailoring and status of the individual delivering the strategy. Features that appeared to be relatively ineffective included didactic teaching format, low-intensity strategies and infrequent feedback.

    View this table:
    Table 2

    Features appeared to be associated with successful implementation

    Evidence on economic evaluations

    Overall there was a lack of economic evaluation data on the use of implementation strategies. Benchmark reviews mentioned that few primary studies reported costs or cost-effectiveness of the strategy.50

    Discussion

    The purpose of this systematic review of reviews was to evaluate the effectiveness of strategies to improve implementation of complex interventions in primary care. We found that there has been a rapid increase in the number of primary studies and reviews examining the effectiveness of implementation strategies. Most of the included reviews evaluated the effects of individual professional-level implementation strategies and they may achieve small to modest improvement (range 2–9%) compared with no strategy. Of these professional-level strategies, educational outreach visits, educational meetings, and audit and feedback had the best evidence base; included a relatively large number of RCTs with low risk of bias. Passive dissemination strategies such as the distribution of educational materials appeared largely ineffective and the effect of local opinion leaders appeared variable.

    There was a lack of evidence directly comparing the effectiveness of different strategies. These findings are largely consistent with those reported in a previous review of reviews on the effectiveness of professional-level strategies to promote the implementation of research findings.94 Although the median effects of most strategies were found to be small to modest, they might have much greater impact when applied at the population level, as 90% of care is delivered in primary care. Their effects may also be greater when applied in certain circumstances or settings. In addition, the follow-up period of the primary studies tended to be relatively short; therefore, long-term effects could not be determined.

    There was limited review evidence on the effectiveness of organisational-level implementation strategies in primary care. There are some ongoing studies especially around promoting leadership and organisational culture, for instance, Curry et al95 have developed a theoretically informed intervention (multifaceted strategy approach) aimed at promoting organisational culture by encouraging organisational leadership which accelerates learning and improvement and integrated evidence-based practices into routine work of the organisation in 10 hospitals. Similarly Aarons et al96 conducted a randomised mixed-methods pilot study of a leadership and organisation development strategy for evidence-based mental health practice implementation. Further work is needed in this area, including identifying, describing and characterising potential organisational-level strategies and evaluating their effectiveness in any healthcare context. We identified even fewer reviews on strategies that addressed characteristics of the wider context level in primary care and most of these focused on financial arrangements and structures. None of the included reviews addressed regulatory strategies such as changes in medical liability laws, licensure standards and governance, or other wider context-level strategies, such as creating new funding for the use of a particular complex intervention or changes in policy.

    Previous literature had suggested that multifaceted strategies could be more effective than single strategies,94 ,97 ,98 and their use was advocated in the 2008 Medical Research Council (MRC) complex intervention guidance as potentially useful approaches to implementation.99 However, we found that multifaceted implementation strategies were not necessarily more effective than single implementation strategies and that the effectiveness of multifaceted strategies did not increase incrementally with the number of components. Another recent systematic review of reviews examining whether multifaceted strategies are more effective than single strategies100 reported similar findings. There could be a number of possible reasons for this: (1) ceiling effect—both groups received co-strategies and any additional strategy would be unlikely to show further benefits; (2) relevance—strategies are often rarely justified theoretically,8 ,101 that is, some strategies included are not necessary or relevant to the context; (3) timing and delivery—all the strategy components included in the primary studies might have been delivered at the same time, and possibly by spacing the components of multifaceted strategies at different times, may be more effective; (4) active features that support effective implementation were not included; and (5) strategies (in terms of combinations, timing/frequency, duration) and settings were too heterogeneous across primary studies to make it appropriate to combine them. In addition, multifaceted implementation strategies are likely to cost more than single implementation strategies.

    Since the completion of this review, Powell et al compiled and published the Expert Recommendations for Implementing Change (ERIC) refined compilation of strategies for implementing change. This is a list of strategies for implementing clinical innovations in health and mental health based on sources such as published reviews and through expert consensus.102 We undertook a post hoc exercise and mapped the included reviews to the ERIC refined compilation of implementation strategies (see online supplementary file 7). We found that the evidence base for the majority of strategies included in this list was limited. This list is a valuable resource of discrete implementation strategies and more primary evaluation studies on the efficacy and effectiveness of these implementation strategies are required. Finally, we found very limited evidence on the cost-effectiveness of implementation strategies. Hoomans and Severens commented that despite the demand for undertaking economic evaluation in health services research, its use is not standard practice in assessing implementation strategies. They also found that studies on implementation strategies tend to assess only their effect on practice and health outcomes, and very few conducted economic evaluations.103

    Strengths and limitations

    There are several strengths to this review of reviews. To the best of our knowledge, this is the most comprehensive review of the available literature on the effectiveness of single and multifaceted implementation strategies and is not restricted to any topic or health condition. It is therefore highly generalisable. The review was conducted using rigorous reviewing methods, including a comprehensive search strategy, double screening of all titles, abstracts and full-text articles, the use of a robust approach to selecting benchmark reviews, with findings elaborated with reference to other reviews. In addition, we were able to identify a tentative list of components of specific strategies that appeared to be associated with effective implementation.

    There are also some limitations, including the possibility that not all relevant primary research studies were captured by included reviews, so some findings may be missed by concentrating on reviews. Moreover, by only focusing on reviews, there is an inevitable time lag, with recent studies less likely to be reported in reviews. Data extraction was conducted by a single reviewer. However, data extraction and synthesis of all benchmark papers plus two other randomly selected papers for each category were checked independently for accuracy by a second reviewer. There are a number of challenges to conducting this narrative synthesis: (1) the heterogeneous nature of the included primary studies and reviews (in terms of topic area, health conditions, type of analysis); (2) each review contained an enormous amount of information and we made a good attempt to focus on the results that best addressed our review question(s) by applying rigorous criteria and using a structured approach to synthesise the results.

    Implications for clinical practice

    Most implementation strategies targeted at changing practice at the professional level can achieve small to modest improvement. To facilitate successful implementation of complex interventions, the choice of strategies needs to be based on barriers relevant to the setting (context) in which the implementation occurs, in order to achieve maximum benefits. Furthermore, these barriers or implementation issues may change over time; they need to be reviewed periodically throughout the change process to ensure that the strategies used continue to be appropriate and relevant. In some circumstances, it may be more effective to use a single strategy and focus on one key problem of implementation instead of trying to tackle numerous problems using complex multifaceted strategies. When applying an implementation strategy, it is important to incorporate features shown to improve the likelihood of successful implementation.

    Implications for research

    This systematic review of reviews suggests that there is an increasing amount of primary and secondary research on the effectiveness of implementation strategies; however, they tended to focus on a small number of strategies with known evidence. Despite the large body of published literature, the evidence base on implementation strategies remains inconclusive. The evidence could not distinguish differences in effectiveness between various professional-level implementation strategies. Better designed (ie, development of strategies based on theoretical framework, tailored to relevant barriers) and described (ie, reporting of strategy components in accordance with reporting guidelines) studies are needed. Passive strategies alone are unlikely to be effective and in the authors’ opinion, no further studies of this kind are needed. Future research and systematic reviews should focus on why and how an implementation strategy (or combinations of strategies) works differently in different contexts and on more rigorous research testing a broad range of strategies that work at the organisational and wider contextual levels (What are they? How do they work? How effective and/or cost-effective are they?).

    Conclusion

    The effects of professional-level implementation strategies were small to modest. Limited evidence was found in relation to the effectiveness of organisational-level and wider contextual-level implementation strategies. Our findings suggest multifaceted strategies may not always be more effective than a single strategy. Development and evaluation of implementation strategies should be informed by theoretical frameworks. There is no ‘one size fits all’ implementation strategy; they are likely to work best if tailored to local circumstances and takes account of broader policy context.

    Acknowledgments

    The authors wish to acknowledge the contribution of Professor Paul Little and Dr Evangelos Kontopantelis for sharing their valuable ideas and insights.

    References

    1. ↵
      1. Woolf SH
      . The meaning of translational research and why it matters. JAMA 2008;299:211–13. doi:10.1001/jama.2007.26
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      HM Treasury. A Review of UK Health Research Funding. December 2006. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/228984/0118404881.pdf.
    3. ↵
      1. Craig P,
      2. Dieppe P,
      3. Macintyre S, et al
      . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655. doi:10.1136/bmj.a1655
      OpenUrlFREE Full Text
    4. ↵
      1. Jumaa PA
      . Hand hygiene: simple and complex. Int J Infect Dis 2005;9:3–14. doi:10.1016/j.ijid.2004.05.005
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Bambra C,
      2. Gibson M,
      3. Sowden A, et al
      . Tackling the wider social determinants of health and health inequalities: evidence from systematic reviews. J Epidemiol Community Health 2010;64:284–91. doi:10.1136/jech.2008.082743
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Harnden A,
      2. Lehman R
      . New primary care series: easily missed? BMJ 2009;338:b491. doi:10.1136/bmj.b491
      OpenUrlFREE Full Text
    7. ↵
      1. Smith J,
      2. Holder H,
      3. Edwards N, et al
      . Securing the future of general practice: new models of primary care. Nuffield Trust and King's Fund, 2013.
    8. ↵
      1. Proctor EK,
      2. Powell BJ,
      3. McMillen JC
      . Implementation strategies: recommendations for specifying and reporting. Implement Sci 2013;8:139. doi:10.1186/1748-5908-8-139
      OpenUrlCrossRefPubMed
    9. ↵
      EPOC 2002. Cochrane Effective Practice and Organisation of Care Group: Data Collection Checklist. EPOC resources for review authors Revised 2002. 2002. [http://www.epoc.cochrane.org/en/handsearchers.html].
    10. ↵
      Royal College of General Practitioners (2007). The future direction of general practice, a roadmap. London: RCGP, 2007. http://www.rcgp.org.uk/policy/rcgp-policy-areas/~/media/Files/Policy/A-Z-policy/the_future_direction_rcgp_roadmap.ashx.
    11. ↵
      1. Mair FS,
      2. May C,
      3. O'Donnell C, et al
      . Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bull World Health Organ 2012;90:357–64. doi:10.2471/BLT.11.099424
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Forsetlund L,
      2. Bjørndal A,
      3. Rashidian A, et al
      . Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2009;(2):CD003030. doi:10.1002/14651858.CD003030.pub2
    13. ↵
      1. Grimshaw JM,
      2. Thomas RE,
      3. MacLennan G, et al
      . Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8:iii–iv, 1–72. doi:10.3310/hta8060
      OpenUrlPubMed
    14. ↵
      1. Liberati A,
      2. Altman DG,
      3. Tetzlaff J, et al
      . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009;151:W65–94. doi:10.7326/0003-4819-151-4-200908180-00136
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Lau R,
      2. Stevenson F,
      3. Ong BN, et al
      . Addressing the evidence to practice gap for complex interventions in primary care: a systematic review of reviews protocol. BMJ Open 2014;4:e005548. doi:10.1136/bmjopen-2014-005548
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Gardner B,
      2. Whittington C,
      3. McAteer J, et al
      . Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70:1618–25. doi:10.1016/j.socscimed.2010.01.039
      OpenUrlCrossRefPubMed
    17. ↵
      1. Dulko D
      . Audit and feedback as a clinical practice guideline implementation strategy: a model for acute care nurse practitioners. Worldviews Evid Based Nurs 2007;4:200–9. doi:10.1111/j.1741-6787.2007.00098.x
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Mugford M,
      2. Banfield P,
      3. O'Hanlon M
      . Effects of feedback of information on clinical practice: a review. BMJ 1991;303:398–402. doi:10.1136/bmj.303.6799.398
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Unverzagt S,
      2. Oemler M,
      3. Braun K, et al
      . Strategies for guideline implementation in primary care focusing on patients with cardiovascular disease: a systematic review. Fam Pract 2014;31:247–66. doi:10.1093/fampra/cmt080
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Okelo SO,
      2. Butz AM,
      3. Sharma R, et al
      . Interventions to modify health care provider adherence to asthma guidelines: a systematic review. Pediatrics 2013;132:517–34. doi:10.1542/peds.2013-0779
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Schichtel M,
      2. Rose PW,
      3. Sellers C
      . Educational interventions for primary healthcare professionals to promote the early diagnosis of cancer: a systematic review. Educ Prim Care 2013; 24:274–90. doi:10.1080/14739879.2013.11494186
      OpenUrl
    22. ↵
      1. Medves J,
      2. Godfrey C,
      3. Turner C, et al
      . Systematic review of practice guideline dissemination and implementation strategies for healthcare teams and team-based practice. Int J Evid Based Healthc 2010;8:79–89. doi:10.1111/j.1744-1609.2010.00166.x
      OpenUrlPubMed
    23. ↵
      1. Akbari A,
      2. Mayhew A,
      3. Al-Alawi MA, et al
      . Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev 2008;(4):CD005471. doi:10.1002/14651858.CD005471.pub2
    24. ↵
      1. McKenna H,
      2. Ashton S,
      3. Keeney S
      . Barriers to evidence based practice in primary care: a review of the literature. Int J Nurs Stud 2004;41:369–78. doi:10.1016/j.ijnurstu.2003.10.008
      OpenUrlCrossRefPubMed
    25. ↵
      1. Weingarten SR,
      2. Henning JM,
      3. Badamgarav E, et al
      . Interventions used in disease management programmes for patients with chronic illness-which ones work? Meta-analysis of published reports. BMJ 2002;325:925. doi:10.1136/bmj.325.7370.925
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Gross PA,
      2. Pujat D
      . Implementing practice guidelines for appropriate antimicrobial usage: a systematic review. Med Care 2001;39(8 Suppl 2):II55–69.
      OpenUrlPubMedWeb of Science
    27. ↵
      1. Gill PS,
      2. Makela M,
      3. Vermeulen KM, et al
      . Changing doctor prescribing behaviour. Pharm World Sci 1999;21: 158–67. doi:10.1023/A:1008719129305
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Yano EM,
      2. Fink A,
      3. Hirsch SH, et al
      . Helping practices reach primary care goals. Lessons from the literature. Arch Intern Med 1995;155:1146–56. doi:10.1001/archinte.1995.00430110051006
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Bywood PT,
      2. Lunnay B,
      3. Roche AM
      . Strategies for facilitating change in alcohol and other drugs (AOD) professional practice: a systematic review of the effectiveness of reminders and feedback. Drug Alcohol Rev 2008;27:548–58. doi:10.1080/09595230802245535
      OpenUrlPubMed
    30. ↵
      1. Xyrichis A,
      2. Lowton K
      . What fosters or prevents interprofessional teamworking in primary and community care? A literature review. Int J Nurs Stud 2008;45:140–53. doi:10.1016/j.ijnurstu.2007.01.015
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Zwar N,
      2. Harris M,
      3. Griffiths R, et al
      . A systematic review of chronic disease management. Research Centre for Primary Health Care and Equity, School of Public Health and Community Medicine, UNSW, 2006.
    32. ↵
      1. Davis DA,
      2. Taylor-Vaisey A
      . Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ 1997;157:408–16.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Balas EA,
      2. Boren SA,
      3. Brown GD, et al
      . Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials. J Gen Intern Med 1996;11:584–90. doi:10.1007/BF02599025
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Ivers N,
      2. Jamtvedt G,
      3. Flottorp S, et al
      . Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. doi:10.1002/14651858.CD000259.pub3
      OpenUrlPubMed
    35. ↵
      1. Freudenstein U,
      2. Howe A
      . Recommendations for future studies: a systematic review of educational interventions in primary care settings. Br J Gen Pract 1999;49:995–1001.
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Smolders M,
      2. Laurant M,
      3. Roberge P, et al
      . Knowledge transfer and improvement of primary and ambulatory care for patients with anxiety. Can J Psychiatry 2008;53:277–93.
      OpenUrlPubMed
    37. ↵
      1. Conroy M,
      2. Shannon W
      . Clinical guidelines: their implementation in general practice. Br J Gen Pract 1995;45:371–5.
      OpenUrlAbstract/FREE Full Text
    38. ↵
      1. Oxman AD,
      2. Thomson MA,
      3. Davis DA, et al
      . No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995;153:1423–31.
      OpenUrlAbstract
    39. ↵
      1. Grimshaw JM,
      2. Russell IT
      . Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317–22. doi:10.1016/0140-6736(93)92244-N
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Soumerai SB,
      2. McLaughlin TJ,
      3. Avorn J
      . Improving drug prescribing in primary care: a critical analysis of the experimental literature. Milbank Q 1989;67:268–317. doi:10.2307/3350142
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Pippalla RS,
      2. Riley DA,
      3. Chinburapa V
      . Influencing the prescribing behaviour of physicians: a metaevaluation. J Clin Pharm Ther 1995;20:189–98. doi:10.1111/j.1365-2710.1995.tb00648.x
      OpenUrlCrossRefPubMedWeb of Science
    42. ↵
      1. Nilsen P,
      2. Aalto M,
      3. Bendtsen P, et al
      . Effectiveness of strategies to implement brief alcohol intervention in primary healthcare. A systematic review. Scand J Prim Health Care 2006; 24:5–15. doi:10.1080/02813430500475282
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Lomas J
      . Words without action? The production, dissemination, and impact of consensus recommendations. Annu Rev Public Health 1991;12:41–65. doi:10.1146/annurev.pu.12.050191.000353
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Giguère A,
      2. Légaré F,
      3. Grimshaw J, et al
      . Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;10:CD004398.
      OpenUrlPubMed
    45. ↵
      1. Qureshi N,
      2. Allen J,
      3. Hapgood R
      . A systematic review of educational outreach visits for non-prescribing interventions in general practice. Eur J Gen Pract 2002;8:31–6. doi:10.3109/13814780209160826
      OpenUrl
    46. ↵
      1. Koch T,
      2. Iliffe S
      . Dementia diagnosis and management: a narrative review of changing practice. Br J Gen Pract 2011;61:e513–25. doi:10.3399/bjgp11X588493
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Lineker SC,
      2. Husted JA
      . Educational interventions for implementation of arthritis clinical practice guidelines in primary care: effects on health professional behavior. J Rheumatol 2010;37:1562–9. doi:10.3899/jrheum.100045
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Anderson P,
      2. Jane-Llopis E
      . How can we increase the involvement of primary health care in the treatment of tobacco dependence? A meta-analysis. Addiction 2004;99:299–312. doi:10.1111/j.1360-0443.2003.00672.x
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Beilby JJ,
      2. Silagy CA
      . Trials of providing costing information to general practitioners: a systematic review. Med J Aust 1997;167:89–92.
      OpenUrlPubMedWeb of Science
    50. ↵
      1. O'Brien MA,
      2. Rogers S,
      3. Jamtvedt G, et al
      . Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2007;(4):CD000409.
    51. ↵
      1. Thomas DC,
      2. Johnston B,
      3. Dunn K, et al
      . Continuing medical education, continuing professional development, and knowledge translation: improving care of older patients by practicing physicians. J Am Geriatr Soc 2006;54:1610–18. doi:10.1111/j.1532-5415.2006.00879.x
      OpenUrlCrossRefPubMed
    52. ↵
      1. Cauffman JG,
      2. Forsyth RA,
      3. Clark VA, et al
      . Randomized controlled trials of continuing medical education: what makes them most effective? J Contin Educ Health Prof 2002;22:214–21. doi:10.1002/chp.1340220405
      OpenUrlCrossRefPubMed
    53. ↵
      1. Davies A,
      2. O'Brien MA,
      3. Freemantle N, et al
      . Impact of formal continuing medical education do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 1999;282:867–74. doi:10.1001/jama.282.9.867
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Waddell DL
      . The effects of continuing education on nursing practice: a meta-analysis. J Contin Educ Nurs 1991;22:113–18.
      OpenUrlPubMed
    55. ↵
      1. Beaudry JS
      . The effectiveness of continuing medical education: a quantitative synthesis. J Contin Educ Health Prof 1989;9:285–307. doi:10.1002/chp.4750090414
      OpenUrl
    56. ↵
      1. Davis DA,
      2. Thomson MA,
      3. Oxman AD, et al
      . Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA 1992;268:1111–17. doi:10.1001/jama.1992.03490090053014
      OpenUrlCrossRefPubMedWeb of Science
    57. ↵
      1. Gual A,
      2. Sabadini MB
      . Implementing alcohol disorders treatment throughout the community. Curr Opin Psychiatry 2011;24:203–7. doi:10.1097/YCO.0b013e3283459256
      OpenUrlCrossRefPubMedWeb of Science
    58. ↵
      1. Glynn LG,
      2. Murphy AW,
      3. Smith SM, et al
      . Self-monitoring and other non-pharmacological interventions to improve the management of hypertension in primary care: a systematic review. Br J Gen Pract 2010;60:e476–88. doi:10.3399/bjgp10X544113
      OpenUrlAbstract/FREE Full Text
    59. ↵
      1. Perry M,
      2. Draskovic I,
      3. Lucassen P, et al
      . Effects of educational interventions on primary dementia care: a systematic review. Int J Geriatr Psychiatry 2011;26:1–11. doi:10.1002/gps.2479
      OpenUrlPubMed
    60. ↵
      1. Hulscher M,
      2. Wensing M,
      3. van der Weijden T, et al
      . Interventions to implement prevention in primary care. Cochrane Database Syst Rev 2001;(1):CD000362. doi:10.1002/14651858.CD000362
    61. ↵
      1. Gilbody S,
      2. Whitty P,
      3. Grimshaw J, et al
      . Educational and organizational interventions to improve the management of depression in primary care: a systematic review. JAMA 2003; 289:3145–51. doi:10.1001/jama.289.23.3145
      OpenUrlCrossRefPubMedWeb of Science
    62. ↵
      1. Flodgren G,
      2. Deane K,
      3. Dickinson HO, et al
      . Interventions to change the behaviour of health professionals and the organisation of care to promote weight reduction in overweight and obese adults. Cochrane Database Syst Rev 2010;(3):CD000984. doi:10.1002/14651858.CD000984.pub2
    63. ↵
      1. Gould DJ,
      2. Moralejo D,
      3. Drey N, et al
      . Interventions to improve hand hygiene compliance in patient care. Cochrane Database Syst Rev 2010;(9):CD005186. doi:10.1002/14651858.CD005186.pub3
    64. ↵
      1. Goodwin V,
      2. Jones-Hughes T,
      3. Thompson-Coon J, et al
      . Implementing the evidence for preventing falls among community-dwelling older people: a systematic review. J Safety Res 2011;42:443–51. doi:10.1016/j.jsr.2011.07.008
      OpenUrlCrossRefPubMed
    65. ↵
      1. Flodgren G,
      2. Parmelli E,
      3. Doumit G, et al
      . Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2011;(8):CD000125. doi:10.1002/14651858.CD000125.pub4
    66. ↵
      1. Dexheimer JW,
      2. Talbot TR,
      3. Sanders DL, et al
      . Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2008;15:311–20. doi:10.1197/jamia.M2555
      OpenUrlAbstract/FREE Full Text
    67. ↵
      1. Balas EA,
      2. Austin SM,
      3. Mitchell JA, et al
      . The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996;5:271–8. doi:10.1001/archfami.5.5.271
      OpenUrlCrossRefPubMedWeb of Science
    68. ↵
      1. Austin SM,
      2. Balas EA,
      3. Mitchell JA, et al
      . Effect of physician reminders on preventive care: meta-analysis of randomized clinical trials. Proc Annu Symp Comput Appl Med Care 1994:121–4.
    69. ↵
      1. Van CJ,
      2. Kuhlthau KA,
      3. Bloom S, et al
      . Interventions to improve screening and follow-up in primary care: a systematic review of the evidence. Acad Pediatr 2012;12:269–82. doi:10.1016/j.acap.2012.02.004
      OpenUrlCrossRefPubMed
    70. ↵
      1. Nam S,
      2. Chesla C,
      3. Stotts NA, et al
      . Barriers to diabetes management: patient and provider factors. Diabetes Res Clin Pract 2011;93:1–9. doi:10.1016/j.diabres.2011.02.002
      OpenUrlCrossRefPubMed
    71. ↵
      1. Bauer MS
      . A review of quantitative studies of adherence to mental health clinical practice guidelines. Harv Rev Psychiatry 2002;10:138–53. doi:10.1080/10673220216217
      OpenUrlCrossRefPubMedWeb of Science
    72. ↵
      1. Kupets R,
      2. Covens A
      . Strategies for the implementation of cervical and breast cancer screening of women by primary care physicians. Gynecol Oncol 2001;83:186–97. doi:10.1006/gyno.2001.6387
      OpenUrlCrossRefPubMedWeb of Science
    73. ↵
      1. Mandelblatt J,
      2. Kanetsky PA
      . Effectiveness of interventions to enhance physician screening for breast cancer. J Fam Pract 1995;40:162–71.
      OpenUrlPubMedWeb of Science
    74. ↵
      1. Harvey EL,
      2. Glenny AM,
      3. Kirk SF, et al
      . An updated systematic review of interventions to improve health professionals’ management of obesity. Obes Rev 2002;3:45–55. doi:10.1046/j.1467-789X.2002.00053.x
      OpenUrlPubMed
    75. ↵
      1. Vernon SW,
      2. McQueen A,
      3. Tiro JA, et al
      . Interventions to promote repeat breast cancer screening with mammography: a systematic review and meta-analysis. J Natl Cancer Inst 2010;102:1023–39. doi:10.1093/jnci/djq223
      OpenUrlAbstract/FREE Full Text
    76. ↵
      1. Shojania KG,
      2. Jennings A,
      3. Mayhew A, et al
      . The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev 2009;(3):CD001096. doi:10.1002/14651858.CD001096.pub2
    77. ↵
      1. Holden DJ,
      2. Jonas DE,
      3. Porterfield DS, et al
      . Systematic review: enhancing the use and quality of colorectal cancer screening. Ann Intern Med 2010;152:668–76. doi:10.7326/0003-4819-152-10-201005180-00239
      OpenUrlCrossRefPubMedWeb of Science
    78. ↵
      1. Franx G,
      2. Dixon L,
      3. Wensing M, et al
      . Implementation strategies for collaborative primary care-mental health models. Curr Opin Psychiatry 2013;26:502–10. doi:10.1097/YCO.0b013e328363a69f
      OpenUrl
    79. ↵
      1. Nagykaldi Z,
      2. Mold JW,
      3. Aspy CB
      . Practice facilitators: a review of the literature. Fam Med 2005;37:581–8.
      OpenUrlPubMedWeb of Science
    80. ↵
      1. Thomas RE,
      2. Russell M,
      3. Lorenzetti D
      . Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database Syst Rev 2010;(9):CD005188. doi:10.1002/14651858.CD005188.pub2
    81. ↵
      1. Langberg JM,
      2. Brinkman WB,
      3. Lichtenstein PK, et al
      . Interventions to promote the evidence-based care of children with ADHD in primary-care settings. Expert Rev Neurother 2009;9:477–87. doi:10.1586/ern.09.5
      OpenUrlCrossRefPubMed
    82. ↵
      1. Baskerville NB,
      2. Liddy C,
      3. Hogg W
      . Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med 2012;10:63–74. doi:10.1370/afm.1312
      OpenUrlAbstract/FREE Full Text
    83. ↵
      1. Arroyave AM,
      2. Penaranda EK,
      3. Lewis CL
      . Organizational change: a way to increase colon, breast and cervical cancer screening in primary care practices. J Community Health 2011;36:281–8. doi:10.1007/s10900-010-9309-7
      OpenUrlCrossRefPubMedWeb of Science
    84. ↵
      1. Horrocks S,
      2. Anderson E,
      3. Salisbury C
      . Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors. BMJ 2002;324:819–23. doi:10.1136/bmj.324.7341.819
      OpenUrlAbstract/FREE Full Text
    85. ↵
      1. Van HP,
      2. De SD,
      3. Annemans L, et al
      . Systematic review: effects, design choices, and context of pay-for-performance in health care. BMC Health Serv Res 2010;10:247. doi:10.1186/1472-6963-10-247
      OpenUrlCrossRefPubMed
    86. ↵
      1. Petersen LA,
      2. Woodard LD,
      3. Urech T, et al
      . Does pay-for-performance improve the quality of health care? Ann Intern Med 2006;145:265–72. doi:10.7326/0003-4819-145-4-200608150-00006
      OpenUrlCrossRefPubMedWeb of Science
    87. ↵
      1. Town R,
      2. Kane R,
      3. Johnson P, et al
      . Economic incentives and physicians’ delivery of preventive care: a systematic review. Am J Prev Med 2005;28:234–40. doi:10.1016/j.amepre.2004.10.013
      OpenUrlCrossRefPubMedWeb of Science
    88. ↵
      1. Gosden T,
      2. Forland F,
      3. Kristiansen IS, et al
      . Impact of payment method on behaviour of primary care physicians: a systematic review. J Health Serv Res Policy 2001;6:44–55. doi:10.1258/1355819011927198
      OpenUrlAbstract/FREE Full Text
    89. ↵
      1. Chaix-Couturier C,
      2. Durand-Zaleski I,
      3. Jolly D, et al
      . Effects of financial incentives on medical practice: results from a systematic review of the literature and methodological issues. Int J Qual Health Care 2000;12:133–42. doi:10.1093/intqhc/12.2.133
      OpenUrlAbstract/FREE Full Text
    90. ↵
      1. Giuffrida A,
      2. Gosden T,
      3. Forland F, et al
      . Target payments in primary care: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2000;(3):CD000531. doi:10.1002/14651858.CD000531
    91. ↵
      1. Scott A,
      2. Sivey P,
      3. Ait OD, et al
      . The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev 2011;(9):CD008451. doi:10.1002/14651858.CD008451.pub2
    92. ↵
      1. Baker R,
      2. Camosso SJ,
      3. Gillies C, et al
      . Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2010;(3):CD005470. doi:10.1002/14651858.CD005470.pub2
    93. ↵
      1. Parmelli E,
      2. Flodgren G,
      3. Schaafsma ME, et al
      . The effectiveness of strategies to change organisational culture to improve healthcare performance. Cochrane Database Syst Rev 2011;(1):CD008315. doi:10.1002/14651858.CD008315.pub2
    94. ↵
      1. Bero LA,
      2. Grilli R,
      3. Grimshaw JM, et al
      . Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ 1998;317:465–8. doi:10.1136/bmj.317.7156.465
      OpenUrlFREE Full Text
    95. ↵
      1. Curry LA,
      2. Linnander EL, et al
      . Organizational culture change in U.S. hospitals: a mixed methods longitudinal intervention study. Implement Sci 2015;10:29. doi:10.1186/s13012-015-0218-0
      OpenUrl
    96. ↵
      1. Aarons GA,
      2. Ehrhart MG,
      3. Farahnak LR, et al
      . Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci 2015;10:11. doi:10.1186/s13012-014-0192-y
      OpenUrl
    97. ↵
      1. Davis DA,
      2. Thomson MA,
      3. Oxman AD, et al
      . Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995; 274:700–5. doi:10.1001/jama.1995.03530090032018
      OpenUrlCrossRefPubMedWeb of Science
    98. ↵
      1. Wensing M,
      2. van der Weijden T,
      3. Grol R
      . Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract 1998;48:991–7.
      OpenUrlAbstract/FREE Full Text
    99. ↵
      Medical Research Council. A framework for the development and evaluation of RCTs for complex interventions to improve health. London: MRC, 2000.
    100. ↵
      1. Squires JE,
      2. Sullivan K,
      3. Eccles MP, et al
      . Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci 2014;9:152. doi:10.1186/s13012-014-0152-6
      OpenUrlCrossRefPubMed
    101. ↵
      1. Michie S,
      2. Fixsen D,
      3. Grimshaw JM, et al
      . Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci 2009;4:40. doi:10.1186/1748-5908-4-40
      OpenUrlCrossRefPubMed
    102. ↵
      1. Powell BJ,
      2. Waltz TJ,
      3. Chinman MJ, et al
      . A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015;10:21. doi:10.1186/s13012-015-0209-1
      OpenUrlPubMed
    103. ↵
      1. Hoomans T,
      2. Severens JL
      . Economic evaluation of implementation strategies in health care. Implement Sci 2014;9:168. doi:10.1186/s13012-014-0168-y
      OpenUrl
    104. ↵
      1. Brusamento S,
      2. Legido-Quigley H,
      3. Panteli D, et al
      . Assessing the effectiveness of strategies to implement clinical guidelines for the management of chronic diseases at primary care level in EU Member States: a systematic review. Health Policy 2012;107:168–83. doi:10.1016/j.healthpol.2012.08.005
      OpenUrlCrossRefPubMed
    105. ↵
      1. Tooher R,
      2. Middleton P,
      3. Babidge W
      . Implementation of pressure ulcer guidelines: what constitutes a successful strategy? J Wound Care 2003;12:373–2. doi:10.12968/jowc.2003.12.10.26551
      OpenUrlPubMed
    106. ↵
      1. Comino EJ,
      2. Davies GP,
      3. Krastev Y, et al
      . A systematic review of interventions to enhance access to best practice primary health care for chronic disease management, prevention and episodic care. BMC Health Serv Res 2012;12:415. doi:10.1186/1472-6963-12-415
      OpenUrlPubMed
    107. ↵
      1. Boonacker CW,
      2. Hoes AW,
      3. Dikhoff MJ, et al
      . Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol 2010;74:1113–21. doi:10.1016/j.ijporl.2010.07.008
      OpenUrlCrossRefPubMed
    108. ↵
      1. Renders CM,
      2. Valk GD,
      3. Griffin SJ, et al
      . Interventions to improve the management of diabetes in primary care, outpatient, and community settings: a systematic review. Diabetes Care 2001;24:1821–33. doi:10.2337/diacare.24.10.1821
      OpenUrlAbstract/FREE Full Text
    109. ↵
      1. Gagnon MP,
      2. Légaré F,
      3. Labrecque M, et al
      . Interventions for promoting information and communication technologies adoption in healthcare professionals. Cochrane Database Syst Rev 2009;(1):CD006093. doi:10.1002/14651858.CD006093.pub2
    110. ↵
      1. Williams EC,
      2. Johnson ML,
      3. Lapham GT, et al
      . Strategies to implement alcohol screening and brief intervention in primary care settings: a structured literature review. Psychol Addict Behav 2011;25:206–14. doi:10.1037/a0022102
      OpenUrlCrossRefPubMed
    111. ↵
      1. Howe A,
      2. Ashton K,
      3. Hooper L
      . Effectiveness of educational interventions in primary care mental health: a qualitative systematic review. Prim Care Community Psychiatry 2006;11:167–77. doi:10.1185/135525707X183049
      OpenUrl
    112. ↵
      1. Snell JL,
      2. Buck EL
      . Increasing cancer screening: a meta-analysis. Prev Med 1996;25:702–7. doi:10.1006/pmed.1996.0109
      OpenUrlCrossRefPubMedWeb of Science
    113. ↵
      1. Rawl SM,
      2. Menon U,
      3. Burness A, et al
      . Interventions to promote colorectal cancer screening: an integrative review. Nurs Outlook 2012;60:172–81. doi:10.1016/j.outlook.2011.11.003
      OpenUrlPubMed
    View Abstract

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors EM is the principal investigator of the project. EM and BNO designed the study and obtained funding. RL developed and refined the study protocol with contributions from all co-authors (FS, BNO, KD, SE, HE, AK, NQ, AR, ST, RP, EM). RL prepared the manuscript. RL undertook data collection (literature search, data extraction), analysis, interpretation, report writing and manuscript preparation. All co-investigators contributed to the design, analysis, interpretation and report writing. All authors read and approved the final manuscript.

    • Funding The Evidence to Practice Project (SPCR FR4 project number: 122) is funded by the National Institute of Health Research (NIHR) School for Primary Care Research (SPCR). KD is part-funded by the NIHR Collaborations for Leadership in Applied Health Research and Care West Midlands and by a Knowledge Mobilisation Research Fellowship from the National Institute for Health Research. This paper presents independent research funded by the NIHR.

    • Disclaimer The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The extraction and synthesis table of the included studies can be made available on request to the corresponding author at r.lau@ucl.ac.uk.