You are here

Article Text

Article menu
Original research
Reducing low-value care: what can we learn from eight de-implementation studies in the Netherlands?
  1. Eva W Verkerk1,
  2. Simone A van Dulmen1,
  3. Gert P Westert1,
  4. Lotty Hooft2,
  5. Pauline Heus2,
  6. Rudolf B Kool1
  7. To do or not to do programme collaborators
    1. 1Department of IQ healthcare, Radboud university medical center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
    2. 2Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    1. Correspondence to Dr Eva W Verkerk; eva.verkerk{at}radboudumc.nl

    Abstract

    Background Reducing the overuse of care that is proven to be of low value increases the quality and safety of care. We aimed to identify lessons for reducing low-value care by looking at: (1) The effects of eight de-implementation projects. (2) The barriers and facilitators that emerged. (3) The experiences with the different components of the projects.

    Methods We performed a process evaluation of eight multicentre projects aimed at reducing low-value care. We reported the quantitative outcomes of the eight projects on the volume of low-value care and performed a qualitative analysis of the project teams’ experiences and evaluations. A total of 40 hospitals and 198 general practitioners participated.

    Results Five out of eight projects resulted in a reduction of low-value care, ranging from 11.4% to 61.3%. The remaining three projects showed no effect. Six projects monitored balancing measures and observed no negative consequences of their strategy. The most important barriers were a lack of time, an inability to reassure the patient, a desire to meet the patient’s wishes, financial considerations and a discomfort with uncertainty. The most important facilitators were support among clinicians, knowledge of the harms of low-value care and a growing consciousness that more is not always better. Repeated education and feedback for clinicians, patient information material and organisational changes were valued components of the strategy.

    Conclusions Successfully reducing low-value care is possible in spite of the powerful barriers that oppose it. The projects managed to recruit many hospitals and general practices, with five of them achieving significant results without measuring negative consequences. Based on our findings, we offer practical recommendations for successfully reducing low-value care.

    • qualitative research
    • quality improvement
    • quality improvement methodologies
    • implementation science

    Data availability statement

    Data are available upon reasonable request. De-identified reports of the eight projects are available (in Dutch) from the corresponding author upon reasonable request.

    http://creativecommons.org/licenses/by-nc/4.0/

    This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

    https://doi.org/10.1136/bmjoq-2021-001710

    Statistics from Altmetric.com

      Request Permissions

      If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

      WHAT IS ALREADY KNOWN ON THIS TOPIC

      • Reducing low-value care can increase the quality and safety of care while maintaining or decreasing healthcare costs.

      • Reducing such care has proven to be difficult and knowledge about its de-implementation is scarce.

      WHAT THIS STUDY ADDS

      • This study shows that clinicians can reduce low-value care successfully, despite barriers such as a lack of time, financial considerations and the need to reassure patients.

      • We provide practical recommendations for de-implementation studies, such as only reduce low-value care that is supported by sufficient evidence, tailor a de-implementation strategy to counter the barriers, use repeated education and feedback for clinicians, and provide carefully developed patient information.

      HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

      • More clinicians and researchers should reduce the overuse of low-value care, and our study will help them to improve their approach.

      Introduction

      Reducing care that is proven to be of low value is a universal and persistent challenge.1 Such low-value care, also called medical overuse, provides no, or very little, benefit to the patient if one takes into consideration its potential harm, costs, alternatives or patient preferences. In addition it also wastes resources.2 The term de-implementation is increasingly being used to describe a move away from ineffective or harmful medical practices.3 Such reduction of low-value care can increase the quality and safety of care. Hence, many initiatives have started worldwide, such as the Choosing Wisely campaign that began in the USA in 2012 and since then has spread to over 20 countries.4 However, reducing low-value care has proven to be difficult and knowledge about de-implementation is scarce.5 Two evaluations of Choosing Wisely recommendations showed marginal and varying results 1.5 years and 2.5 years after their release.6 7 Literature reviews suggest that strategies comprising different components, addressing patients and clinicians, have the potential to reduce overuse.2 8 However, the underlying mechanism in play is unclear and further experimentation and evaluation is needed.2 9

      Several publications describe lessons learnt so far from de-implementation. A review stated that involving physicians from the beginning is of great importance.10 Another study evaluated eight de-implementation projects in a hospital and found that support from the hospital board was a key to their success.11 An interview study among Choosing Wisely team members found that harm reduction is a significant motivator to reduce low-value care and that data collection could be challenging.12 Further in-depth knowledge and experience of de-implementation, including its impact and the barriers and facilitating factors involved, is needed to determine what is necessary for successful de-implementation.13

      In 2015, we started a nationwide programme in the Netherlands, comprising eight multicentre de-implementation projects that we prospectively monitored and evaluated. Each de-implementation project aimed to reduce a different type of low-value care. The projects were led by clinicians and set in multiple hospitals or primary care practices. This paper describes the lessons learnt from these projects and aims to contribute to the knowledge on de-implementation in clinical practice by answering three questions:

      • What effects can be achieved by a multicentre de-implementation project?

      • What barriers and facilitating factors might be encountered in de-implementation?

      • What are the effective components of a de-implementation project, and why?

      Methods

      We prospectively monitored and evaluated eight multicentre de-implementation projects in the Netherlands from June 2016 to October 2018. This study was part of a national programme called ‘To do or not to do? Reducing low-value care’, described in box 1 and in more detail in online supplemental file 1.

      Supplemental material

      Box 1

      Programme characteristics

      • ‘To do or not to do? Reducing low-value care’ was a national programme, coordinated by the eight university hospitals in the Netherlands.

      • The programme was both top-down and bottom-up, supported by stakeholders and initiated and led by clinicians.

      • Eight de-implementation projects were selected from 42 proposals by an independent committee.

      • The eight de-implementation projects received support from a central team, comprising the authors of this paper. The projects all followed similar steps according to the Grol and Wensing Implementation of Change Model.25

      • The projects ran from 2016 to 2018.

      The projects’ structure

      An overview of the projects can be found in table 1. Six projects aimed at reducing low-value hospital care and two projects focused on low-value primary care. Each project leader chose a design and approach that would fit their project best, resulting in a diversity of study designs and strategies. All projects evaluated the effect of the de-implementation strategy on the delivery of care. Six projects also measured the unintended effects of the strategy on patient outcomes and/or the use of other care. All project teams performed a structured process evaluation, and all projects obtained ethical approval before the start of their study. Several projects are described in more detail in other papers.14–23

      View this table:
      Table 1

      Overview of the eight projects

      Evaluation

      We used the Medical Research Council framework for process evaluation of complex interventions.24 This framework helps to analyse why, and how, the planned intervention has led to the effect observed. Using this framework, we evaluated three components of the projects: (1) The effects of the projects on clinical practice; (2) The contextual barriers and facilitating factors that emerged; and (3) The experiences of the project leaders and the participating clinicians and patients with the different components of the projects.

      For the first component, we report the quantitative outcomes of the eight projects on the volume of low-value care and on other outcomes that were measured. For components 2 and 3, EWV performed a qualitative analysis using Atlas.ti V.8.4.20 of the project teams’ experiences and evaluations. We collected data on this using logbooks, reports and interviews. The project teams kept a logbook and delivered a report on their results and evaluation, for which they used a variety of qualitative and quantitative methods (table 1). In May 2018, two researchers from the coordinating team (EWV and PH) interviewed the project leaders of the eight teams. Details on these interviews are reported in the COnsolidated criteria for REporting Qualitative research checklist (online supplemental file 2). The interviews included open-ended questions about the barriers and facilitating factors, the project leaders’ experiences with different components of their project, the lessons they have learnt and their advice for other project leaders. Reports of the audiotaped interviews were sent to the project leaders for correction and confirmation.

      Supplemental material

      EWV analysed the information reported in the logbooks, reports and interviews. Barriers and facilitators were classified using the framework of the determinants of change.25 This framework identifies individual health professional factors, patient factors, professional interactions, incentives and resources, and social, political and legal factors. These categories were used for coding. We added one category (low-value care related) and three subcategories (interaction with patient, interaction with clinician, patient environment) to this framework. This was because some factors that we identified did not fit in the existing categories. The coding and description of results were verified by PH and discussed until consensus was reached.

      Patient and public involvement

      We analysed eight de-implementation projects, each of which involved patients in their problem analysis, process evaluation or both (specified in table 1). Patients who had been involved in the problem analysis contributed to the development of the de-implementation strategy. In addition, a representative of the Dutch patient federation became a member of the programmes advisory board. This board regularly met and advised the coordinating team on the design and progress of the programme.

      Results

      First, we report the quantitative outcomes of the eight projects on the volume of care. Then, we report the results of our qualitative evaluation of the barriers and facilitating factors for de-implementation, and the experiences of the project leaders and the participating clinicians and patients with the different components of the projects.

      Effects on clinical practice

      The quantitative effects of the projects are shown in online supplemental file 3 and summarised in the text below. Five projects (4, 5, 6, 7, 8) showed a positive effect of the de-implementation strategy, the reduction in low-value care ranging from 11.4% to 61.3%. Project 5 compared two interventions and found a larger reduction in the group that received the additional patient information (10% extra reduction for vitamin D and a non-significant extra reduction of 4% for vitamin B12). Project 6 and 8 also collected data from a concurrent control group and both found a larger reduction (reduction of 11.4% in project 6 and 61.3% in project 8) in the intervention group compared with the control group (increase of 2.4% in project 6 and reduction of 17.5% in project 8). The remaining projects 4 and 7 studied one intervention arm and no control group.

      Supplemental material

      Three projects (1, 2, 3) found no effect of the de-implementation strategy. Project 1 found a significant reduction in the control group, compared with no difference in the intervention group. Project 2 found no change in both groups. Lastly, project 3 found a reduction in low-value care in both the intervention and the control groups, but no difference between these groups. Six projects monitored balancing measures and found no negative effects of the de-implementation on use of other care and patient outcomes.

      Barriers and facilitating factors for de-implementation

      The project teams found multiple factors that either hindered or facilitated the de-implementation of their low-value care practices. All the factors are presented in online supplemental file 4. Below, we describe the most frequently reported factors.

      Supplemental material

      Factors related to low-value care

      Regarding the factors that relate to the low-value care, evidence and a consensus among clinicians were the most frequently mentioned factors. These factors both facilitated de-implementation when they were present, and hindered de-implementation when they were absent.

      Individual health professional factors

      A major barrier related to individual health professionals was a lack of knowledge about the low-value care. The knowledge and a belief that the care’s harms outweighed its benefits facilitated de-implementation. For example, receiving a reminder of the fact that urinary catheters cause discomfort and lead to infections motivated clinicians to remove them more promptly. Another major barrier is the clinicians’ fear of missing disease, and discomfort with uncertainty. In addition, clinicians felt that by providing low-value care they were meeting their patient’s wishes or were able to reassure them. On the other hand, they were motivated to reduce low-value care by a focus on improving patient care.

      Patient factors

      Patients’ knowledge of the potential harm, lack of benefit and cost of low-value care, facilitated its reduction. For example, when patients with chronic obstructive pulmonary disease were informed in a focus group about the lack of benefit of inhaled corticosteroids, they felt a need to immediately reduce them. However, de-implementation was hindered by frightening stories or incorrect information on the internet. Patients were sometimes afraid of a disease, such as gastric cancer when they had dyspepsia, and wanted reassurance. A lack of trust in, or suspicion of, their clinician also hindered de-implementation.

      Professional interactions

      Regarding the professional interactions, de-implementation was hindered by a lack of support and trust, as well as a lack of coordination and collaboration. For example, it was sometimes unclear which clinician was responsible for reducing the low-value care. The convenience and high accessibility of the low-value care also hindered de-implementation. An example of this is the use of standard laboratory packages in the medical ordering system. The growing consciousness among clinicians that more is not always better, as well as good collaboration and support, facilitated de-implementation.

      Incentives and resources

      Regarding incentives and resources, de-implementation was hindered by a lack of time, both to communicate with the patient and to participate in the project. It takes more time not to provide low-value care, for example, because patients need to be taught how to check their own skin for cancer in order to reduce follow-up visits to the dermatologist. A potential reduction of revenue was also a barrier to de-implementation in many projects. Clinicians felt hindered to reduce procedures that are reimbursed, such as surveillance visits and insertion of a catheter. In addition, several hospitals and clinicians did not participate in a project because of a fear of reduced revenue.

      Experiences with strategy components

      Below, we describe the experiences reported frequently by the project teams, the target clinicians, and the patients regarding the different components of their de-implementation projects. Online supplemental file 5 shows all experiences.

      Supplemental material

      Education

      Educating clinicians was seen as a useful component of the de-implementation strategy as it enabled them to receive up-to-date information about the low-value care and its side effects. Project 5 included a second educational meeting which focused on practising on a simulated patient, and project 3 showed and discussed a video on communicating with a patient, which helped clinicians to explain to the patient why the care provided is of low value. However, meetings were sometimes either hard to schedule, or could not be attended by all the clinicians. It helped to use existing structures such as weekly meetings. Clinicians found educational material, such as a pocket card, useful. We noted that a lack of repetition contributed to falling back into old patterns. Some terminology, such as ‘unnecessary care’, and the focus on costs, caused resistance among clinicians.

      Clinical champions

      Two projects appointed clinical champions in the participating hospitals. Their task was to bring the subject regularly to the attention of their colleagues and to further spread the educational materials or feedback reports. The way clinical champions fulfilled their role varied. Some spread the messages more actively than others. Clinical champions who left the department or worked in a laboratory did not have as much influence because they did not work near the target group.

      Feedback

      Giving feedback to clinicians offered insight into the prevalence of low-value care and comparing their own performance to those of their peers motivated them to perform better. Some clinicians’ first reaction was scepticism towards the validity of the data. After reassurance that the data were valid, these clinicians were able to acknowledge that there was room for improvement. Moreover, they were willing to improve. Some projects found the data collection for the feedback time-consuming or even impossible to achieve in time.

      Patient information

      Patient information was a valuable de-implementation strategy component, especially in the projects where the patient was an important factor, such as in the reduction of surveillance visits after basal cell carcinoma. However, some factors regarding the spread and content of the material may have limited its effect in other projects. Distribution of the material to patients was not always optimal. Some clinicians considered the information too difficult for patients to understand. Lastly, some clinicians reported that, contrary to its aim, the video clip and poster on vitamin testing in the waiting room led to more requests for vitamin tests, especially for general practices with low preintervention rates of vitamin tests.

      Organisational changes

      Organisational improvements in ordering systems or the structure of electronic patient records helped to break habits, although implementing these changes was difficult and took a long time. According to the clinicians, giving routine attention to the subject helped them to remember the message.

      Financial incentives

      One project tried to arrange a shared savings contract with insurers, but this could not be achieved within the time frame of the project.

      Project approach

      The project leaders reported that they found it very valuable to perform a problem analysis and so achieve greater insight into the context surrounding the practice of low-value care. They used this information to tailor their de-implementation strategy to meet the needs of clinicians and patients and to tackle the barriers that they experience. The problem analysis also created support for the upcoming strategy among the target group. Several project leaders also thought that having a clinician in their project team was essential for recruiting hospitals or general practitioners (GPs) and for providing the education. Lastly, some project leaders found it challenging to collect the right data to evaluate their strategy, because routine hospital or GP data proved to be time-consuming to acquire, was not up to date, or provided insufficient detail to distinguish low-value from high-value care.

      Discussion

      Effects on clinical practice

      Five out of the eight projects found a reduction of low-value care following their de-implementation strategy. Two of these five projects compared their results to a control group and found greater reductions in the intervention group. Three out of the eight projects found no effect of the de-implementation strategy. One of these did show a significant reduction in the control group, while another project showed equal reductions in both the control and intervention groups. Both projects reported that the low-value care they targeted received a lot of attention from clinicians nationally, which could have blurred the effect of the strategy and explain the reduction that they found across the country. A comparable dissemination process of seven Choosing Wisely recommendations that recommended against low-value care practices has resulted in a reduction in two out of the seven low-value care practices.6 This could suggest that dissemination of recommendations including publicity can be sufficient for reducing a part of low-value care practices. The last project with no effect found a non-significant reduction in low-value care in the intervention period, but this was followed by a significant increase in low-value care use after the intervention period, indicating that any potential effect disappeared directly. Unfortunately, this happens more often to de-implementation projects.26 This shows the importance of choosing interventions that have sustained results, such as system-focused interventions.27

      Barriers and facilitating factors for de-implementation

      A lack of time for the patient, an inability to reassure the patients, a desire to meet the patients’ wishes and the financial consequences, were frequent barriers to successful de-implementation experienced by clinicians in our study. Both clinicians and patients were hindered by their fear of disease and their search for reassurance, and facilitated by knowledge of the harm associated with low-value care. Reducing low-value care is easier when it is sufficiently supported by the evidence and by consensus among clinicians. Improved collaboration between professions, improved accessibility of the alternative to low-value care and media attention can help to reduce low-value care.

      Several of these barriers and facilitators, such as the clinicians’ move away from harmful care and their fear of missing a diagnosis, could be connected to the clinicians’ motivation to provide the best care for their patients. Two recent studies confirm the importance of harm reduction as a motivator.12 28 Another connecting theme seems to be the effort that goes into providing less care and communicating this with patients. Patient expectations and a lack of time to turn these around are frequently reported barriers to reducing low-value care.29–37 In our eight projects, a fear of malpractice was not identified as a barrier, contrary to several other studies from the USA.29 33 This might indicate that malpractice claims have a smaller influence in the Netherlands. Other studies confirm this. Only 10% of GPs in the Netherlands provide low-value care because of a fear of claims35 compared with 50%–73% of the primary care physicians in the USA.33 Fear of malpractice did not emerge at all in our study, possibly because of clinicians’ socially desirable responses.

      Experiences with strategy components

      Repeated education on the low-value care and on patient communication, as well as feedback were highly valued components of the de-implementation strategies. However, they were hindered by a lack of time to participate in the projects, and difficulties with the availability of data. Patient information was highly valuable when the low-value care was requested by patients. Choosing the right message and content appeared to be crucial for successful patient information.

      Two systematic reviews found that multicomponent interventions have the greatest potential in reducing low-value care.2 8 Two of our projects which targeted only patients achieved significant reductions in low-value care. This suggests that a single intervention can also be effective, although the success of any intervention is generally hard to predict and is likely to depend on the match between barriers and facilitating factors as well as the chosen strategy.38 Furthermore, Colla and colleagues concluded that supporting clinical decisions, performance feedback and provider education are promising strategies.2 Our study confirms this while adding patient information as another promising strategy. Additionally, in our practical recommendations (box 2), we provide conditions for the success of these strategies.

      Box 2

      Practical recommendations for de-implementation projects

      Practical recommendations for de-implementation projects based on our evaluation are:

      • To reduce only low-value care that has sufficient evidence, and consensus among clinicians, of being of low value. When the field is not ready for de-implementation, you risk provoking discussions among clinicians, achieving less or no effect.

      • To perform a problem analysis of the low-value care practice you are aiming to reduce and study the context of your project. Then tailor the de-implementation strategy to the barriers and facilitating factors you have found.

      Some tips about specific parts of the strategy are:

      • Educating clinicians and improving their communication skills can be useful, especially when existing meetings are used and the message is repeated.

      • To provide regular feedback if data are easily available in order to motivate clinicians to reduce their use of low-value care.

      • To provide information material for patients when they request the low-value care, while ensuring it is the right length, has the right message and is distributed by clinicians.

      • To promote organisational changes such as providing tools to support clinical decision-making in order to challenge previous patterns of practice.

      • To be aware that a lack of time and a loss of revenue can be major barriers to de-implementation. There may be no easy solution for this.

      • To focus on improving the quality and safety of care instead of saving costs. Clinicians and patients are motivated to reduce low-value care when they learn about its burden and harm.

      • To be aware that reducing low-value care can evoke fear and uncertainty in both clinicians and patients.

      Our study is the first that combines the lessons from multiple multicentre de-implementation projects. It is complementary to the study by Stinnett-Donnelly and colleagues that described the lessons from local de-implementation projects in one medical centre.11 They found that the value of a project, such as the reduction in patient harm, promotes de-implementation. They also showed that more controversial care practices among clinicians require more effort to de-implement, and that data collection could be labour-intensive.11 Parker and colleagues identified several challenges and facilitators for leaders of de-implementation projects, such as the availability of data and harm reduction.12 We confirmed their findings and identified more lessons regarding both the barriers and facilitating factors, and the promising components of a de-implementation project.

      Strengths and limitations

      The strength of our study is the prospective design, which enabled us to observe the project leaders’ experiences throughout all steps of the projects. Another strength is that we were able to combine their experiences since the projects had the same structure, even though they were performed in different regions and targeted different practices. However, this diversity can also be a limitation with regard to their comparability.

      The validity of our results depends on the quality of the methodology used in the eight projects. Three of the five projects that achieved a reduction in low-value care did not compare their intervention to a concurrent control group. Before that reason we do not know to what extent their reduction in low-value care can be attributed to a national trend instead of to the de-implementation strategy adopted by the project. It could therefore be the case that our results overestimate the effects of a de-implementation strategy. Other items that indicate the quality of a project, such as blinding or randomisation, are reported in the papers of the individual projects (referenced in online supplemental file 3).

      A second limitation is that the qualitative analysis of the projects is conducted by two authors who were part of the coordinating team that supported the eight projects. This might have biased both the experiences that the project leaders reported in the interviews and logs, and the authors in their analysis, to present a more favourable picture of the projects.

      The projects’ method and time point of identifying the barriers and factors facilitating de-implementation varied. It is possible that some projects missed relevant factors. Regarding the experiences with the different components of the projects, the results are based on the evaluation and subjective experiences of the project leaders. Other project leaders may have different experiences.

      Implications for research and practice

      Many hospitals and general practices in the Netherlands participated in the eight projects described. This has amounted to the prevention of tens of unnecessary endoscopies and dermatology visits, hundreds of unnecessary catheters, and thousands of unnecessary vitamin and laboratory tests. The next step is to sustain these results and spread them to other hospitals in the Netherlands. The five successful projects are currently being spread throughout the Netherlands and of three projects the long-term effects will be measured. The changes that our projects achieved should transcend their project setting and become a permanent part of clinical practice. However, few de-implementation projects evaluate long-term sustainability and more knowledge on this is required.3 The majority of the literature on the spread and dissemination of projects is focused on implementation rather than de-implementation, such as the theory of Rogers.39 Research is necessary to evaluate whether these theories are also relevant for de-implementation projects.

      The costs saved to Dutch society associated with a reduction in low-value care are hard to achieve and measure. Some savings can only be realised by reducing equipment and personnel, which is hard to realise in the short term. Also, the costs associated with all potential unintended consequences of the strategy, such as an increase in the use of other care, should be monitored. Further research is necessary into the potential for cost savings.

      Our findings can support clinicians and researchers in leading more successful de-implementation initiatives by providing examples of the barriers, facilitating factors and valuable components drawn from our eight de-implementation projects. We have combined their results and experiences and translated them into practical recommendations for de-implementation projects (box 2).

      Conclusions

      Successfully reducing low-value care is possible in spite of the powerful barriers opposing it. The eight de-implementation projects managed to recruit many hospitals and general practices. Five of these achieved significant results without measuring negative consequences. We offer practical recommendations for reducing low-value care successfully and preventing patient harm. These include: reduce only low-value care that is supported by sufficient evidence; tailor the strategy to counter the barriers; use repeated education and feedback for clinicians; provide carefully developed patient information when patients request the low-value care; and adapt the organisation to support this change.

      Data availability statement

      Data are available upon reasonable request. De-identified reports of the eight projects are available (in Dutch) from the corresponding author upon reasonable request.

      Ethics statements

      Patient consent for publication

      Ethics approval

      This study involves human participants. For this interview study that evaluated the effects and experiences of eight projects, ethical approval was not required under Dutch national law. The eight projects that are studied obtained ethical approval before the start of their study. Participants gave informed consent to participate in the study before taking part.

      Acknowledgments

      The authors thank the independent committee that selected the eight projects, the advisory board that advised on the programme, and editor Tony Sheldon who helped to create this manuscript, for their time and effort.

      References

      1. ↵
        1. Brownlee S,
        2. Chalkidou K,
        3. Doust J, et al
        . Evidence for overuse of medical services around the world. Lancet 2017;390:156–68.doi:10.1016/S0140-6736(16)32585-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/28077234
        OpenUrlCrossRefPubMed
      2. ↵
        1. Colla CH,
        2. Mainor AJ,
        3. Hargreaves C, et al
        . Interventions aimed at reducing use of low-value health services: a systematic review. Med Care Res Rev 2017;74:507–50.doi:10.1177/1077558716656970pmid:http://www.ncbi.nlm.nih.gov/pubmed/27402662
        OpenUrlCrossRefPubMed
      3. ↵
        1. Born K,
        2. Kool T,
        3. Levinson W
        . Reducing overuse in healthcare: advancing choosing wisely. BMJ 2019;367:l6317.doi:10.1136/bmj.l6317pmid:http://www.ncbi.nlm.nih.gov/pubmed/31690578
        OpenUrlFREE Full Text
      4. ↵
        1. Levinson W,
        2. Born K,
        3. Wolfson D
        . Choosing wisely campaigns: a work in progress. JAMA 2018;319:1975–6.doi:10.1001/jama.2018.2202pmid:http://www.ncbi.nlm.nih.gov/pubmed/29710232
        OpenUrlCrossRefPubMed
      5. ↵
        1. Kerr EA,
        2. Kullgren JT,
        3. Saini SD
        . Choosing wisely: how to fulfill the promise in the next 5 years. Health Aff 2017;36:2012–8.doi:10.1377/hlthaff.2017.0953pmid:http://www.ncbi.nlm.nih.gov/pubmed/29137505
        OpenUrlPubMed
      6. ↵
        1. Rosenberg A,
        2. Agiro A,
        3. Gottlieb M, et al
        . Early trends among seven recommendations from the choosing wisely campaign. JAMA Intern Med 2015;175:1913–20.doi:10.1001/jamainternmed.2015.5441pmid:http://www.ncbi.nlm.nih.gov/pubmed/26457643
        OpenUrlPubMed
      7. ↵
        1. Hong AS,
        2. Ross-Degnan D,
        3. Zhang F, et al
        . Small Decline In Low-Value Back Imaging Associated With The 'Choosing Wisely' Campaign, 2012-14. Health Aff 2017;36:671–9.doi:10.1377/hlthaff.2016.1263pmid:http://www.ncbi.nlm.nih.gov/pubmed/28373333
        OpenUrlAbstract/FREE Full Text
      8. ↵
        1. Cliff BQ,
        2. Avanceña ALV,
        3. Hirth RA, et al
        . The impact of choosing wisely interventions on low-value medical services: a systematic review. Milbank Q 2021;99:1024–58.doi:10.1111/1468-0009.12531pmid:http://www.ncbi.nlm.nih.gov/pubmed/34402553
        OpenUrlCrossRefPubMed
      9. ↵
        1. Elshaug AG,
        2. Rosenthal MB,
        3. Lavis JN, et al
        . Levers for addressing medical underuse and overuse: achieving high-value health care. Lancet 2017;390:191–202.doi:10.1016/S0140-6736(16)32586-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/28077228
        OpenUrlCrossRefPubMed
      10. ↵
        1. Mayer J,
        2. Nachtnebel A
        . DISINVESTING from ineffective technologies: lessons learned from current programs. Int J Technol Assess Health Care 2015;31:355–62.doi:10.1017/S0266462315000641pmid:http://www.ncbi.nlm.nih.gov/pubmed/26694654
        OpenUrlPubMed
      11. ↵
        1. Stinnett-Donnelly JM,
        2. Stevens PG,
        3. Hood VL
        . Developing a high value care programme from the bottom up: a programme of faculty-resident improvement projects targeting harmful or unnecessary care. BMJ Qual Saf 2016;25:901–8.doi:10.1136/bmjqs-2015-004546pmid:http://www.ncbi.nlm.nih.gov/pubmed/26700543
        OpenUrlFREE Full Text
      12. ↵
        1. Parker G,
        2. Kastner M,
        3. Born K, et al
        . Understanding low-value care and associated de-implementation processes: a qualitative study of choosing wisely interventions across Canadian hospitals. BMC Health Serv Res 2022;22:92.doi:10.1186/s12913-022-07485-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/35057805
        OpenUrlPubMed
      13. ↵
        1. Niven DJ,
        2. Mrklas KJ,
        3. Holodinsky JK, et al
        . Towards understanding the de-adoption of low-value clinical practices: a scoping review. BMC Med 2015;13:255.doi:10.1186/s12916-015-0488-zpmid:http://www.ncbi.nlm.nih.gov/pubmed/26444862
        OpenUrlCrossRefPubMed
      14. ↵
        1. Bindraban RS,
        2. van Beneden M,
        3. Kramer MHH, et al
        . Association of a multifaceted intervention with ordering of unnecessary laboratory tests among caregivers in internal medicine departments. JAMA Netw Open 2019;2:e197577.doi:10.1001/jamanetworkopen.2019.7577pmid:http://www.ncbi.nlm.nih.gov/pubmed/31339544
        OpenUrlPubMed
      15. ↵
        1. Hofstede H,
        2. van der Burg HAM,
        3. Mulder BC, et al
        . Reducing unnecessary vitamin testing in general practice: barriers and facilitators according to general practitioners and patients. BMJ Open 2019;9:e029760.doi:10.1136/bmjopen-2019-029760pmid:http://www.ncbi.nlm.nih.gov/pubmed/31594878
        OpenUrlAbstract/FREE Full Text
      16. ↵
        1. Laan BJ,
        2. Nieuwkerk PT,
        3. Geerlings SE
        . Patients knowledge and experience with urinary and peripheral intravenous catheters. World J Urol 2020;38:57–62.doi:10.1007/s00345-018-02623-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/30680418
        OpenUrlPubMed
      17. ↵
        1. Laan BJ,
        2. Maaskant JM,
        3. Spijkerman IJB, et al
        . De-implementation strategy to reduce inappropriate use of intravenous and urinary catheters (RICAT): a multicentre, prospective, interrupted time-series and before and after study. Lancet Infect Dis 2020;20:864–72.doi:10.1016/S1473-3099(19)30709-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/32151333
        OpenUrlPubMed
      18. ↵
        1. Rietbergen T,
        2. Diercks RL,
        3. Anker-van der Wel I, et al
        . Preferences and beliefs of Dutch orthopaedic surgeons and patients reduce the implementation of "Choosing Wisely" recommendations in degenerative knee disease. Knee Surg Sports Traumatol Arthrosc 2020;28:3101–17.doi:10.1007/s00167-019-05708-8pmid:http://www.ncbi.nlm.nih.gov/pubmed/31555844
        OpenUrlPubMed
      19. ↵
        1. van Egmond S,
        2. Wakkee M,
        3. van Rengen A, et al
        . Factors influencing current low-value follow-up care after basal cell carcinoma and suggested strategies for de-adoption: a qualitative study. Br J Dermatol 2019;180:1420–9.doi:10.1111/bjd.17594pmid:http://www.ncbi.nlm.nih.gov/pubmed/30597525
        OpenUrlPubMed
      20. ↵
        1. de Jong JJ,
        2. Lantinga MA,
        3. Tan ACITL,
        4. Tan A, et al
        . Web-Based educational intervention for patients with uninvestigated dyspepsia referred for upper gastrointestinal tract endoscopy: a randomized clinical trial. JAMA Intern Med 2021;181:825.doi:10.1001/jamainternmed.2021.1408pmid:http://www.ncbi.nlm.nih.gov/pubmed/33900373
        OpenUrlPubMed
      21. ↵
        1. Vugt Svan,
        2. de Schepper E,
        3. van Delft S, et al
        . Effectiveness of professional and patient-oriented strategies in reducing vitamin D and B12 test ordering in primary care: a cluster randomised intervention study. BJGP Open 2021;5:BJGPO.2021.0113.doi:10.3399/BJGPO.2021.0113pmid:http://www.ncbi.nlm.nih.gov/pubmed/34407963
        OpenUrlPubMed
      22. ↵
        1. van Egmond S,
        2. van Vliet ED,
        3. Wakkee M, et al
        . Efficacy, cost-minimization, and budget impact of a personalized discharge letter for basal cell carcinoma patients to reduce low-value follow-up care. PLoS One 2022;17:e0260978.doi:10.1371/journal.pone.0260978pmid:http://www.ncbi.nlm.nih.gov/pubmed/35073333
        OpenUrlPubMed
      23. ↵
        1. Rietbergen T,
        2. Marang-van de Mheen PJ,
        3. de Graaf J, et al
        . A tailored intervention does not reduce low value MRI’s and arthroscopies in degenerative knee disease when the secular time trend is taken into account: a difference-in-difference analysis. Knee Surg Sports Traumatol Arthrosc 2022;474.doi:10.1007/s00167-022-06949-w
      24. ↵
        1. Moore GF,
        2. Audrey S,
        3. Barker M, et al
        . Process evaluation of complex interventions: medical Research Council guidance. BMJ 2015;350:h1258.doi:10.1136/bmj.h1258pmid:http://www.ncbi.nlm.nih.gov/pubmed/25791983
        OpenUrlFREE Full Text
      25. ↵
        1. Grol R,
        2. Wensing M,
        3. Eccles MP
        . Improving patient care: the implementation of change in health care. John Wiley & Sons, 2013.
      26. ↵
        1. Bhatia RS,
        2. Dudzinski DM,
        3. Milford CE, et al
        . Educational intervention to reduce inappropriate transthoracic echocardiograms: the need for sustained intervention. Echocardiography 2014;31:916–23.doi:10.1111/echo.12505pmid:http://www.ncbi.nlm.nih.gov/pubmed/24447139
        OpenUrlCrossRefPubMed
      27. ↵
        1. Soong C,
        2. Shojania KG
        . Education as a low-value improvement intervention: often necessary but rarely sufficient. BMJ Qual Saf 2020;29:353–7.doi:10.1136/bmjqs-2019-010411pmid:http://www.ncbi.nlm.nih.gov/pubmed/31843878
        OpenUrlFREE Full Text
      28. ↵
        1. Pickering AN,
        2. Walter EL,
        3. Dawdani A, et al
        . Primary care physicians' approaches to low-value prescribing in older adults: a qualitative study. BMC Geriatr 2022;22:152.doi:10.1186/s12877-022-02829-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/35209856
        OpenUrlPubMed
      29. ↵
        1. Colla CH,
        2. Kinsella EA,
        3. Morden NE, et al
        . Physician perceptions of choosing wisely and drivers of overuse. Am J Manag Care 2016;22:337.pmid:http://www.ncbi.nlm.nih.gov/pubmed/27266435
        OpenUrlPubMed
      30. ↵
        1. Bishop TF,
        2. Cea M,
        3. Miranda Y, et al
        . Academic physicians' views on low-value services and the choosing wisely campaign: a qualitative study. Healthc 2017;5:17–22.doi:10.1016/j.hjdsi.2016.04.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/28668198
        OpenUrlPubMed
      31. ↵
        1. Colla CH,
        2. Mainor AJ
        . Choosing wisely campaign: valuable for providers who knew about it, but awareness remained constant, 2014-17. Health Aff 2017;36:2005–11.doi:10.1377/hlthaff.2017.0945pmid:http://www.ncbi.nlm.nih.gov/pubmed/29137515
        OpenUrlPubMed
      32. ↵
        1. Lyu H,
        2. Xu T,
        3. Brotman D, et al
        . Overtreatment in the United States. PLoS One 2017;12:e0181970.doi:10.1371/journal.pone.0181970pmid:http://www.ncbi.nlm.nih.gov/pubmed/28877170
        OpenUrlCrossRefPubMed
      33. ↵
        1. Zikmund-Fisher BJ,
        2. Kullgren JT,
        3. Fagerlin A, et al
        . Perceived Barriers to Implementing Individual Choosing Wisely® Recommendations in Two National Surveys of Primary Care Providers. J Gen Intern Med 2017;32:210–7.doi:10.1007/s11606-016-3853-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/27599491
        OpenUrlCrossRefPubMed
      34. ↵
        1. van Dulmen SA,
        2. Naaktgeboren CA,
        3. Heus P, et al
        . Barriers and facilitators to reduce low-value care: a qualitative evidence synthesis. BMJ Open 2020;10:e040025.doi:10.1136/bmjopen-2020-040025pmid:http://www.ncbi.nlm.nih.gov/pubmed/33127636
        OpenUrlAbstract/FREE Full Text
      35. ↵
        1. Kool RB,
        2. Verkerk EW,
        3. Winnemuller LJ, et al
        . Identifying and de-implementing low-value care in primary care: the GP's perspective-a cross-sectional survey. BMJ Open 2020;10:e037019.doi:10.1136/bmjopen-2020-037019pmid:http://www.ncbi.nlm.nih.gov/pubmed/32499273
        OpenUrlAbstract/FREE Full Text
      36. ↵
        1. Ingvarsson S,
        2. Augustsson H,
        3. Hasson H, et al
        . Why do they do it? A grounded theory study of the use of low-value care among primary health care physicians. Implement Sci 2020;15:93.doi:10.1186/s13012-020-01052-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/33087154
        OpenUrlPubMed
      37. ↵
        1. Augustsson H,
        2. Ingvarsson S,
        3. Nilsen P, et al
        . Determinants for the use and de-implementation of low-value care in health care: a scoping review. Implement Sci Commun 2021;2:13.doi:10.1186/s43058-021-00110-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/33541443
        OpenUrlPubMed
      38. ↵
        1. Grimshaw JM,
        2. Patey AM,
        3. Kirkham KR, et al
        . De-implementing wisely: developing the evidence base to reduce low-value care. BMJ Qual Saf 2020;29:409–17.doi:10.1136/bmjqs-2019-010060pmid:http://www.ncbi.nlm.nih.gov/pubmed/32029572
        OpenUrlAbstract/FREE Full Text
      39. ↵
        1. Rogers EM
        . Lessons for guidelines from the diffusion of innovations. Jt Comm J Qual Improv 1995;21:324–8.doi:10.1016/S1070-3241(16)30155-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/7581733
        OpenUrlPubMed

      Supplementary materials

      Footnotes

      • Collaborators To do or not to do programme collaborators: Corina de Jong, Janwillem Kocks, Aniek de Coninck, Harry C Schouten, Leti van Bodegom-Vos, Tessa Rietbergen, Bart J Laan, Suzanne E Geerlings, Evelien IT de Schepper, Saskia F van Vugt, Prabath WB Nanayakkara, Renuka S Bindraban, Sven van Egmond, Marlies Wakkee, Judith J de Jong, Joost PH Drenth.

      • Contributors EWV, SAvD, GPW, LH, PH and RBK were involved in the study concept and design. RBK obtained funding. EWV and PH performed the interviews. EWV performed the analyses, which PH verified. EWV, SAvD, PH and RBK interpreted the data. EWV drafted the manuscript. RBK was the guarantor of this study. The To do or not to do programme collaborators led the eight projects that were evaluated. All authors participated in critical revision of the manuscript for important intellectual content. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. All authors approved the final version of this paper and agree to be accountable for all aspects of the work. The To do or not to do programme collaborators group consists of the following authors: Corina de Jong, Janwillem Kocks, Aniek de Coninck, Harry C Schouten, Leti van Bodegom-Vos, Tessa Rietbergen, Bart J Laan, Suzanne E Geerlings, Evelien IT de Schepper, Saskia F van Vugt, Prabath WB Nanayakkara, Renuka S Bindraban, Sven van Egmond, Marlies Wakkee, Judith J de Jong, Joost PH Drenth. The To do or not to do programme collaborators led the eight projects that were evaluated. They recruited the participants, performed the problem analysis, the strategy, and the evaluation of their project, and they were interviewed for this study. Corina de Jong and Janwillem Kocks led project 1; Aniek de Coninck and Harry C Schouten led project 2; Leti van Bodegom-Vos and Tessa Rietbergen led project 3; Bart J Laan and Suzanne E Geerlings led project 4; Evelien IT de Schepper and Saskia F van Vugt led project 5; Prabath WB Nanayakkara and Renuka S Bindraban led project 6; Sven van Egmond and Marlies Wakkee led project 7; and Judith J de Jong and Joost PH Drenth led project 8.

      • Funding This work was supported by ZonMw, the Netherlands Organisation for Health Research and Development (grant number 839201 002).

      • Competing interests All authors had financial support from ZonMw for the submitted work. ZonMw is an independent organisation for health research and development that distributes grants on behalf of the Dutch government. Kocks reports grants and personal fees from AstraZeneca, grants and personal fees from Boehringer Ingelheim, grants from Chiesi, grants and personal fees from GSK, grants from Mundi Pharma, grants from TEVA, outside the submitted work; all fees go to the institute. Drenth reports grants from Gilead outside the submitted work; all the proceeds go directly to the institute.

      • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

      • Provenance and peer review Not commissioned; externally peer reviewed.

      • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.