You are here

  • Home
  • Archive
  • Volume 11, Issue 4
  • ‘Give me something meaningful’: GPs perspectives on how to improve an audit and feedback report provided by health insurers – an exploratory qualitative study

Article Text

Article menu
Original research
‘Give me something meaningful’: GPs perspectives on how to improve an audit and feedback report provided by health insurers – an exploratory qualitative study
  1. P J G M de Bekker1,2,
  2. V de Weerdt1,3,
  3. M D H Vink1,4,
  4. A B van der Kolk5,
  5. M H Donker6,
  6. E J E van der Hijden1,7
  1. 1Department of Health Economics & Talma Institute, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
  2. 2Zorgvuldig Advies, Utrecht, Netherlands
  3. 3Amsterdam University Medical Centres, Holendrecht, Netherlands
  4. 4Gynaecology, Amsterdam Universitair Medische Centra, Duivendrecht, Netherlands
  5. 5Talma Institute, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
  6. 6Department of Health Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
  7. 7Zilveren Kruis Health Insurance, Zeist, Netherlands
  1. Correspondence to Mr P J G M de Bekker; pietdebekker{at}zorgvuldigadvies.nl

Abstract

Background Audit and feedback (A&F) is a valuable quality improvement strategy, which can contribute to de-implementation of low-value care. In the Netherlands, all health insurers collaboratively provide A&F to general practitioners (GPs), the ‘Primary Care Practice Report’ (PCPR). Unfortunately, the use of this report by GPs is limited. This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR.

Method We used an interpretative qualitative design, with think-aloud tasks to uncover thoughts of GPs on the usability of the PCPR and semistructured interview questions to ask GPs’ recommendations for improvement of the PCPR. Interviews were audiorecorded and transcribed ad verbatim. Data were analysed using thematic content analysis.

Results We identified two main themes: ‘poor usability of the PCPR’, and ‘minimal motivation to change based on the PCPR’. The GPs found the usability of the PCPR poor due to the feedback not being clinically meaningful, the data not being recent, individual and reliable, the performance comparators offer insufficient guidance to assess clinical performance, the results are not discussed with peers and the definitions and visuals are unclear. The GPs recommended improving these issues. The GPs motivation to change based on the PCPR was minimal.

Conclusions The GPs evaluated the PCPR as poorly usable and were minimally motivated to change. The PCPR seems developed from the perspective of the reports’ commissioners, health insurers, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback could contribute to their motivation to improve clinical performance.

Furthermore, the GPs stated that they receive a multitude of A&F reports, which they hardly use. Thus, we see a need for policy makers to invest in less, but more usable A&F reports.

  • Audit and feedback
  • Quality improvement
  • General practice
  • Health policy

Data availability statement

Data are available on reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjoq-2022-002006

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Audit and feedback (A&F) is an often-used quality improvement strategy, with varying effectiveness.

    WHAT THIS STUDY ADDS

    • This study strengthens findings on how to design A&F that professionals deem usable and add the perspective of general practitioners on claims data as the source for A&F.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • A call for action to make use of existing knowledge on A&F and user-centred design and invest in less, but better A&F.

    Background

    Audit and feedback (A&F) is one of the most frequently used quality improvement strategies across healthcare settings, including primary care.1 A&F is defined as ‘a summary of clinical performance over a specific period of time, and the provision of that summary to individual practitioners, teams or healthcare organisations’.1 Theory behind A&F is that, by providing clinicians feedback on clinical performance, they are stimulated to change their clinical behaviour and thereby improve clinical practice and patient outcomes.2–4 While numerous trials show A&F is effective, the effectiveness of A&F interventions varies widely.1

    There remains uncertainty regarding how to design A&F in such a way it is maximally effective in contributing to quality improvement.5 6 Research shows that the effectiveness of A&F depends on the design of the intervention itself, recipient characteristics, and the context in which the intervention is performed.1 7–9 Regarding the design of the intervention evidence shows that A&F is most effective when the ‘frequency of feedback is more than once, the format is both written and verbal, feedback is given by a respected peer or colleague and feedback is accompanied by specific goals and an action plan’.1 10 Furthermore, A&F may be more effective if it is ‘actionable’, meaning the feedback is timely, individual and non-punitive.11–13 Also, research suggests that performance comparators in A&F ideally include trends and specific targets and are compared with similar peers.14

    Quality improvement strategies are necessary in many Western countries, as they struggle with the provision of low-value care (LVC), that is, care is not effective or not cost-effective.15 16 The provision of LVC reduces the overall quality of care, potentially causes patient harm and is ineffective use of scarce healthcare resources. In primary care, LVC entails prescribing of ineffective medications and unnecessary referrals, diagnostics and interventions.17–19 A&F has been shown to be effective in de-implementing LVC in primary care.20–22

    In the Netherlands, health insurers collectively commission the development of an A&F report, the ‘Primary Care Practice Report’ (PCPR)[1, to stimulate general practitioners (GPs) to improve their clinical behaviour. The PCPR presents benchmark information on consultations, diagnostics, interventions and prescriptions in primary care; and numbers of patients using medical specialist care and mental care. Unfortunately, the use of the PCPR by Dutch GPs is limited: only 60% of the GPs have ever downloaded the PCPR between 2015 and 2019 and repeated use of the PCPR is not customary.2 While literature gives some theoretical guidance regarding how to design A&F, large uncertainty remains on how to design A&F. The limited use of the PCPR clearly illustrates a gap between theory and how an A&F report is designed in practice. Therefore, we examined the thoughts of GPs on the usability of the PCPR and second, we examined GPs recommendations for improving the PCPR.

    Method

    For reporting of this study, we followed the COnsolidated criteria for REporting Qualitative research, the COREQ-checklist can be found in online supplemental file A.23

    Supplemental material

    Design

    We used an interpretative qualitative design to explore the perspectives of GPs on the usability of the PCPR.

    We used think-aloud tasks to examine the thoughts of GPs on the usability of the PCPR and we used a semi-structured interview to ask GPs for their recommendations on how to improve the usability of the PCPR (online supplemental file B).

    Supplemental material

    In think-aloud methodology, participants are asked to perform a task and verbalise any occurring thoughts while doing so.24 25 This methodology is valuable to get a complete and realistic view of users’ perspective on tools, such as the PCPR.26 In each interview, we showed GPs a representative selection of tables and graphs (online supplemental file C) from the PCPR and asked the participating GPs to ‘think aloud’ while interpreting these tables and graphs.

    Supplemental material

    Patient and public involvement

    Due to the nature of the study, patients and public were not involved in the design of this study.

    The PCPR

    The PCPR is a nationwide benchmark tool for GPs developed since 2015 by VEKTIS in collaboration with health insurers and GPs. VEKTIS is a business intelligence centre that is established and financed by the Dutch Association of Health insurers,3 in which all Dutch health insurers are represented.

    The PCPR provides GPs insight in the care use of their patients and aims to contribute to GPs' insights in the quality and cost-effectiveness of the care they provide. The PCPR is based on claims data, received by VEKTIS from all Dutch health insurers on insured citizens which covers about 99% of the Dutch population. The feedback includes costs and/or volume indicators on seven main topics: (1) the practice patient population, which includes figures on the volume of the patient population, presented by age, gender and socioeconomic status (SES); (2) the practice total care costs, (3) the GP care provided, which includes cost and volume indicators on consultations, visits and minor surgical procedures performed in primary care; (4) costs of prescribed drugs; (5) referrals to mental healthcare; (6) referrals to medical specialist care and (7) primary care diagnostics. Excluding general descriptive information, the PCPR consists of 24 pages of feedback, with 86 figures and 22 tables.

    Feedback is presented for the total GP practice, thus if multiple GPs work in one practice the feedback is on group level. Only one figure in the PCPR presents feedback on individual level, this is for the number of diagnostic requests per year. Furthermore, the feedback in the PCPR is given for a specific year, about 1.5 years prior to the release of the report, the PCPR contains some figures with historical trends, and the performance results of a specific practice are in many figures compared to an ‘expected value’. This expected value is based on the national average for a practice with a similar patient population which is case-mix corrected for age, gender, income, pharmaceutical use, and history of Chronic Obstructive Pulmonary Disease (COPD), diabetes, and cardiovascular diseases. This expected value is given as a benchmark to enable GPs to compare their own clinical behaviour. The PCPR is updated once or twice annually, after which GPs are invited via email to freely download the report from the VEKTIS website. Since 2015, seven versions of the PCPR were made available for download.

    Participant recruitment strategy

    We purposively recruited GPs that were diverse in respondent characteristics, such as gender, type of practice (health centre, group practice and duo practice) and geographical distribution of the Netherlands. We received contact details of 80 GPs from the medical faculty and via our personal network. GPs related to the medical faculty offer internships to medical students from the ‘Vrije Universiteit, in Amsterdam, the Netherlands’ and represent a diverse group that work in varying regions of the Netherlands (widely dispersed in the North and Middle region of the Netherlands) and work in various settings (solo practices, duo practices, group practices and health centres). Besides one GP, of whom the interviewer was enlisted in the GP practice, none of the contacted GPs were directly related to the personal network of the authors. Of the 80 GPs that were invited by email or phone for participation in this study, 32 responded to the invitation of which 18 GPs were interested in participation. Fourteen GPs declined; reasons for decline were high work pressure, retirement and acting as a locum. The study aimed to enrol 10–20 participants. This sample size was based on existing think-aloud studies, the likelihood of drop-out and the expected number required to meet data saturation and congruency.27 Data saturation was reached after 10 interviews, a further two interviews were conducted to increase the rigour of the results. GPs participated voluntarily in the study and did not receive any kind of compensation.

    Data collection

    All interviews were conducted by a master student (ABvdK, male), with no clinical affiliation. The interviewer knew one GP prior to the interview, as he was enrolled at this GPs practice. He had no personal relationship with this GP or relation to any of the other GPs.

    Interviews were held from May to July 2018 in the GP practice of the participant. ABvdK was supervised by two of the authors (PJGMdB, MD), with no clinical affiliations or existing relationships with the participants. All participants provided written informed consent.

    Prior to the think-aloud, we asked GPs for descriptive information regarding whether and which types of A&F they used. Then, we used think-aloud tasks to examine the thoughts of GPs on the usability of the PCPR. At the start of each think-aloud session the interviewer briefly explained think-aloud as a method to uncover participants thoughts. The interviewer explained to GPs they would be shown different pages of the PCPR and asked them to reflect on anything they saw or thought when reviewing each page. We asked the GPs to perform the think-aloud task six times, for six different pages of the PCPR, covering 18 out of 86 graphs and 3 out of 22 tables. Since the PCPR does not require an active task but an interpretation and reflection on clinical performance, we used guiding questions to stimulate GPs to think-aloud. Thus, for each page of the PCPR shown, the GPs were asked to answer three subsequent questions (1) What do you see and do you understand what you see? (2) What are your thoughts on what you see? and (3) How does what you see influence you and/or your practice? The six pages for the think-aloud tasks were selected in collaboration with VEKTIS based on diversity in topic of the PCPR, type of healthcare presented and design of graphs. We selected pages that are representative of the total PCPR. The selection included three out of seven topics in the PCPR, namely: (1) patient population: showing the volume of the patient population, presented by age, gender and SES, (2) general practice care: showing volumes of use of care, volumes of various types of consultations and visits, and volumes of minor surgical procedures performed in the GPs practice and (3) general practice diagnostics: showing costs and volumes of diagnostic requests. To assess accuracy of the interpretation of the participant, two researchers discussed the transcripts of the think-aloud tasks and compared results to the legend of the PCPR.

    Second, we asked the GPs semistructured interview questions to share their recommendations on how to improve the usability of the PCPR. Lastly, participants were given the opportunity to make additional statements or ask questions.

    After the first interview, the wording of some questions was altered to more concrete and accurate wording.

    All interviews were audiorecorded. Field notes were made during and immediately after the interviews to aid analysis. We sent a summary of each interview to the respective participant for a member check. None of the participants responded. Interviews lasted 30–55 min. No repeat interviews were carried out.

    Data analysis

    The interview recordings were transcribed ad verbatim. We used thematic content analysis to analyse the interviews.27 The analysis was done in two phases. First, a deductive analysis was performed in the context of a master thesis internship, for which two coders independently coded a transcript after each interview was held. The resulting code book was discussed within the wider research team, but since this scheme closely reflected the topic guide it did not fully answer the research questions. To gain deeper understanding in the interviews the research team decided to recode the transcripts inductively, for which initially one reviewer recoded all transcripts. The resulting coding scheme of the inductive approach was briefly discussed in the wider research team to assess the added value of inductive coding. After the research team decided inductive coding was valuable, another researcher (who previously coded deductively) recoded all manuscripts inductively. Subsequently, the two coders discussed the codes per transcript, to ensure consistency in coding. Lastly, the coding scheme was discussed within a wider research team to identify recurrent themes and formulate the final themes. In the themes, we grouped results from the think-aloud tasks and recommendations, since these two approaches identified similar themes. Think-aloud identified negatively framed items, while recommendations identified similar items framed positively.

    All coding was performed using qualitative computer analysis software package, Atlas.ti (V.9.1.3) software. The quotes that best illustrated the different themes and subthemes were selected and translated from Dutch to English. All quotes were then evaluated by the remaining authors and changes were made to reach agreement among all authors.

    Results

    Participant characteristics

    We interviewed 12 GPs, from 10 different practices, aged 35–64 years. The respondents varied in years of work experience, active years in current practice and digital experience, type of practice and work setting (table 1). All GPs were familiar with A&F reports in some form, although most GPs did not regularly and systematically used A&F to review their own performance. Respondents receive many types of A&F reports, such as reports from their own GP information system, accreditation programmes, a recognised health services research institute, health groups or regional networks, individual health insurers, pharmacotherapy networks and diagnostic laboratories. Eight GPs (67%) had heard of the PCPR before the invitation, but only three GPs (25%) had tried to use the PCPR for quality improvement purposes.

    View this table:
    Table 1

    Respondent characteristics

    Themes

    We identified two main themes from the interviews: ‘poor usability of the PCPR’, and ‘minimal motivation to change based on the PCPR’. In the theme ‘poor usability of the PCPR’, we identified the subthemes ‘desire for clinically meaningful feedback’; ‘limitations of claims data for audit’; ‘performance comparators are difficult to interpret’; ‘desire to discuss results with peers’ and ‘unclear definitions and figures’ (online supplemental file D). The second theme did not have subthemes. Recommendations for improvement of the PCPR, if provided from the interviews, are described per theme.

    Supplemental material

    Poor usability of the PCPR

    Desire for clinically meaningful feedback

    The GPs stated that they did not find the feedback clinically meaningful, since feedback on quality of care is lacking, cost indicators reflect prices which GPs cannot influence and detail on conditions is lacking. The GPs did not see how the feedback shown coud help them to improve their clinical practice.

    R7: "It is unclear to me how these figures might help the patient sitting at the other side of my desk."

    R11: “We strive for high quality healthcare, but feedback usually reflects euros or percentages, instead of health gains. For how many people suffering from high blood pressure did my prescription result in a lower blood pressure? And how many of my patients have not had a stroke as a consequence? How do I relate to peers?”

    The GPs found the cost indicators uninteresting and frustrating, since these include variation in prices or costs made in secondary care, which both cannot be influenced by the GP. Also, GPs found the feedback on high level, in which all types of conditions are combined, not clinically meaningful.

    GPs recommended including quality indicators on either clinical outcomes regarding their treatment role, such as the blood pressure of their patients, or indicators on the effectiveness of their gatekeeper role, such as whether longer GP consultations lead to less referrals to secondary care. Furthermore, GPs recommended excluding feedback outside their control, such as costs influenced by prices. Lastly, GPs recommended including feedback with lower detail level, such as for common disorders.

    R1: "I would like to see more figures on my referrals for separate conditions. And this information is mostly cost-related, so price increases have an effect. However, I can’t control those costs."

    Limitations of claims data as a source for A&F

    The GPs found that the claims data on which the PCPR is based has several important limitations. First, half of the GPs indicated that the timeliness of the data was insufficient, with data dating back to 2 years prior to the report release date.

    R11: “Drawback is that it is of course much too late, in the sense that I’m looking at numbers of 2016 (in 2018). Thus, it is not exactly management information. I can only see that I am at the bottom of the canyon, because I missed an exit in 2016”

    R4: "Anyway this feedback is of over two years ago, I find that difficult. The longer ago, the more difficult it is to apply in practice. I would appreciate feedback on my figures from last year somewhere within the first three to four months of the current year. That would really help me."

    Second, the GPs stated that the group level feedback makes it unclear whether deviations in performance are attributable to themselves or to a colleague. Even for the only figure with individual feedback, GPs stated that this may not be attributable to themselves, since diagnostic requests have been made in their name by physician assistants, residents GPs and part-time GPs.

    Third, the GPs questioned the reliability and validity of the data.

    R1: "Then my thoughts are: is it correct? Is it correct for our population?"

    R5: "This seems to be untrue; I do not know how the benchmark is developed; this difference is too big. So, I have my reservations."

    Furthermore, two GPs expressed a lack of trust in the source of the PCPR, they were sceptical about the role of health insurance companies as the supplier of data.

    GPs recommended using recent, individual, reliable, and valid data as the source for feedback.

    Performance comparators offer insufficient guidance to assess clinical performance

    GPs found that the historical trends, the ‘expected value’ as the benchmark and the lack of targets in the PCPR offered insufficient guidance to assess their clinical performance.

    Futhermore, GPs noticed unstable trends in their performance, with high outliers in one year and low outliers in the next. This caused confusion and GPs did not know how to interpret this or how to act on these unstable trends. Also, GPs indicated that policy changes in healthcare organisation or claim codes can make data incomparable over time, hindering interpretation of historical trends.

    R5: "If one year I do too much and the other year I am below expected, then yeah, I think yeah, I really won’t do my work differently."

    Last, GPs found it difficult to compare their clinical performance to the ‘expected value’, since this value is derived from national data and thus includes incomparable practices. Eight of the 12 GPs attributed deviations in their clinical performance to casemix differences, while the figures were already case-mix corrected. In addition, several GPs missed specific targets in the PCPR.

    R2: "Are a hundred colleagues providing too many treatments or am I providing too few? Hmmm, who knows?"

    GPs recommended comparing performance to similar peers, rather than national data. GPs would like to compare to peers within their practice as well as peers from other practices. GPs found it important that the comparator is similar in setting (rural versus urban) and type of care services (whether an ultrasound is available, a Doppler, etc). Also, GPs recommended including regional comparisons and to account for integrated networks. Furthermore, GPs recommended including an advice or an action plan in the feedback, which helps them assess their clinical performance.

    Desire to discuss results with peers

    Based on the feedback, multiple GPs expressed they would like to discuss this feedback with peers from their own practice or peers from other practices. GPs wanted to discuss the causes of deviations in audit results with peers, to gain insights in other ways of organising and delivering care and thus on if and how to change their clinical behaviour.

    Unclear definitions and figures

    Every GP found one or more tables/graphs in the PCPR hard to interpret. The GPs stated that several definitions were unclear, such as ‘primary care diagnostics’, ‘request for diagnostics’. GPs did not know how these indicators were defined and what classified under such categories.

    R12: "I also don’t really know what primary care diagnostics entails, is that only blood tests or does it include more?"

    R11: "I don’t even know about what kind of diagnostics this is now: special diagnostics, scopies, no clue what this is about.”

    GPs recommended including more explanations on definitions and case-mix correction in the tables/graphs instead of only in the preface of the report. Furthermore, GPs found there was an unclear presentation of numbers with varying scales in similar graphs, unclear units and indistinctive colours.

    Minimal motivation for change based on the PCPR

    Most GPs found that the PCPR did not provide the correct information to change, due to the content of the feedback, the type of audit data that was used or the lack of guidance of performance comparators.

    R5: "If you are higher than expected at one year and lower at the year after, then I think: ‘the average seems good, there’s no reason to work differently’. If I have a consultation with a patient who is coughing for half a year, I do not think ‘oh, I’ve applied too much photos so far, let’s not refer this patient.’ You still look at the patient and his needs."

    There were only a few cases in which GPs stated they were motivated to change their behaviour based on the PCPR. They were either motivated to change clinical behaviour, such as increasing intensive visits and increasing Dopplers, or motivated to change claims behaviour.

    R4: "This number strikes me. I think this is due to under registration on our part."

    R11: "We perform less 24hours blood pressure measurements than average. That is good to know, because then we will perform those a bit more, because they earn well. For Dopplers the same. Those we can nicely increase a bit."

    R6: "We should look at our administrative process. Probably we sometimes forget to submit declarations. That is no medical difference, but purely administrative."

    Interestingly, most small deviations of performance did not give GPs motivation to change, while large deviations resulted in GPs questioning the data thus also did not evoke motivation to change. Also, many deviations in performance were explained by GPs as conscious practice management.

    R1: "ECG we indeed don’t perform often, we only do this for rhythmic disorders, and we do that on purpose, we have them done in the hospitals because we believe our quality in assessing ECGs is inadequate."

    R4: "Well, as I said we perform less phone consultations, we agreed not to perform triage, that is conscious policy and we are very happy with that."

    Importantly, the majority of the GPs stated that well designed A&F could potentially motivate them to change behaviour and thus have a meaningful impact on quality of care. Two GPs stated that A&F in general could not change their behaviour. Both GPs stated that this was because aggregated data could not influence how they provide care for an individual patient, one GP states this was because he applies critical thinking continuously and that he would find it difficult to weigh aggregated data more than his own critical thinking and the other that he provides care one patient at a time and that you do want your actions to be ‘medically responsible’, thus that aggregated data does not influence patient care.

    Discussion

    This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR. GPs found the usability of the PCPR poor due to (1) the feedback not being clinically meaningful; (2) the audit data not being recent, individual and reliable; (3) the performance comparators offered insufficient guidance to assess clinical performance; and (4) the results are not discussed with peers and the definitions and visuals are unclear. GPs recommend improving these issues. The poor usability of the report led to minimal motivation to change based on the PCPR.

    Our results are consistent with previous literature on effective A&F and user-centred design; and indicate that policy makers still do not draw sufficiently from existing evidence when developing A&F reports.13 28–31 The PCPR contains information on process and cost indicators aggregated across all types of conditions, based on the healthcare procurement perspective of health insurers. However, this does not match the clinical perspective of the GPs. In our study GPs requested information on clinical outcomes regarding their treatment role and effectiveness of their gatekeeper role, which can be developed based on claims data. Also, the PCPR could present feedback for common disorders and common interventions specifically instead of aggregated data. Aligning feedback with recipient priorities is considered key to improve the effectiveness of A&F.32 Furthermore, the PCPR does not specifically focus on LVC and lacks targets, which makes it difficult for GPs to interpret whether and how they can improve their quality of care. Since claims are not identifiable to one GP and claim processing has a lag-time, it is difficult to meet requirements of recent and individual A&F based on claims data.

    Our study also adds to the knowledge on how to design effective A&F. First, GPs specifically asked for feedback on clinical outcomes and the effectiveness of their gatekeeper role, such as whether more long consultations prevent referrals or whether more email consultations in primary care prevent face-to-face consultations in primary care. It is interesting that GPs in our study requested clinical outcomes as feedback, even though outcomes are not a direct reflection of clinical performance.11 GPs in our study stated that outcome indicators, such as irregulated blood pressures, are important indicators for them to review their clinical behaviour, such as whether they should change their prescriptions or referral behaviour. Second, although using trends as a performance comparator is recommended since trends can create positive motivation towards change, we found that unstable trends could cause confusion in interpreting the feedback.14 To help clinicians interpret trends we advise to include descriptions of relevant policy changes or possible changes of claims code definitions in the caption of feedback. Third, we found that using claims data as the source for A&F in some cases motivated GPs to change claims behaviour instead of clinical behaviour. Since it is the aim of A&F to stimulate reflection on clinical behaviour, it is questionable whether using claims data as the source for feedback distracts from this purpose and whether patient record data or medical registries are better suited data sources. However, it is unknown whether the type of data source used for A&F influences the effectiveness of A&F interventions.33 Lastly, some GPs were sceptical about the role of health insurance companies as the supplier of the data. Health insurers in the Netherlands have the legal task of guarding affordability, accessibility, and efficiency of healthcare. Therefore, professionals express a fear that health insurers will use audit data to reduce healthcare reimbursements. Using A&F in a punitive manner might lower professionals’ motivation to change and thus result in less effective A&F interventions.11

    A comparable qualitative study, which examined how the PCPR and group discussions contribute to GPs motivation to change, found many similar results, such as that feedback should be reliable, recent, individual and concern the own influence sphere.34 Furthermore, this study found that group discussions on the PCPR contributed to GPs motivation to change. It could be that not the quality of the feedback itself, but feedback combined with peer discussion is a vital strategy in A&F effectiveness.34 35 However, studies examining the effectiveness of peer-discussion in A&F interventions and quality improvement show varying results.1 36–38

    Theoretical underpinning of using A&F for quality improvement purposes is that A&F motivates professionals to change.2 4 In our study, most GPs stated they were a priori motivated to change to improve quality and that A&F could contribute to their motivation to change. However, the GPs stated that they find the quality of the PCPR insufficient, thus it did not motivate them to change. In A&F literature lack of quality of feedback is frequently mentioned as a barrier for motivation to change.39 It is possible that lack of quality is mentioned as a barrier for change, because of cognitive dissonance or external attribution, in which professionals search for an external explanation for underperformance.14 40 Experts state that the optimal design of A&F likely depends on recipient factors and context, but it is yet unclear which thresholds exists for quality of feedback to be ‘sufficient’ to contribute to motivation to change6 Furthermore, Desveaux et al35 describe that ‘A&F may not be translated into actual change, due to incapability of participants to interpret A&F in an actionable way’. Thus, even when the quality of the feedback itself is sufficient, recipients may still need education on how to interpret and translate A&F into change.41

    Multiple studies throughout the years describe that GPs find A&F reports unusable, while these same studies do find that GPs are motivated to use A&F for quality improvement.41–43 In addition in our study, GPs state that they receive a multitude of A&F reports, many of which they hardly use. Using low-quality A&F creates waste of resources and leads to ineffective use of clinician’s time. While the PCPR has important strengths, such as support from national policy makers, national coverage, it covers all healthcare provided and it is based on routinely collected data, not all concerns of GPs regarding usability of the PCPR can be solved, such as the timeliness of the data or individual feedback. Thus, it might be more effective to invest in another type of A&F for quality improvement. For more effective quality improvement, we see a need for policy makers to invest in less, but more usable A&F reports.44

    Strengths and limitations

    An important strength of this study was the think-aloud methodology, which enabled us to capture direct reactions to specific issues of the PCPR, instead of capturing general reflections on the PCPR or A&F. Hereby, we were able to identify items that explain the limited uptake and influence the usage of the PCPR. Another strength is that we examined an existing A&F report which is routinely available.

    This study also had several limitations. First, we had a relatively limited recruitment strategy. While the group of GPs was diverse in several characteristics, they were relatively old which may have caused bias. Second, the interviewer was enlisted at the GP practice of one respondent, but since this prior relationship was limited, we do not believe this relationship has altered our results. Third, some of the respondents were familiar with the PCPR prior to the interview, which may have affected their responses to the think-aloud tasks. Lastly, since we did not work iteratively and completely independent during the second analysis phase this might have caused bias in the interpretation of our results. However, we tried to overcome this bias by discussing codes per transcript and discussing themes within the wider research team and since another study regarding the PCPR found similar results, we believe that our results are reliable.

    Conclusions

    The GPs in our study evaluated the PCPR as poorly usable and were minimally motivated to change behaviour based on the PCPR. The PCPR seems developed from the perspective of the commissioner, the health insurer, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback would contribute to their motivation to improve clinical performance.

    Furthermore, GPs in our study stated that they receive a multitude of A&F reports, which they hardly use. For more effective quality improvement, we see a need for policy makers to invest in less, but more usable A&F reports.

    Data availability statement

    Data are available on reasonable request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    According to the regulations of the BETHCIE, this study did not require full ethical review. Participants gave informed consent to participate in the study before taking part.

    Acknowledgments

    We would like to thank Dr. Hanna Willems for her contributions to the manuscript.

    References

    1. ↵
      1. Ivers N,
      2. Jamtvedt G,
      3. Flottorp S, et al
      . Audit and feedback : effects on professional practice and healthcare outcomes (Review). Cochrane Database of Systematic Review 2012;6:1–217.
      OpenUrl
    2. ↵
      1. Kluger AN,
      2. DeNisi A
      . The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119:254–84.doi:10.1037/0033-2909.119.2.254
      OpenUrlCrossRefWeb of Science
    3. ↵
      1. Gardner B,
      2. Whittington C,
      3. McAteer J, et al
      . Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70:1618–25.doi:10.1016/j.socscimed.2010.01.039pmid:http://www.ncbi.nlm.nih.gov/pubmed/20207464
      OpenUrlCrossRefPubMed
    4. ↵
      1. Brown B,
      2. Gude WT,
      3. Blakeman T, et al
      . Clinical performance feedback intervention theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implementation Science 2019;14:1–25.doi:10.1186/s13012-019-0883-5
      OpenUrl
    5. ↵
      1. Foy R,
      2. Eccles MP,
      3. Jamtvedt G, et al
      . What do we know about how to do audit and feedback? pitfalls in applying evidence from a systematic review. BMC Health Serv Res 2005;5:1–7.doi:10.1186/1472-6963-5-50
      OpenUrlCrossRefPubMed
    6. ↵
      1. Ivers NM,
      2. Sales A,
      3. Colquhoun H, et al
      . No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implementation Science 2014;9:1–8.doi:10.1186/1748-5908-9-14
      OpenUrl
    7. ↵
      1. Ivers NM,
      2. Grimshaw JM,
      3. Jamtvedt G, et al
      . Growing literature, stagnant science? systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014;29:1534–41.doi:10.1007/s11606-014-2913-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/24965281
      OpenUrlCrossRefPubMed
    8. ↵
      1. Brown B,
      2. Balatsoukas P,
      3. Williams R, et al
      . Multi-method laboratory user evaluation of an actionable clinical performance information system: implications for usability and patient safety. J Biomed Inform 2018;77:62–80.doi:10.1016/j.jbi.2017.11.008pmid:http://www.ncbi.nlm.nih.gov/pubmed/29146562
      OpenUrlPubMed
    9. ↵
      1. Brehaut JC,
      2. Colquhoun HL,
      3. Eva KW, et al
      . Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med 2016;164:435–41.doi:10.7326/M15-2248pmid:http://www.ncbi.nlm.nih.gov/pubmed/26903136
      OpenUrlCrossRefPubMed
    10. ↵
      1. Hysong SJ
      . Meta-Analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009;47:356–63.doi:10.1097/MLR.0b013e3181893f6bpmid:http://www.ncbi.nlm.nih.gov/pubmed/19194332
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Hysong SJ,
      2. Best RG,
      3. Pugh JA
      . Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci 2006;1:1–10.doi:10.1186/1748-5908-1-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/16722539
      OpenUrlCrossRefPubMed
    12. ↵
      1. Larson EL,
      2. Patel SJ,
      3. Evans D, et al
      . Feedback as a strategy to change behaviour: the devil is in the details. J Eval Clin Pract 2013;19:230–4.doi:10.1111/j.1365-2753.2011.01801.xpmid:http://www.ncbi.nlm.nih.gov/pubmed/22128773
      OpenUrlCrossRefPubMed
    13. ↵
      1. Payne VL,
      2. Hysong SJ
      . Model depicting aspects of audit and feedback that impact physicians’ acceptance of clinical performance feedback. BMC Health Serv Res 2016;16:1–12.doi:10.1186/s12913-016-1486-3
      OpenUrlCrossRefPubMed
    14. ↵
      1. Gude WT,
      2. Brown B,
      3. van der Veer SN, et al
      . Clinical performance comparators in audit and feedback: a review of theory and evidence. Implementation Science 2019;14:1–14.doi:10.1186/s13012-019-0887-1
      OpenUrl
    15. ↵
      1. Mafi JN,
      2. Parchman M
      . Low-Value care: an intractable global problem with no quick fix. BMJ Qual Saf 2018;27:333–6.doi:10.1136/bmjqs-2017-007477pmid:http://www.ncbi.nlm.nih.gov/pubmed/29331955
      OpenUrlFREE Full Text
    16. ↵
      1. Sypes EE,
      2. de Grood C,
      3. Clement FM, et al
      . Understanding the public’s role in reducing low-value care: a scoping review. Implementation Science 2020;15:1–13.doi:10.1186/s13012-020-00986-0
      OpenUrl
    17. ↵
      1. Müskens JL,
      2. van Dulmen SA,
      3. Wiersma T, et al
      . Low-Value pharmaceutical care among Dutch GPs: a retrospective cohort study. Br J Gen Pract 2022;72:e369–77. Feb 10;BJGP.2021.0625.doi:10.3399/BJGP.2021.0625pmid:http://www.ncbi.nlm.nih.gov/pubmed/35314429
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Kool RB,
      2. Verkerk EW,
      3. Meijs J, et al
      . Assessing volume and variation of low-value care practices in the Netherlands. Eur J Public Health 2020;30:236–40.doi:10.1093/eurpub/ckz245
      OpenUrlPubMed
    19. ↵
      1. Pendrith C,
      2. Bhatia M,
      3. Ivers NM, et al
      . Frequency of and variation in low-value care in primary care: a retrospective cohort study. CMAJ Open 2017;5:E45–51.doi:10.9778/cmajo.20160095pmid:http://www.ncbi.nlm.nih.gov/pubmed/28401118
      OpenUrlAbstract/FREE Full Text
    20. ↵
      Kerry Influence of postal distribution of The Royal College of Radiologists’ guidelines, together with feedback on radiological referral rates, on X-ray referrals from general practice- a randomized controlled trial.
    21. ↵
      1. Gerber JS,
      2. Prasad PA,
      3. Fiks AG, et al
      . Effect of an Outpatient Antimicrobial Stewardship Intervention on Broad-Spectrum Antibiotic Prescribing by Primary Care Pediatricians A Randomized Trial [Internet]. Available: www.jama.com
    22. ↵
      1. WHJM V,
      2. van der Weijden T,
      3. Sijbrandij J, et al
      . Effect of a Practice-Based Strategy on Test Ordering Performance of Primary Care Physicians A Randomized Trial [Internet]. Available: http://jama.jamanetwork.com/
    23. ↵
      1. Tong A,
      2. Sainsbury P,
      3. Craig J
      . Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19:349–57.doi:10.1093/intqhc/mzm042pmid:http://www.ncbi.nlm.nih.gov/pubmed/17872937
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. van Someren MaartenW
      . The think aloud method. Vol. 31, Information Processing & Management 1994.
    25. ↵
      1. Lewis C
      . Using the “thinking-aloud” method in cognitive interface design. Yorktown Heights: IBM TJ Watson Research Center, 1982.
    26. ↵
      1. de Lusignan S,
      2. Liyanage H,
      3. Sherlock J, et al
      . Atrial fibrillation dashboard evaluation using the think aloud protocol. BMJ Health Care Inform 2020;27:e100191–7.doi:10.1136/bmjhci-2020-100191pmid:http://www.ncbi.nlm.nih.gov/pubmed/33087337
      OpenUrlPubMed
    27. ↵
      1. Sullivan GM,
      2. Sargeant J
      . Qualities of qualitative research: Part I. J Grad Med Educ 2011;3:449–52.doi:10.4300/JGME-D-11-00221.1pmid:http://www.ncbi.nlm.nih.gov/pubmed/23205188
      OpenUrlCrossRefPubMed
    28. ↵
      1. Colquhoun HL,
      2. Sattler D,
      3. Chan C, et al
      . Applying User-Centered design to develop an audit and feedback intervention for the home care sector. Home Health Care Manag Pract 2017;29:148–60.doi:10.1177/1084822317700883
      OpenUrl
    29. ↵
      1. Landis-Lewis Z,
      2. Kononowech J,
      3. Scott WJ, et al
      . Designing clinical practice feedback reports: three steps illustrated in veterans health affairs long-term care facilities and programs. Implementation Science 2020;15:1–13.doi:10.1186/s13012-019-0950-y
      OpenUrl
    30. ↵
      1. Colquhoun H,
      2. Michie S,
      3. Sales A
      . Reporting and design elements of audit and feedback interventions: a secondary review. vol. 26, BMJ quality and safety. BMJ Publishing Group 2017:54–60.
    31. ↵
      1. Foy R,
      2. Skrypak M,
      3. Alderson S, et al
      . Revitalising audit and feedback to improve patient care. BMJ 2020;368:m213.doi:10.1136/bmj.m213pmid:http://www.ncbi.nlm.nih.gov/pubmed/32107249
      OpenUrlFREE Full Text
    32. ↵
      1. Colquhoun HL,
      2. Carroll K,
      3. Eva KW, et al
      . Informing the research agenda for optimizing audit and feedback interventions: results of a prioritization exercise. BMC Med Res Methodol 2021;21:20.doi:10.1186/s12874-020-01195-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/33435873
      OpenUrlPubMed
    33. ↵
      1. van der Veer SN,
      2. de Keizer NF,
      3. Ravelli ACJ, et al
      . Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform 2010;79:305–23.doi:10.1016/j.ijmedinf.2010.01.011pmid:http://www.ncbi.nlm.nih.gov/pubmed/20189451
      OpenUrlCrossRefPubMed
    34. ↵
      1. Cooke LJ,
      2. Duncan D,
      3. Rivera L, et al
      . How do physicians behave when they participate in audit and feedback activities in a group with their Peers? Implementation Science 2018;13.doi:10.1186/s13012-018-0796-8
    35. ↵
      1. Desveaux L,
      2. Ivers NM,
      3. Devotta K, et al
      . Unpacking the intention to action gap: a qualitative study understanding how physicians engage with audit and feedback. Implementation Science 2021;16:1–9.doi:10.1186/s13012-021-01088-1
      OpenUrl
    36. ↵
      1. Verstappen WHJM,
      2. ter Riet G,
      3. Dubois WI, et al
      . Variation in test ordering behaviour of GPs: professional or context-related factors? Fam Pract 2004;21:387–95.doi:10.1093/fampra/cmh408pmid:http://www.ncbi.nlm.nih.gov/pubmed/15249527
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Trietsch J,
      2. van Steenkiste B,
      3. Grol R, et al
      . Effect of audit and feedback with peer review on general practitioners’ prescribing and test ordering performance: a cluster-randomized controlled trial. BMC Fam Pract 2017;18:1–13.doi:10.1186/s12875-017-0605-5
      OpenUrlCrossRefPubMed
    38. ↵
      1. Rohrbasser A,
      2. Harris J,
      3. Mickan S, et al
      . Quality circles for quality improvement in primary health care: Their effectiveness, gaps of knowledge, origins and significance - a scoping review. bioRxiv.2018:1–19.
    39. ↵
      1. Ivers N,
      2. Jamtvedt G,
      3. Flottorp S, et al
      . Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;2012:CD000259.doi:10.1002/14651858.CD000259.pub3pmid:http://www.ncbi.nlm.nih.gov/pubmed/22696318
      OpenUrlPubMed
    40. ↵
      1. van Braak M,
      2. Visser M,
      3. Holtrop M, et al
      . What motivates general practitioners to change practice behaviour? A qualitative study of audit and feedback group sessions in Dutch general practice. BMJ Open 2019;9:e025286–8.doi:10.1136/bmjopen-2018-025286
      OpenUrlAbstract/FREE Full Text
    41. ↵
      1. Rouleau G,
      2. Reis C,
      3. Ivers NM, et al
      . Lipstick on a pig : Understanding efforts to redesign audit and feedback reports for primary care 2022:1–16.
    42. ↵
      1. Ivers N,
      2. Barnsley J,
      3. Upshur R, et al
      . "My approach to this job is one person at a time": Perceived discordance between population-level quality targets and patient-centred care. Can Fam Physician 2014;60:258–66.pmid:http://www.ncbi.nlm.nih.gov/pubmed/24627384
      OpenUrlAbstract/FREE Full Text
    43. ↵
      1. Arvidsson E,
      2. Dahlin S,
      3. Anell A
      . Conditions and barriers for quality improvement work: a qualitative study of how professionals and health centre managers experience audit and feedback practices in Swedish primary care. BMC Fam Pract 2021;22:1–13.doi:10.1186/s12875-021-01462-4
      OpenUrlCrossRef
    44. ↵
      1. Ivers NM,
      2. Desveaux L
      . De-implementation of low-value care: use audit and feedback wisely. Healthcare Papers 2019;18:41–7.
      OpenUrl

    Supplementary materials

    Footnotes

    • Contributors All authors contributed substantially to the conception, design or execution of the reported study. EvdH, PdB, MD and AvdK took initiative for the study and designed the study. ABvdK collected the data. MD did preparatory work for the interviews and delivered feedback on the qualitative research methods that were used. PdB, AvdK, MD, VdW and EvdH participated in data analysis and interpretation of the findings. PdB and VdW were responsible for the write-up of the study and act as guarantor. All authors critically revised and approved the final manuscript.

    • Funding This research was partly carried out within the Academic Working Place Care Practice and Policy of the Consortium Quality of Care of the Netherlands Federation of University Medical Centers (NFU) and National Health Care Institute.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

    • ↵In Dutch, this report is called Praktijkspiegel

    • ↵Personal data 2019 from VEKTIS, the Dutch research Centre responsible for the development and distribution of the PCPR

    • ↵In Dutch: Zorgverzekeraars Nederland