Article Text
PDF
Abstract
Objectives Ensuring that healthcare is patient-centred, safe and harm free is the cornerstone of the NHS. The Scottish Patient Safety Programme (SPSP) is a national initiative to support the provision of safe, high-quality care. SPSP promotes a coordinated approach to quality improvement (QI) in primary care by providing evidence-based methods, such as the Institute for Healthcare Improvement’s Breakthrough Series Collaborative methodology. These methods are relatively untested within dentistry. The aim of this study was to evaluate the impact to inform the development and implementation of improvement collaboratives as a means for QI in primary care dentistry.
Design A multimethod study underpinned by the Theoretical Domains Framework and the Kirkpatrick model. Quantitative data were collected using baseline and follow-up questionnaires, designed to explore beliefs and behaviours towards improving quality in practice. Qualitative data were gathered using interviews with dental team members and practice-based case studies.
Results One hundred and eleven dental team members completed the baseline questionnaire. Follow-up questionnaires were returned by 79 team members. Twelve practices, including two case studies, participated in evaluation interviews. Findings identified positive beliefs and increased knowledge and skills towards QI, as well as increased confidence about using QI methodologies in practice. Barriers included time, poor patient and team engagement, communication and leadership. Facilitators included team working, clear roles, strong leadership, training, peer support and visible benefits. Participants’ knowledge and skills were identified as an area for improvement.
Conclusions Findings demonstrate increased knowledge, skills and confidence in relation to QI methodology and highlight areas for improvement. This is an example of partnership working between the Scottish Government and NHSScotland towards a shared ambition to provide safe care to every patient. More work is required to evaluate the sustainability and transferability of improvement collaboratives as a means for QI in dentistry and wider primary care.
- quality improvement
- quality improvement methodologies
- health services research
- patient safety
- patient care bundles
Data availability statement
Data are available on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- quality improvement
- quality improvement methodologies
- health services research
- patient safety
- patient care bundles
Introduction
Ensuring that health service provision is patient-centred, safe and harm free is one of the cornerstones of the National Health Service (NHS). Despite this, healthcare is identified as a major source of avoidable harm.1 2 A number of influential reports have highlighted significant concerns about safety in healthcare,3 4 including a House of Commons Select Committee report which stated, ‘The extent of medical harm is substantial, even on a conservative estimate, and much is avoidable’5; and a 2011 review of evidence estimated that 1%–2% of primary care consultations may involve an ‘error’ which could result in patient harm.6 However, apart from a minority of high profile cases, patient harm is rarely linked to deliberate wrong-doing or incompetence,5 rather it is considered to be the result of systems-level failures.3 A wide range of safety-related activities have been introduced by UK policy-makers to address this; however, despite primary care accounting for around 90% of patient contact, these have mainly been directed towards acute care settings.7
The Scottish Patient Safety Programme (SPSP) was launched in 2008 with a 5-year aim to reduce harm and improve the safety and reliability of health and social care.8 The primary aim was to reduce mortality by 15% and adverse events by 30% across Scotland’s acute hospitals by the end of 2012.9 A second phase of the programme was launched in 2015, delivered through Healthcare Improvement Scotland’s (HIS) Improvement Hub (ihub). It took a coordinated approach to increase awareness and support of the provision of safe, high-quality care, whatever the setting, and for the first time included safety improvement programmes within primary care. A key element of the programme was that staff involved with direct patient care led the changes and monitored improvement through the collection of real-time data at an individual level. To facilitate this, a range of evidence-based quality improvement (QI) methods were developed to support practitioners assess and improve the care they provided.10
The first SPSP collaborative involving primary care dentistry was developed and undertaken between 2015 and 2017, led by HIS.10 All territorial health boards (HBs) were invited to participate. Three were recruited: NHS Ayrshire and Arran; NHS Dumfries and Galloway and NHS Fife. During this first phase participating practices from the three NHS HBs developed and implemented a medical histories care bundle tailored to their individual practice. This phase emulated the Institute for Healthcare Improvement’s (IHI) Breakthrough Series Collaborative methodology. This methodology is a relatively short-term learning system, designed to help organisations learn from each other and bring together teams seeking improvement in a focused topic area.11 This team approach works on the basis that a small number from each team attend organised learning sessions, with additional team members working on improvements in the local organisation. This was the first time such QI methods had been used within primary care dentistry.
In 2017/18, a second phase of the programme was carried out. Two NHS Education for Scotland (NES) Dental Vocational Training (VT) schemes participated, alongside the same three HBs. Two care bundles were implemented during this phase: a high-risk medication care bundle and a prevention of periodontal disease care bundle. The high-risk medication care bundle was an evolution of the original medical histories care bundle used in phase I. Both care bundles were developed by consensus with practitioners and topic experts and were based on recommendations made in Scottish Dental Clinical Effectiveness Programme (SDCEP) clinical guidance.
Improvement collaboratives are a relatively untested method of improving quality in general dental practices and little is known about their impact on individual staff, practice teams or the quality of care delivery. Therefore, during the second phase of the Primary Care Dentistry Collaborative (2017/18), an evaluation of its impact was carried out. The aim of this evaluative study was to inform the future development and implementation of improvement collaboratives as a means for QI in primary care dentistry. This paper presents the individual and team learning, beliefs and behaviours towards QI in general dental practice; the barriers and facilitators to implementing QI methods in general dental practice and potential refinements to the SPSP in Primary Care Dentistry Collaborative approach.
Methods
Study design
This evaluation focuses on the second phase of the Primary Care Dentistry Collaborative only. This second phase broadly followed the Breakthrough Series Collaborative methodology within the constraints of the busy VT curriculum. Two care bundles were implemented during this phase of testing: (1) a periodontal care bundle, focusing on the prevention of periodontal disease and (2) a high-risk medication medical histories bundle. During the collaborative, dental teams were asked to implement one of the care bundles and complete a safety climate survey within their practice team. Each participating NHS Board was provided with resources to appoint a clinical lead and a facilitator to support the collaborative. Representatives from each dental team were invited to attend an induction event, as well as two national learning events over the course of the collaborative. In turn, these team members were asked to share learning from these events with the rest of their team. For the VT practices, regular study days were identified for the introduction of QI education and training and to monitor progress. Practice data were submitted monthly and the HIS clinical lead convened a feedback session at the end of the project.
The evaluation comprised a multimethod approach underpinned by the Theoretical Domains Framework (TDF) and the Kirkpatrick model.12 The TDF is derived from a number of behavioural theories and constructs and proposes that determinants of behaviour can be clustered into 14 domains. The TDF has been used extensively to explore and explain variation in clinical practice.13–15 The Kirkpatrick model is a framework that can be used to objectively analyse the effectiveness and impact of training.
Quantitative data were gathered using a baseline and follow-up questionnaire-based survey to team members within participating practices. Qualitative data were gathered using telephone and face-to-face interviews with participating dental team members and HB representatives, as well as practice-based case studies.
Setting and participants
The evaluation was set in NHS General Dental Practices in Scotland. All NHS HBs were invited to participate in the first phase of the collaborative led by HIS. The aim was to recruit three HBs, with each participating HB receiving funding of around £20 000 towards the cost of participation (eg, appointment of a dental clinical lead and dental practice teams). Fifteen dental practices were recruited from the three recruited HBs (NHS Ayrshire and Arran; NHS Dumfries and Galloway and NHS Fife). In addition to these 15 HB practices from phase I, for phase II, 24 practices were identified to participate through the NES VT scheme. VT practices were selected by the NES Vocational Dental Practice Director.
Data collection
Questionnaire data
The vocational dental practitioner (VDP) and the trainer from all participating VT practices attended a learning session before implementing their allocated care bundle. At the beginning of the session, information was provided about the evaluation, and VDPs and trainers were invited to complete a baseline questionnaire. The questionnaires were designed to explore beliefs and behaviours towards improving quality in general dental practice, self-reported knowledge and skills towards QI concepts and methodologies as well as current practice in relation to the two care bundles. The TDF was used to guide the choice of questionnaire items. They were also provided with additional questionnaires to distribute to members of other staff groups in their dental practice (eg, dental care professionals (DCPs), administrative staff). Freepost envelopes were provided for return to the evaluation team.
Dental team members from practices that had participated in phase I of the Primary Care Dentistry Collaborative, who had previously attended a learning session, were not required to attend this collaborative learning session. In these practices, information about the evaluation, the baseline questionnaire and freepost envelopes were posted to the member of the practice team recorded as the practice contact. The practice contact was asked to distribute these materials to at least one individual from each staff group (dentist, DCP, administrative staff) in their practice for completion and return.
Follow-up questionnaires were posted to all practices approximately 9 months later. A range of dental team members were again asked to complete the questionnaire and return it to the evaluation team.
Interviews and case study data
Interview participants were chosen using a purposive sampling frame which included a range of professional roles and practice types. Case study practices were selected to include one which was fully compliant with their existing care bundle and one which was not, based on the self-reported questionnaire data.
A pre-notification email and information sheet providing details of interview timescales were sent to participating practices. Following this, participants were contacted by telephone to arrange a convenient date and time for interview. HB practices were asked to nominate one dentist and at least one other member of the dental team to participate in the interviews. In VT practices, the trainer, VDP and at least one other member of the dental team were invited to participate. Participation in case studies was discussed with selected practices by a member of the evaluation team. Interviews with team members from case study practices took place face-to-face as part of the case study visit. In addition, representatives from participating HBs as well as NES VT schemes were contacted by email and invited to be interviewed.
Interview data were collected using semi-structured telephone or face-to-face interviews using open-ended questions and probing. Interview topic guides were developed using the TDF as a broad framework as well as by data from the baseline questionnaire and piloted before use. The topic guide for HB representatives was also partially informed by initial analysis of the dental team member interview data. Interviews were conducted by an experienced qualitative researcher (HC) and a dental core trainee (VM) and digitally recorded with participant consent. A case study observation schedule was developed, and handwritten notes were taken during each practice visit.
Data collection ceased when data saturation was achieved (ie, no new information or insights were gained).
Data handling and analysis
Questionnaire data
Descriptive statistics were used to explore the data, check data distribution and ensure there were no disparities in data entry. Thematic analysis was used to analyse free-text responses.
Practices were asked to ensure that the same staff completed the questionnaire at baseline and follow-up, however, it was not possible to reliably confirm that this was the case for all participants. Consequently, only descriptive comparisons are presented for the full sample of participants between baseline and follow-up, and statistical comparisons are presented for the subset of participants where the responses could be linked by a unique identifier. Results from both the full sample and the linked sample are presented in this paper.
The majority of data were found not to be normally distributed, and non-parametric tests were used to compare baseline and follow-up responses. To account for multiple comparisons, the a priori criterion for significance was set conservatively at p<0.01.
The χ2 goodness-of-fit tests were conducted to explore any differences in the demographic characteristics of the baseline and follow-up participants. For sections 1 and 2 of the questionnaires, which explored participants’ views around improving quality in primary care dentistry and their knowledge about QI methods and concepts, Wilcoxon signed-rank tests were used to compare baseline and follow-up responses. Section 3 explored current practice in relation to the two care bundles and is presented at a practice level. To measure compliance with the two care bundles, a practice score was calculated by taking the highest score for each question from all participants within each practice. The highest score was then re-coded with the responses never, rarely, sometimes or usually being grouped and classified as not always resulting in two mutually exclusive response categories: always or not always. The data were analysed using McNemar’s test to compare the difference between baseline and follow-up practice-level compliance.
Interviews and case study data
The framework approach to qualitative data management was adopted.16 This is a matrix-based method, using a thematic framework to organise and classify data according to key issues, concepts and emerging themes.17 All recordings were professionally transcribed, anonymised and accuracy checked. Data management was facilitated using QSR International’s NVivo V.11 software. The study findings are explored and synthesised using the Kirkpatrick model as a lens through which to interpret the data. The Consolidated Criteria for Reporting Qualitative Research and the Standards for Reporting Qualitative Research Checklist were employed to guide reporting of the data.
Consent, ethical review and governance
NHS ethical approvals were not required as all participants were recruited by virtue of their professional role.18 All questionnaire data, recordings and interview transcripts and notes were anonymised and stored confidentially and securely in accordance with NES Information Governance procedures and the Data Protection Act, 1998 and latterly the General Data Protection Regulation, 2018.
Patient and public involvement
Patients and the public were not involved in this study.
Results
Sample characteristics
All 15 HBs and 16 VT practices participating in the collaborative participated in the evaluation. One hundred and eleven dental team members, from across all practices completed the questionnaire at baseline. Of those, 60% were dentists and 66% were female, working on average 7.7 sessions per week. At follow-up, questionnaires were returned by 79 team members, from 20 practices, resulting in a practice level response of 69% (20/29). Two practices dropped out of the evaluation after completion of the baseline questionnaire. Participants’ characteristics at follow-up and baseline were broadly similar, with χ2 tests revealing no significant differences. From the sample of participants completing the questionnaire at baseline and follow-up it was possible to link a subset of 32 participants, from 17 practices, using their individual participant ID. The χ2 goodness-of-fit tests indicated no significant differences in the demographic characteristics of the baseline participants and the linked sample.
Dental team members from 12 of the participating dental practices took part in the interviews. A recruitment sampling frame was developed to aid the inclusion of a range of professional roles and practice types. In addition, a pragmatic approach based on ‘willingness to participate’ was employed. In total, 34 dental team member interviews were conducted, this included 11 interviews from two practice case studies. An additional seven interviews were conducted with representatives from HBs and NES, resulting in a total of 41 interviews. The two case study practices comprised one lower compliance and one higher compliance practice based on the self-reported questionnaire data. Interviews took place between February and June 2018 and ranged from 20 to 60 minutes in duration.
Questionnaire findings
Participants’ beliefs about improving quality in primary dental care at both baseline and follow-up were generally positive with descriptive comparisons across the full sample showing that most beliefs were more positive at follow-up than at baseline. Comparison of the responses from the subset of participants who were identified as having completed the questionnaire at both baseline and follow-up identified six belief statements that were significantly more positive at follow-up than at baseline. These were: “I know how to improve quality in dental practice” (p<0.01, z=3.58); “I have been trained in how to improve quality in dental practice” (p<0.01, z=3.99); “I have the skills I need to improve quality in my practice” (p<0.01, z=3.64); “I am confident that I can improve quality in my practice” (p<0.01, z=2.83); “I have a clear plan about how I will improve quality in my practice” (p<0.01, z=3.10) and “Within my dental practice, I am able to access records/information to help me improve quality” (p<0.01, z=2.79).
Across the full sample, participants also reported an increase in their knowledge and skills across a range of QI concepts and methodologies for all included statements. Comparison of responses from the linked subsample identified statistically significant, self-reported increases in understanding or skills for all statements. Despite these increases however, knowledge and skills towards QI concepts and methodologies were generally low. In the full sample, there was a relatively high proportion of participants reporting at follow-up that they did not understand the Breakthrough Series Collaborative approach to QI (41%), driver diagrams (42%), the PDSA process (38%), care bundles (36%) and run charts (44%). Likewise, a relatively high proportion of participants reported they did not have the skills to develop driver diagrams (27%), use driver diagrams (46%), apply the PDSA process (43%), develop care bundles (41%), use care bundles (42%), develop run charts (49%) and use run charts (48%).
The questionnaire also explored current practice in relation to the two care bundles. Baseline and follow-up practice scores were determined for all participating practices to identify any significant differences at a practice level between the beginning and end of their participation in the Primary Care Dentistry Collaborative. No change was identified regarding carrying out a basic periodontal examination at dental check-ups as all practices reported at baseline and follow-up that they always did this. All other items in both bundles showed an increase in the percentage of practices ‘always’ carrying out the behaviour between baseline and follow-up, however, no items demonstrated a statistically significant increase.
Interview and case study findings
Knowledge and beliefs
Most participants reported having limited previous knowledge and experience of QI and this was particularly evident among non-clinical team members. Clinical team members reported that any prior knowledge or experience of QI they had was through audit. They reflected that prior to the study they would not have felt confident implementing QI in their practice nor been able to say confidently that any changes made as part of an audit had been sustained. Some dental nurses reported knowing little or nothing about the collaborative project itself and commented that they felt ‘out of their depth’ taking part in the interviews.
I didn’t think I really had much of an important role…I just basically started at the bottom so really didn’t have any knowledge at all. Dental Nurse_30 [periodontal care bundle]
Having participated in the study most clinical team members reported having increased confidence and greater understanding of the systems that can be used to implement QI. Non-clinical team members reported less of an increase in knowledge and confidence, but this varied by practice and seemed dependent on the communication systems and culture that existed within their practice.
I would be more confident if I wanted to implement a change in the practice, how to go about that and how to gauge whether or not that was having an effect, and actually doing the project in a different way. Dentist_19 [periodontal care bundle]
I feel like I’ve learnt so much that I would be able to use it on other projects and, um, to bring to the team to try and motivate them. Dentist_20 [periodontal care bundle]
Barriers to using QI in practice
The main barriers to using the QI methods identified by interviewees centred around time, lack of patient interest/engagement, poor team engagement, communication and leadership.
Time was the most commonly cited barrier with almost all interviewees referencing it. This related to the requirement for longer appointments, time to carry out data collection, time to record information or time to identify relevant patients. Time was also identified as a barrier to having team meetings and attending training. For VDPs, time was less of an issue as they tended to have longer appointment times and allocated tutorial time.
It’s trying to find the time to do it [practice meetings] because right now we’re gonna have to do it on a Saturday morning because the owner doesn’t like to close the practice. Dental Nurse_1 [periodontal care bundle]
Time is the biggest thing in practice you know, it’s great doing quality improvement but do you have the time to implement all these things. Dentist_4 [high-risk medication care bundle]
…as a VDP I had that time available, which was good. As an associate I think it would be more difficult to be able to find that time. Dentist_27 [high-risk medication care bundle]
One of the biggest challenges I’m getting feedback on is that it is taking an awful lot more time than I arranged for it to be remunerated for. HB Representative_1
Whether patients were willing to engage with the initiative was reported as impacting on the team’s ability to implement the care bundles. In some cases, more specifically in relation to the prevention of periodontal disease care bundle, it was reported that patients were not always interested in receiving advice. Despite this, some interviewees did report instances where patients were grateful for the advice and the additional time the dental team was spending with them. In relation to the high-risk medication care bundle, patients’ awareness about their own medication and medical history were barriers to implementation.
Sometimes you’re talking to patients about things like this and they’re just not that interested in it to be honest with you. Dentist_24 [high-risk medication care bundle]
Not getting all the information that’s required from patients, sometimes they can intentionally or unintentionally not give you all the information. Receptionist_26 [high-risk medication care bundle]
Another key barrier to the use of QI methods in practice was around team engagement and motivation. Some of those taking the lead on a project in their practice reported difficulties keeping the whole team motivated and arranging opportunities to provide regular whole team communication and updates. Where VDPs were leading the projects, some reported that being new to the practice was an added challenge to engagement with existing team members and attempting to change their behaviour was difficult.
I think that, perhaps the staff who have been here a wee bit longer, it was more difficult for them to sort of change their habits. Dentist_24 [high-risk medication care bundle]
They were all probably initially quite involved, but as things went on, you know, the dentists here were… it’s quite difficult for us to keep them motivated. Dentist_9 [periodontal care bundle]
Intrinsically linked to team engagement was leadership. Leadership was most evident as a barrier where the practice owner was not leading the project or not actively involved. However, leadership was considered more positively when a VDP was taking the lead. As previously reported, VDPs are allocated longer appointments and time to carry out data collection. Some positive examples of shared leadership were also identified where dentists, VDPs and dental nurses demonstrated working together with their own clear roles and responsibilities.
The main barrier really is that I am not the practice owner. So, it can sometimes be a little bit tricky arranging those meetings. To then suggest changes, and we can suggest changes all we like but we also need to have the practice principal on board. Dentist_11 [periodontal care bundle]
I think the way that we work together as a team; me with my, the clinical, in the surgery and [Dental Nurse] doing the admin and reception desk. Dentist_2 [periodontal care bundle]
I think a lot of the time as a practice owner you feel that sometimes all these extra things should come down to you doing them and I’ve deliberately not done that, you know, I’ve been quite keen to take part but not be at all the lead. Dentist_21 [periodontal care bundle]
Communication varied by practice with some reporting regular team meetings, meetings arranged specifically to discuss the project, presenting information on staff notice boards as well as ad hoc discussions. In terms of cascading project information within the team, those actively involved were generally clear about the purpose, however other team members seemed less aware.
I still think some of the staff don’t really know what it is we’re doing. Dentist _12 [periodontal care bundle]
To be honest, I don’t think, many other people in the practice know about it. Dental Nurse_17 [high-risk medication care bundle]
The care bundle topic was also highlighted as a barrier. The prevention of periodontal disease care bundle was considered straightforward to implement and easy to identify appropriate patients. However, some participants felt they were already carrying out those behaviours to an appropriate standard and therefore saw less benefit. Identifying patients taking high-risk medications was considered challenging and many did not manage to recruit the required number of patients. The clarity and interpretation of the bundle questions were also highlighted as hindering implementation.
I think if it was something that we were doing quite well already, stuff like the perio one … we generally do that as standard, so I think it would’ve been harder to convince ourselves that this is something that’s as important. Dentist_28 [high-risk medication care bundle]
For example, the first question—does that patient have an updated medical history? That’s … you know, how recently is updated, you know, it needs to qualify exactly what they mean by updated. Dentist_23 [high-risk medication care bundle]
Facilitators to using QI in practice
The main facilitators identified during interviews were good team working, clear roles and responsibilities, strong leadership as well as training, peer support and the ability to see the benefit of using QI in practice. The care bundle topic and the wording of the bundle questions also emerged as having a direct impact on team engagement and hence helped or hindered implementation. Communication emerged strongly as both a barrier and facilitator to using QI methods in practice.
The factor highlighted as being most helpful in enabling communication was the existence of ‘WhatsApp’ chat groups, both within practices and with other practices. In addition, the project provided an opportunity for the VDP to speak to the whole practice team during team meetings, providing them with a clear role within the team.
If there was anything we didn’t know we would take it to the WhatsApp group and then they were all really good at replying and giving us help… Dentist_5 [high-risk medication care bundle]
From [VDPs] point of view, it’s given [them] the opportunity for instance to have a role in the practice meetings, whereas maybe before she wouldn’t have had such a lead role. Dentist_19 [periodontal care bundle]
Overall, the training provided to the dental teams was considered beneficial, providing a forum for discussion, an opportunity to report progress and discuss individual challenges. There were mixed views about how helpful the online modules were.
I’m old fashioned that I like face to face training, I don’t really trust myself with the online seminar things because I tend to not pay as much attention… Dentist_28 [high-risk medication care bundle]
Understanding the potential benefits and impact of the changes they were making also emerged as a facilitator to implementation. This was in terms of benefits directly impacting on themselves, the practice and to patient care. It was agreed that, just by being involved, some team members who were not traditionally involved in decision-making and practice development felt more motivated about QI.
I’ve got a far better understanding in the process now; I’m taking part again in this coming year for different reasons that is, because I’m far more positive about the process and I understand the benefits to the practice and the patients. HB Representative_7
Support to participate in the project came in various formats. The support and advice provided by team members within HIS, being part of a VT scheme, having the support of peers, the ability to have discussions at study days and resources such as guidance documents were all identified as crucial to successful implementation.
I’d say peer-to-peer sort of stuff was probably the most helpful because you’re much more likely to chat about it amongst people who were your friends and going through it at the same time as you. Dentist_18 [high-risk medication care bundle]
I think the local support groups are good, having been part of [health board] team. Because it’s small enough that you know everybody quite well and it’s big enough to get enough people. Dentist_21 [periodontal care bundle]
Good team working, having clear roles and responsibilities and involving the whole team strongly emerged as facilitators. In relation to the high-risk medication care bundle, there was an appreciation that the whole dental team could be involved, starting from the receptionist making the appointment and greeting the patient, through to the clinicians treating them in the surgery. It was also noted that this was a new area for most participants, where they could see a need and importance to changing their behaviour, ultimately motivating and engaging them in the process and having a motivational impact within practices.
One of our practices, it was the receptionist that gathered the data so that was great for her because she now understands about data collection. HB Representative_2
It was really nice for some of the nurses within the team, the practice manager, they were taking the lead on this rather than it being the dentist. HB Representative_4
So [health board] loved being involved in the medical history one, and I think that’s because they felt real ownership right from the start; developing it, testing it, piloting it. And then they found that one to be really team inclusive and everyone had a role and could see the benefits. HB Representative_3
Data synthesis
Given the breadth and depth of data collected the Kirkpatrick model was used to synthesise the data. The Kirkpatrick model is a recognised framework used for the evaluation of training and learning and is based on the concept that there are four key factors that determine the effectiveness of an educational intervention: (1) reaction; (2) learning; (3) behaviour and (4) results. In order to draw together all of the findings from the questionnaires, interviews with dental team members and key stakeholders and the case studies, the data were explored using these four factors as a lens through which to examine this evaluative study. Each of the four factors are examined within the context of this study and are presented below.
Reaction
Reaction relates to how participants feel about the learning or training experience they received. Interviewees were generally positive about the training provided, however suggestions on how the training sessions could be improved for the future were provided. One example was to tailor the QI training more to a clinical audience and provide information about the ‘nuts and bolts’ of what was required for the actual project before focusing on QI theory. While training on how to implement the care bundles had been provided, the questionnaire results highlighted that there was still room for improvement regarding the development of care bundles as well as the development and use of QI concepts such as driver diagrams and run charts. Findings suggest there may be a requirement for team members to understand and gather experience on how to develop these tools themselves and that further support may be required. Suggestions for training formats included involving the whole team, using the ‘in practice training’ model where possible as well as having the online modules available for those who prefer them.
I think if we were in that situation where they were like, ‘Right, you can go and do your own now’ I think we’d need to know, we’d need to get some training in setting up a care bundle and stuff like that… We’ve literally answered the questions that somebody else sets up as a project. Dentist_21 [periodontal care bundle]
I am a big fan of in-practice training because then you and your team are working together.
Dentist _9 [periodontal care bundle]
Learning
Learning refers to measurement of any increase in knowledge. For the purpose of this evaluation this related to whether participants felt their knowledge and understanding of QI concepts and methods had increased as a result of participating and if they believed they had the skills and confidence to implement future QI initiatives in general dental practice.
Increased knowledge around QI methods was evident from the interview and questionnaire data however confidence varied, with some feeling very confident about carrying out further QI projects and others reporting that they would still require additional support. A number of those who had participated in the first phase of the collaborative commented on feeling more confident implementing their second care bundle, suggesting evidence of increased knowledge through experience. It was also highlighted that having a VDP in the practice facilitated the QI process by keeping the rest of the dental team up to date as well as by having someone in post who could take on the ‘driver for change’ role.
Yes, I would feel a lot better, yeah, now that we’ve done one. We are more aware of what to do, and what you are looking for, and what the main result is that you want to be. Dentist_14 [high-risk medication care bundle]
I feel that time wise that would be a barrier, skill wise that would be a barrier. So, it’s, you know, creating charts and all of these would be a barrier. We’re not skilled enough at all to do that. Dentist_21 [periodontal care bundle]
Behaviour
Within the Kirkpatrick model, behaviour is defined as ‘the extent of applied learning back on the job’, in respect of this evaluation it related to dissemination of learning within the team, changes in practice, sustainability of QI methods within practice and changes observed to the practice culture. In terms of shared learning and dissemination this varied by practice. However, many examples of changes made in practices were described both during the dental team interviews and by the other stakeholders such as HB dental practice advisers who had witnessed these changes themselves. Examples included producing patient information leaflets and posters, introducing new patient management systems, raising awareness around high-risk medicines during team meetings and standardisation of note taking and recording.
Note taking, much better. it’s more standardised now. Everyone previously was writing it in a slightly different way, but as a result of this project we are all pretty much on the same page now. Dentist_11 [periodontal care bundle]
I think the medical histories are getting taken a lot more, they are getting checked at every single appointment… Dental Nurse_17 [high-risk medication care bundle]
Impact on culture varied with some practices reporting subtle changes and others reporting no change at all. In many cases, interviewees felt it was too early to comment on changes in culture. Despite this, interview data suggested that there were some shifts in culture although these shifts tended to be in practices which appeared to be relatively motivated and have good communication systems in place prior to starting the project. In practices where staff morale appeared to have been low and communication systems appeared poorer, participation in the collaborative seemed to have reinforced and highlighted these challenges. It could be argued however that this may ultimately motivate a cultural change.
I think it challenges the culture of a practice, and I would like to think that it has encouraged greater dialogue within the practice because there was this sense that you wanted to involve the practice team at a greater level and allow all members of the team to input their ideas. HB Representative_7
We did the little quiz at the start that everyone was kind of working in teams…it brought up the staff engagement and things, so they were quite interested in the start. And it’s nice that people like the receptionists that don’t necessarily have a clinical role but do have an extra sort of element in patient care rather than just sort of appointments and payments and that kinda thing. So, I think they kind of engaged quite a bit that they now knew, ‘Oh, this is why that patient can’t do that’. Dentist_27 [high-risk medication care bundle]
Results
Results refers to the impact or effect on the business or environment. This includes improvements to practice systems, efficiency savings and sustainability. Most of these elements are hard to explore or evidence as part of a relatively short evaluation, however some of the examples already provided in terms of sustained changes to practice systems and cultural shifts do suggest a move towards improvements to the business and practice environment itself. Questionnaire data also demonstrate improvements in current practice in relation to these two specific care bundles and although not all the improvements were statistically significant, the findings do indicate changes in the right direction.
I know we’re talking about improved patient care but with some of these efficiencies and some of these better ways of doing things there could actually be commercial advantages as well for the practice. HB Representative_1
Discussion
This theoretically underpinned evaluative study explored individual and team learning, beliefs and behaviours towards using QI in general dental practice. It examined the barriers and facilitators to implementing QI methodologies in practice and identified potential refinements for future development and implementation of improvement collaboratives in this setting. Findings identified positive beliefs and increased knowledge and skills towards QI. Participants also reported increased confidence about using QI methodologies in practice through participation in the Primary Care Dentistry Collaborative. Areas for improvement were highlighted, particularly in relation to participants’ knowledge and skills, which although increased during this study, also identified room for growth.
The main barriers to using QI methods were identified as time, patient and team engagement and motivation. Leadership and communication emerged as barriers in some practice contexts and facilitators in others. The care bundle topic was also identified as either enabling or hindering the process. Factors identified as facilitators were teamwork, clearly defined roles within the team, training in the methodology and understanding the potential impact the changes could have on improved patient care and the practice as a whole.
The findings of this study reinforce those of other studies evaluating QI collaboratives.19–23 Øvretveit et al identified a number of challenges which must be met to ensure the success of a collaborative approach.23 These included choosing the right subject, defining clear roles and responsibilities, the importance of team leading, forming and building and then motivating these teams. Their work suggested that the strongest motivation and confidence is gained from team members seeing the improvements that have been made. These themes also emerged strongly from this study. More recently, a similar study carried out in general practice identified competing workload priorities, time and communication with colleagues as the major challenges. They highlighted the physical challenge of getting all team members together at the same time in the same place, another barrier identified from our data. This work in general practice also reported non-clinical staff feeling ‘largely excluded’ from the process and reporting that their roles were limited or even diminished by the decision-making hierarchies that exist in general practice. This echoes the culture within general dental practices. Patient compliance was also identified as a barrier. This suggests that the barriers to implementing QI activities faced by dental practice teams are similar to that faced by general medical practitioners despite the differences in the contexts and structures within which they work.
This study benefits from a number of strengths. To our knowledge this is the first time a project following IHI’s Breakthrough Series Collaborative methodology has been tested and evaluated in primary care dentistry. Another strength is the adoption of both qualitative and quantitative approaches. Evidence suggests that the use of multimethods can enhance research findings, allowing the strengths of each approach to reinforce the overall study design.24 25 This can produce a more complete contextual portrayal of the phenomenon being studied, and the multiple viewpoints achieved through the use of both methods can improve the accuracy of any judgements and interpretations made from the data.
Practices participating in the Primary Care Dental Collaborative covered a range of practice type, size and geographical location. However, it was conducted in Scotland alone, and did not include all HB areas. Caution should also be taken when interpreting the findings due to the potential bias which might result from a comparatively small number of mainly enthusiastic volunteer practices within these HBs who could be considered ‘early adopters’. Furthermore, as is the case in most questionnaire and interview studies, participants might have provided socially desirable responses.26 The use of observations in addition to interviews seeks to address this. Two case studies were undertaken, these practices were selected to include one higher and one lower performing practice, based on self-reported questionnaire data. There were few practices at the lower end of the compliance scale, resulting in limited practices to sample from. Taking these limitations on board, consideration should be taken in relation to the generalisability of these findings more broadly. However, as is the nature of qualitative research, these findings are not intended to be generalisable, but to provide an insight into the behaviour of interest to inform future research, practice and policy.
This study presents beliefs and behaviours towards QI in general dental practice as well as perceived barriers and facilitators to implementing these methods. It includes the views of the dental team members and stakeholders from the HBs engaged in supporting the implementation process. At its core, QI in dentistry is about making changes that lead to improved patient outcomes.27 Given its rich history in medical practice, it is likely to become more common, spreading to more primary care settings and increasing the evidence base. The similarities of our findings to those gleaned from general medical practice suggest that lessons can be learnt across primary care settings and findings may be transferrable. Future work could explore settings such as optometry using a similar methodology, who operate within similar settings to general dental practice, to test the transferability of these findings. Future research may also consider the cost-effectiveness of the collaborative approach. The time, resources and expertise required to undertake this approach is significant and it is unclear whether similar gains could be made using alternative approaches and whether the changes are sustainable. Longitudinal evaluations may be required.
Our findings demonstrate increased knowledge, skills and confidence in relation to QI methodology and highlight areas to target for improvement. The study is an example of partnership working between the Scottish Government and NHSScotland, towards a shared ambition to provide safe care to every patient. The Healthcare Quality Strategy for NHSScotland has three quality ambitions: to provide safe, effective and person-centred care. The SPSP is key to the delivery of these ambitions and supports the Scottish Government’s 2020 vision to provide safe, high-quality care, whatever the setting.28 However, more work is required to evaluate the sustainability and transferability of improvement collaboratives as a means for QI in dentistry and wider primary care settings.
Data availability statement
Data are available on reasonable request.
Acknowledgments
The authors would like to thank all of the participating dental teams, colleagues at Healthcare Improvement Scotland (HIS), the Clinical Effectiveness business support team and in particular, Lorna Barnsley, who scheduled and coordinated interviews and managed the distribution of questionnaires.
Footnotes
Contributors HC: manuscript production and revisions, contributed to the scientific development, conduct, data collection, analysis and interpretation of the study. VM: contributed towards data collection, data coding, analysis and interpretation of the study. LB: contributed towards data analysis and interpretation of the study. IB: contributed to the scientific development, conduct and interpretation of the study. JC: contributed to the scientific development, conduct and interpretation of the study. LY: led the study design, conduct and interpretation of the study.
Funding This study was funded by NHS Education for Scotland and conducted by the Dental Directorate’s Dental Clinical Effectiveness Workstream. There is no grant or award number associated with this work.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.