Article Text
PDF
Abstract
Strong primary care is foundational for effective, efficient health systems but remains variable in health systems around the world. Increasing quality improvement capacity in primary care support organisations has potential to improve primary care and health systems. This project worked with staff from primary healthcare support organisations with the aim of improving by 20% the confidence and competence scores of participants seeking to implement a quality improvement activity. The Breakthrough Collaborative approach was used to design a programme of learning workshops, action periods, data feedback and local support. Improvement measures included confidence in quality improvement (self-rated) and competence in quality improvement (using the validated Quality Improvement Knowledge Assessment Tool). Participants were required to submit quality improvement plans and run local quality improvement projects. The programme was run three times with improvements made between each cycle. Overall, 50 teams consisting of 173 regional staff took part. They engaged a total of 341 front-line primary care services such as general practices, Aboriginal medical services and pharmacies. In the first cycle, there was no improvement in measured regional staff knowledge and skills. In response, the learning workshops were changed to increase didactic teaching and supported practice of fundamental quality improvement skills. Regional organisational leaders were consulted to decrease the turnover in participants during each collaborative cycle. In the final cohort, a 38% improvement in knowledge and skills was recorded. All teams submitted quality improvement plans and ran local workshops indicating good engagement. This programme addresses a key need for health systems that is shared around the world: the improvement of primary care. It demonstrates using the Breakthrough Collaborative methodology to increase quality improvement capacity in the sector. After initial challenges, repeated cycles recorded greater than 30% improvement in the measured competence of participants in quality improvement activities.
- collaborative, breakthrough groups
- primary care
- general practice
- team training
- quality improvement
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Problem
This project sought to improve levels of quality improvement (QI) capacity in Australian primary care services, such as general practices, Aboriginal health services and pharmacies, by building capacity in the regional organisations that support them.
High-quality person-centred primary care has the potential to transform the efficiency and effectiveness of national health systems.1 2 Despite this being internationally recognised, primary care services in many countries, including Australia, remain variable in quality.3 4
The project was delivered by the Improvement Foundation Australia, a ‘not for profit’ company which has experience in primary care QI through leading the government-funded Australian Primary Care Collaboratives (APCC) programme between 2004 and 2015.5–7 It was delivered by a small team including two project managers, an improvement expert and a general practitioner.
After 8 years of working directly with general practices, this project was an opportunity to address sustainability of QI in primary care through working with regional primary care support organisations. The project modelled and taught QI skills by using a quality improvement collaborative (QIC) paradigm for implementation.
Consistent with QIC methodology,8 an expert reference panel, convened for the project, set the aim in box 1.
Project Aim
To improve by 20%, or exceed an 80% rating in, the confidence and competence of participants to develop and implement a quality improvement activity.
Background
Improving primary care processes and outcomes has been identified as a key strategy for improving health systems.2 9 While Australia has good health outcomes, demonstrated by high life expectancy and low smoking rates, the nation has relatively poor outcomes in some measures such as obesity.10 11 The Caretrack study found care in Australia was appropriate in just 57% of health encounters.12
QICs have been identified internationally as effective interventions for implementing system improvements.13–15 They are multifaceted interventions that bring groups of professionals together to work on accelerating better outcomes for targeted topics. Their common characteristics have been well described13–15 and emphasise collaborative learning, support and exchange of insights among participating healthcare organisations. This project employed the Breakthrough Series Collaborative model pioneered by the Institute for Healthcare Improvement.8 It was part of the APCC programme delivered between 2004 and 2015. The APCC has demonstrated improvements in measures related to various clinical topics such as diabetes, chronic obstructive pulmonary disease, chronic kidney disease and cardiovascular disease.6 7
Measurement
The measures for the project were developed by the expert reference panel and are listed in box 2.
Project improvement measures
Participants’ confidence and competence with developing and delivering quality improvement (QI) activities.
The number of PDSA (Plan, Do, Study, Act) cycles submitted each month (first participant cohort only).
The development of a QI plan.
The number of general practices and health services:
providing data to their regional support organisation each month
engaged in local QI workshops.
Confidence in using QI was measured in the participants by self-rating questionnaires administered before and after the collaborative learning workshop series. These questionnaires were developed for the project (see online supplementary file 1). Competence in QI was measured using the Quality Improvement Knowledge Assessment Tool (QIKAT). The QIKAT is a validated instrument developed to measure QI competence in resident medical staff.16 The format and scoring rubric of the QIKAT were used with permission and the scenarios were modified to reflect the work environment of participants. The success of participating regional support organisation staff in engaging local front-line teams was measured by their data submission rates and local workshop attendance.
Supplemental material
Baseline measurements
Average baseline QI confidence self-assessment scores were 75%, 69% and 70% for cohorts 1, 2 and 3, respectively. Average baseline QI competence scores using the QIKAT were 16, 15 and 14 out of a possible score of 27 in cohorts 1,2 and 3, respectively. Baseline measures are reported in more detail as part of the Results section given below.
Design
The project followed the Breakthrough Series Collaborative design8 using a series of learning workshops, action periods, local support and continuous measurement as outlined in figure 1.
Collaborative cycle diagram.
Regional primary care support organisations were invited to send teams to participate in the project. All teams received an orientation and support to identify local improvement priorities. The project was delivered to three consecutive participant cohorts. In each cohort, small teams from participating support organisations attended a series of three face-to-face, 2-day learning workshops addressing QI techniques, the psychology of change, learning styles, measurement and methods for spreading change. Each regional team developed aims, measures and change ideas related to their chosen improvement topic. These were articulated in structured QI plans. Improvement ideas were shared between teams at the workshops and throughout the Collaborative.
During the project, teams received support from the central implementation team through monthly teleconferences and personal contact. Following each workshop, teams implemented skills they had learned in the workshops by working in their local organisations and with the local primary care services they support. During these action periods, they engaged regional colleagues and recruited front-line health services to participate in improvement work on their chosen topics. As part of the project, each team was required to hold at least one local workshop with the primary care services they recruited. At these workshops, front-line clinicians received training in QI skills and training in specific knowledge relevant to the selected improvement priority. In this way participating teams practised and passed on the QI skills they learnt at the collaborative learning workshops.
Strategy
The project was delivered over three cycles with consecutive cohorts of participants. The faculty implementing the project met after each QIC cohort cycle to analyse the improvement measures and achievement of improvement aims and make revisions based on analysis.
Improvement cycle 1
In cycle 1, we aimed to implement the multifaceted intervention successfully with good engagement of regional teams and improved confidence and competence measures. Our hypothesis was that the QIC structure with its combination of workshops, action periods, measurement and local support would result in good engagement and increased confidence and competence in participating teams.
There was little change in participant QIKAT scores in this cycle (see table 1 below), suggesting little increase in competence in basic QI skills. The implementation team observed that the workshops were heavy on didactic content on a wide range of topics with reduced time for practising skills.
Change in participant competence (QIKAT) score (paired t-test)
Participants in this cycle selected a wide range of improvement topics (see table 2), with a lack of commonality, which limited opportunities for shared learning between regional teams. Some of the selected topics had little published evidence on which teams could draw, necessitating the development of individualised measures of improvement. Participating teams found it challenging at this early stage of training to develop high-quality QI measures that could demonstrate change.
Clinical improvement topics addressed in each cohort
Although there was strong engagement of participating teams with achievement of all engagement measures, our key learning from this cycle was that the programme had not been effective in building measured competence in QI in participants.
Improvement cycle 2
In cycle 2, we aimed to improve the measured skills in QI competence and reduce and refine the improvement topics selected by teams to encourage shared learning. We hypothesised that focusing the workshop content on fewer topics and increasing practical content would result in improved learning.
The QIC faculty sought more input from regional organisation leaders in the design of the programme for cycle 2 to increase organisational buy-in. Teams were asked to submit their QI plans earlier in the cycle to enable feedback and development.
The QIC faculty changed the learning workshop content to focus more on skills in identifying aims, measures and change ideas by answering the three improvement questions of the Langley and Nolan Model for Improvement (see box 3).17 This was complemented by increased mentored ‘team time’ for practising these skills. Sessions on learning styles, the psychology of change and spreading change were retained but reduced. The QIKAT marking rubric was used to inform training and provided an accessible summary of the qualities of a good aim, measure and change idea when designing an improvement strategy (see box 3).
The three improvement questions (Langley and Nolan)17 and characteristics of a good answer (based on the Quality Improvement Knowledge Assessment Tool marking rubric)16
A good aim
focuses on the system-level of the problem presented
includes direction of change (increase or decrease)
includes at least one specific characteristic such as magnitude (% change) or time frame.
A good measure
relates precisely to the aim
is readily available so data can be analysed over time
captures a key process or outcome.
A good change idea
links directly with the aim
uses existing resources
provides details to initiate a test of change.
The QIC faculty provided more guidance to teams in selecting an improvement topic. Teams were required to consult recently completed local health needs analyses to inform their topic choices. A number of topics were identified that were common to most regions, had an established evidence base and had established measures. This resulted in a smaller range of topics (see table 2), which facilitated more shared learning. Teams appeared to be motivated by the importance of the local needs they were addressing and by interactions with others on the same topic.
It was noted that there was a high turnover in participants during cycles 1 and 2. This prevented participants from completing before and after measures and may have affected learning.
Our key learning from this cycle was that reducing and focusing the learning session content with more time to apply knowledge appeared to result in increased foundational QI skills.
Improvement cycle 3
In cycle 3, we aimed to further improve competence measures as well as reducing participant turnover. Our hypothesis was that further emphasis on aim, measure and change idea design and on practical skills, would improve competence in QI. We hypothesised that by engaging with leaders and participants we could increase understanding of the programme and commitment to attendance.
The learning objectives for the workshops were revised to focus even more on the Langley and Nolan Model for Improvement with the aim of ensuring teams were confident in its use. More time was spent critiquing answers to the three improvement questions and draft PDSA (Plan, Do, Study, Act) cycles. More time was allocated to practising the use of a smaller number of QI tools such as Ishikawa diagrams and process mapping.
Participating teams were requested to ensure the same staff attended all workshops. It will be noted from tables 1 and 3 that a higher proportion of participants completed before and after measures in cycle 3 suggesting increased retention. A reduced range of improvement topics was addressed facilitating shared QI learning.
Change in participant confidence self-assessment score (paired t-test)
Our key learning from this cycle was that dialogue with leadership and participants appeared to result in reduced turnover in participants and improved topic selection in the QIC.
Results
Programme implementation
Three cohorts of teams participated in the project commencing between 2012 and 2014 (table 4). There was a high turnover of attendees between the first, second and third learning workshops in cycles 1 and 2. This was related in part to organisational issues in the sector during the early phases of the project.
Participation in project activities
Participants’ confidence and competence with developing and delivering QI activities
Paired sample t-tests were used to assess changes in self-assessed confidence and QIKAT scores of competence in participants completing both before and after tests. There were no significant improvements in QIKAT (QI competence) scores in cohort 1 (table 1). Significant changes in confidence measures were recorded in all groups (table 3). After changes to workshop content, useful and statistically significant improvements in QIKAT scores (QI competence) were recorded in cohorts 2 and 3.
Engagement of front-line health services
Regional support staff participating in the project engaged 341 front-line health services (general practices and pharmacies). These services were expected to submit relevant local QI measure data monthly. The data submission rate from cohort 1 is not available. In cohort 2, 63 of 66 health services (95%) completed 100% of monthly data submissions. In cohort 3, 70 of 95 health services (74%) completed 100% of monthly data submissions. Participating teams met the key engagement milestones, which were the development of QI plans and the engagement of front-line clinical teams in workshops (table 4). Effective engagement of front-line clinical services by participating teams was reflected in the number of front-line services engaged and their data submission rates.
Lessons and limitations
The QIC methodology was effectively applied employing the identified multifaceted components of the model. This approach combined ‘theory’, which was taught in the learning workshops, with ‘practice’ in action periods, as the participating regional teams worked with their front-line services to implement QI plans. The project was kept on track by a clear aim and relevant measures developed by the expert reference panel. The search for a validated tool to measure QI competence was driven by the aim. The QIKAT measure enabled early identification of problems with the design of the workshop series and correction during improvement cycles.
The QIKAT is the most well-known measure of competence in QI despite some questions around its subjective nature and inter-rater reliability. The confidence measure used was developed for this programme and it has not been validated (see online supplementary file 1). The assessment of QI learning remains challenging.16 This was an implementation project rather than a research study and as such the uncontrolled before and after design employed is reasonable though subject to limitations.
The use of the QIKAT drove changes aimed at ensuring participants gained skills in handling the three basic improvement questions of the Langley and Nolan Model for Improvement. Faculty teaching the programme learned the importance of addressing the questions and their logical flow. Designing a good aim (in answer to question 1) makes designing good measures much easier (question 2), which in turn helps in the identification of changes which are likely to be effective (question 3) (see box 3).
Engagement at all levels was a strength of the project. Engagement with regional organisational leadership and participants enabled early problems with topic selection and participant turnover to be addressed. All participating teams submitted their improvement plans and organised local workshops. There was good evidence of engagement with local primary care services in the large number recruited to local workshops and the high levels of data submission.
The project hit a significant problem when it measured no change in QI competence in the first improvement cycle. Participating teams rated the programme workshops highly for satisfaction and measures of QI confidence increased in all cohorts. However, the faculty was concerned that participants showed no significant improvement in QIKAT scores after three learning workshops. Reflecting on this result, the faculty considered how to improve competence as measured by the QIKAT and chose to focus on teaching the three improvement questions (box 3) and increasing mentored practical work. These changes coincided with statistically significant improvements in QIKAT (competence) scores in cohorts 2 and 3 (see table 1).
Encouraging participating teams to select their own topic for local QI plans was seen as strategic in that it would meet the needs of participating organisations as well as motivating participants. However, it resulted in a very diverse range of topics (see table 2) which were in some cases challenging to address for those early in their acquisition of QI skills. This made it difficult for teams to design high-quality aims and improvement measures. It also limited shared learning between teams. The process of learning and mastering the three questions is made easier if the improvement topic selected is evidence based and has well-developed measures. Supporting teams in their choice, provision of a small range of common and well-developed topics and engagement with regional leadership resulted in a more constructive range of improvement topics appropriate for learning foundational skills.
Participant turnover was high, particularly in the first two cycles of the project. Attendees at workshops 2 and 3 often lacked foundational knowledge taught in the first workshop. This also meant that QIKAT assessments at the end of the series were less meaningful. It was decided that the evaluation of the workshop series would depend on paired QIKAT measures (as reported in our analysis) as this would give an insight into the effectiveness of the training. The QIC implementation team also engaged with leadership and participants to decrease turnover with some success as measured in cohort 3 (see table 1).
A national reorganisation of primary care support organisations resulted in cessation of this project. However, many of the staff who participated continued to work in the environment and in the new organisations which perform similar functions. A culture of QI appears to be emerging in the sector with an increasing focus on data collection for improvement. It is likely that the training of 173 participants which occurred has contributed to this culture.
Conclusions
This project addresses a key need for health systems that is shared around the world: the improvement of primary care. It used a strong methodology (the QIC) to address a gap in the Australian environment: the capacity of regional organisations to support front-line services in QI. The project added a missing building block to a decade of work through the APCC programme by providing a model for building regional capacity in this key area. After initial challenges, repeated cycles achieved the aim of a 20% improvement in the confidence and competence scores of participants in QI activities. The project demonstrates that the QIC methodology can be used for capacity building across a system.
Acknowledgments
The authors acknowledge the hard work and enthusiasm of the many participants in this programme.
References
Footnotes
Contributors All authors were involved in the design, delivery and improvement of this programme. AWK developed the article idea, analysed the results and wrote the article. AWK is responsible for the overall content of the article and acts as guarantor. MD and CS extracted and cleaned data and provided comments on the text. JJ provided advice and comments on the text.
Funding This project was funded by the Australian Government Department of Health and Ageing.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.