Article Text
PDF
Abstract
Background The audit-feedback cycle is a behaviour change intervention used to reduce evidence-practice gaps. In this study, repeat audits, feedback, education and training were used to change practice and increase compliance with Australian guideline recommendations for stroke rehabilitation.
Objective To increase the proportion of patients with stroke receiving best practice screening, assessment and treatment.
Methods A before-and-after study design was used. Data were collected from medical records (n=15 files per audit). Four audits were conducted between 2009 and 2013. Consecutive files of patients with stroke admitted to the stroke unit were selected and audited retrospectively. Staff behaviour change interventions included four cycles of audit feedback, and education to assist staff with change. The primary outcome measure was the proportion of eligible patients receiving best practice against target behaviours, based on audit data.
Results Between the first and fourth audit (2009 and 2013), 20 of the 27 areas targeted (74%) met or exceeded the minimum target of 10% change. Practice areas that showed the most change included sensation screening (+75%) and rehabilitation (+100%); neglect screening (+92%) and assessment (100%). Some target behaviours showed a drop in compliance such as anxiety and depression screening (−27%) or little or no overall improvement such as patient education about stroke (6% change).
Conclusions Audit feedback and education increased the proportion of inpatients with stroke receiving best practice rehabilitation in some, but not all practice areas. An ongoing process of quality improvement is needed to help sustain these improvements.
- implementation science
- health services research
- audit and feedback
- evidence-based medicine
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Introduction
Translating research into practice from clinical guidelines is one of the most challenging stages of evidence-based practice. A recent systematic review suggests that tailored interventions which target identified determinants of practice can be effective in producing small-to-moderate amounts of practice change.1 Tailored interventions are planned strategies for improving practice that prospectively target identified determinants of practice, such as knowledge and skill barriers.1 Interventions used are often multifaceted and can include printed educational materials (PEMs) (e.g., guidelines, journal articles)2; outreach visits (e.g., group or 1:1)3; group education (e.g., lectures, workshops or facilitated interactive group discussions)4; opinion leaders5 and audit and feedback.6
Audit and feedback
Systematic reviews have shown that cycles of audit, with written and verbal feedback can change clinical practice and nominated behaviours by small to modest amounts (median change of 1%–6%).6–8 Audit of medical records has been used to change the behaviour of stroke unit teams in the UK,9 the Netherlands,10 Australia,11 Spain12 and Canada.13 To date, most studies have focused on practice areas related to emergency and acute stroke care. Audits of rehabilitation care are needed to determine the provision of optimal interventions for stroke survivors and are relevant to allied health clinicians.
Guideline recommendations are used as the basis for most of these clinical audits. The Stroke Foundation in Australia coordinates an audit of acute stroke services every 2 years, with criteria derived from the National Stroke guidelines.14 15 Australian national audits began to focus on rehabilitation practice in 2011.16 Prior to 2011, feedback was only provided about acute care and selected rehabilitation criteria such as allied health assessment and patient education.17 18 In 2009, our stroke service began internal audits to investigate compliance with stroke rehabilitation guideline recommendations.
Education interventions
Dissemination of PEMs such as clinical guidelines and journal articles is widely used as an implementation intervention. While dissemination is a relatively passive intervention, PEMs do change clinical practice by small amounts (mean change of 13%) when compared with no intervention.2 Group education such as lectures, workshops or facilitated interactive group discussions is another practice change intervention commonly used in implementation research. When used alone or combined with other interventions, group education has also been shown to result in small improvements in professional practice (6%–10%).4
Identifying determinants of practice
Identifying and addressing the determinants of practice may also influence the effectiveness of tailored interventions used to improve practice. Determinants of practice (also known as barriers and enablers) are factors that can either prevent and/or enable improvements.19 In a concurrent qualitative study, we conducted focus group interviews to identify determinants of practice following initial audit feedback.20
Objective
To increase the proportion of patients with stroke receiving best practice screening, assessment and treatment.
Methods
Design
A before-and-after study design was used. The implementation process included four components: four cycles of audit and feedback, identifying determinants of practice, provision of educational materials and group education sessions to assist staff with change. Table 1 gives an overview of the implementation process and the stages. Ethical approval was obtained from the local health district to conduct the study (Ref No. QA 2008/046). Approval was granted to audit patient records without gaining written consent from individual staff, or from patients who had been discharged. The TIDieR21 and SQUIRE guidelines22 were used for reporting the current audit and feedback study.
Overview of implementation process
Setting
The project was conducted at one metropolitan stroke unit in Sydney, Australia. This is a 20-bed comprehensive unit where patients are admitted directly from the emergency department to receive their acute care and remain in the unit for their inpatient rehabilitation.23
Participants
A total of 31 stroke unit staff across eight disciplines (medical, nursing, occupational therapy, physiotherapy, speech pathology, social work, dietetics and orthoptics) were invited to participate in the study. Staff were notified of our intention to audit their written medical records and to provide subsequent feedback. No patients participated in the study.
Baseline audit
The audit checklist
An audit checklist was developed in 2009, which included recommendations from Australian (2005), Scottish, English and Canadian stroke guidelines14 24–26 for the initial audit. The checklist was updated using 2010 Australian guidelines for subsequent audits.15 Practice areas audited included communication, swallowing, upper limb function, mobility, activities of daily living, vision, neglect, return to work, depression/anxiety, sexuality, driving, task practice, education and home assessment. Audit criteria were rated using yes/no/not applicable response options. Questions were asked about screening, assessments conducted and intervention provided in relation to each practice area. A sample of the audit tool is provided in online supplementary 2.
Supplementary file 1
The baseline audit (audit 1) was conducted in November 2009. For each audit, a consecutive, retrospective sample of 15 medical files was selected of patients with stroke admitted to the stroke unit. When a file was not available for audit, the previous patient name on the list was selected.
Audits were conducted by allied health and nursing staff, students and members of the research team. Multiple auditors (n=15) were used to raise staff awareness of the auditing process, individual and team practice and to engage staff in the audits. Each staff member was asked to audit at least one medical record. Two medical files from each phase of audits were double audited and coded by one of the two research team members to check for rater consistency. Differences were discussed and consensus reached when necessary. No formal study of rater agreement was conducted.
Audit feedback and guideline dissemination
After the baseline audit, the research project manager (RPM) presented the results to allied health, nursing and medical staff about each profession’s compliance with the relevant criteria, in a face-to-face feedback session. Presentations with graphs and tables were used to present results for each practice area (e.g., education), and summarise individual practice areas with good compliance or areas needing improvement (see online supplementary 3 for an example of audit feedback presented to staff). An audit summary report was provided to all staff describing the audit results. Reports included a summary of individual practice areas with good compliance and those areas needing improvement. For each practice area (e.g., communication), results were displayed in tables showing the % compliance for each criterion audited (see online supplementary 4 for an example of a feedback report). Each discipline was then invited to nominate at least one area for practice improvement. The RPM provided each discipline with a printed copy of the guidelines14 and highlighted recommendations relevant to them and the nominated areas. The RPM provided feedback sessions and an audit summary report to all staff after each subsequent audit.
Identifying determinants of practice
Assessing the problem and linking to Theoretical Domains Framework (TDF)
A barrier analysis was conducted by the researchers in 2009 after the initial audit. To identify barriers, we used two methods which are recommended when conducting implementation research.27 We conducted focus group interviews between July and October 2010 (described elsewhere20) with allied health, nursing and medical staff which were transcribed and analysed for themes using the TDF.28 This parallel qualitative study was approved as a separate project (ref: 2009/012). The TDF comprises 14 domains and factors that are likely to influence healthcare professional behaviour and can be targeted to facilitate practice change: knowledge; skills; social/professional role and identity; beliefs about capabilities; optimism; beliefs about consequences; reinforcement; intentions; goals; memory, attention and decision processes; environmental context and resources; social influences; emotion and behavioural regulation.28
Six key barriers were identified and were different for each discipline and intervention. (1) Beliefs about the capabilities of individual clinicians or their discipline to deliver an assessment or intervention, and beliefs about patients’ ability to participate in an intervention. For example, physiotherapists (PTs) had concerns about the physical demands on staff and patients during treadmill training with weight support. (2) Beliefs about the consequences, both positive and negative, of providing or not providing an assessment or intervention was the second category of barrier. For example, speech pathologists (SPs) were concerned about incorrectly interpreting results of aphasia assessments for patients with limited English and whether vision problems would alter test scores. (3) Memory attention and decision processes was a third category of barrier and refers to systems and prompts that reminded clinicians to deliver an intervention. For example, medical staff reported a lack of systems that prompted discussion and documentation regarding return to work, driving and anxiety/depression. (4) Limited knowledge and skills required to deliver evidence-based assessment and intervention was another category of barrier. For example, occupational therapists (OTs) lacked knowledge and skills in assessment and interventions for managing sensation and neglect. (5) Intention and goals was the fourth category of barrier and refers to clinicians’ lack of intention to provide assessment/intervention and having/prioritising goals. For example, OTs reported difficulty completing sensation and neglect assessment and intervention due to competing rehabilitation/goal priorities. (6) Limited resources was the final category of barrier. For example, nursing staff lacked access to educational materials (including translated material) to give to patients. Further details of barriers are provided in online supplementary 1 and reported elsewhere.20
Developing the intervention
The researchers used the behaviour change wheel (BCW) to design the intervention.29 The online supplementary 1 shows how the TDF and BCW were applied to targeted practice areas, linking the barriers and enablers to the TDF, BCW domains, intervention functions (IF) and resulting intervention.
Mapping the BCW Capability-Opportunity-Motivation-Behaviour (COM-B) system to TDF domains
using the BCW, the TDF domains were mapped to the COM-B domains. The process assumes that capability, opportunity and motivation interact to generate behaviour change.29 For example, in order to improve the OTs’ knowledge and skills in neglect rehabilitation, we needed to maximise their psychological capability.
Linking components of the COM-B system to relevant intervention functions: the BCW links COM-B and TDF domains to IF. This linkage process helped to identify intervention options that were most likely to address key barriers identified by each discipline. Using the above example, if knowledge is an identified barrier, education is the recommended IF.
Forming possible solutions (behaviour change techniques (BCTs))
The researchers then selected interventions to be implemented with each discipline and for each practice area, based on relevance, feasibility and acceptability of the intervention.
Education and training sessions
During phase I of the intervention period (February–November 2010), allied health, nursing and medical staff met with the RPMs monthly and participated in education and training sessions. During phase II (November 2010–May 2011), training sessions were reduced to bimonthly/quarterly. Compliance targets were set and progress was also reviewed during the sessions.
Several different BCTs were used with each discipline during the education/training sessions to improve their practice. Within the domain/IF of education, two main BCTs were used. First, feedback on behaviour was provided to all disciplines following medical record audits about their compliance with delivery of screening/assessments and interventions. Second, information was provided about the health consequences and outcomes of performing a behaviour. For example, using published research to provide OTs with information about the benefits and outcomes of delivering interventions for sensation and neglect.
Within the IF of training, three key BCTs were used, including demonstration, instruction and practice of a behaviour. For example, physiotherapy staff participated in a training session where senior staff demonstrated and staff practised skills required for training sitting balance and treadmill training. Occupational therapy staff participated in training about neglect assessment and rehabilitation with a known clinical expert. Joint neglect rehabilitation sessions were also conducted with the stroke unit orthoptist.
Within the IFs of environmental restructuring and enablement, two key BCTs were used. First, prompts and cues were provided, such as regular caseload reviews with PTs to prompt provision of sitting balance and treadmill training. A case conference checklist was introduced for use by medical staff to prompt action regarding anxiety and depression, driving and return to work. Second, resources (objects) were ordered and added to the environment/setting to assist with screening and assessment e.g. the Nottingham Sensory Assessment (OT) and Mississippi Aphasia Screening Tool (SP). Other materials introduced to the environment included education packs and an ink stamp for documentation of education.
Within the IF of persuasion, three primary BCTs were used. First, information was provided about the health consequences and outcomes as described earlier. Second, the use of visual and verbal communication from credible sources to encourage clinicians to use assessments or interventions. For example, use of supporting published research in discussions with OTs and PTs and an expert neglect clinician and orthoptist with OTs to encourage increased use of interventions. Third, encouraging staff to focus on past success with implementing assessments or intervention. For example, encouraging physiotherapy staff to reflect on and discuss previous success with delivering sitting balance and treadmill training to patients.
Evaluation
Subsequent audits were conducted in November 2010 (audit 2), May 2011 (audit 3) and July 2013 (audit 4) to measure change using the same methods as the baseline audit. A total of 60 medical files (n=15 per audit) were audited over a 4-year period. The primary outcome measure was the proportion of eligible patients receiving best practice against target behaviours, based on audit data. Based on the reviews by Grimshaw et al 7 and Jamtvedt et al, 8 consensus was reached that a 10% change in the target practice behaviours would be the goal for each practice area (i.e., the predetermined minimum clinically worthwhile difference).
Results
Sample characteristics
Stroke unit staff participant characteristics
A total of 26 staff and students from the stroke unit audited at least one record. The majority of auditors were female (n=21, 81%). Disciplines included OTs (n=7, 27%) PTs (n=6, 23%), students (n=5, 19%), SPs (n=2, 8%) and nurses (n=2, 8%) as well as researchers (n=4, 15%).
Patient participant characteristics
In the 2009 audit (n=15), the mean patient age was 73 years (SD=13.3), and 47% were male. For the 2010 audit, the mean patient age was 75 years (SD=12.8), and 60% were male. For the 2011 audit, the mean age was 78 years (SD=9.4), and 47% were male. For the 2013 sample, the mean age was 77 years (SD=7.9), and 67% were male. There was no significant difference in patient age or gender across the samples.
Baseline results
The baseline audit of medical records revealed a number of practice areas to be improved. Staff agreed that compliance of 60% or less indicated areas requiring practice improvement. Compliance in the following areas was low and became the target for practice improvement: management of communication (0%–53% of eligible files complied); treadmill training (17%), sitting balance training (25%), management of upper limb sensation (0%); management of neglect (0%–53%); patient and carer education (0%–27%); anxiety and depression screening (27%) and discussion regarding return work and driving (0%–33%) (see table 2 for detailed baseline results).
Number and proportion of patients receiving best practice screening, assessment and intervention between 2009 and 2013
Audit results
Table 2 and figure 1 present results over time for each discipline and practice area targeted.
Proportion of patients receiving best practice screening, assessment and intervention between 2009 and 2013.
Speech pathology
After the final audit in 2013, overall improvements of between 5% and 100% were noted in the management of communication and swallowing. For dysphagia screening, a small overall improvement of 5% was reported from 80% (95% CI 60% to 100%) in 2009, up to 85% (95% CI 65% to 104%) in 2013. A large change was recorded in motor speech intervention from 0% in 2009, up to 100% in 2013.
Physiotherapy
Overall, moderate improvements in the delivery of treadmill training with partial body weight support and sitting balance training practice were noted. An overall improvement of 33% was recorded in the provision of treadmill training, from 18% (95% CI −5% to 41%) in 2009 up to 50% (95% CI −19% to 119%) in 2013. Provision of sitting balance training improved from 25% (95% CI −5% to 55%) to 67% (95% CI 29% to 104%), an overall improvement of 42%.
Occupational therapy
Overall improvements of between 0% and 100% were noted in management of upper limb sensation and neglect impairments. While there was some improvement noted after the second and third audits, no overall change (0%) was seen in the standardised assessment of sensation with no patients being assessed at baseline in 2009 or after the final audit in 2013. A large change was recorded in provision of sensory training from no patients receiving the intervention at baseline (0%) to all eligible patients (100%) receiving the intervention in 2013.
Nursing
Improvements of between 6% and 62% were noted in delivery of patient education and 10% and 37% for carer education. Little overall change (6%) was seen in patient education about the nature of stroke with 27% (95% CI 4% to 49%) receiving education in 2008 and 33% (95% CI -8% to 75%) in 2013. A large overall improvement (62%) was seen in patient education about stroke impairments and management from 7% (95% CI −6% to 19%) to 69% (95% CI 44% to 94%).
Medicine
Results relating to screening for anxiety and depression and discussion regarding return work and driving were variable after the final audit. There was a drop in documented evidence of screening for anxiety and depression from 27% (95% CI 5% to 49%) in 2009 to 0% in 2013. A large improvement (25%) was noted in discussion around return to work from 0% up to 100%.
Discussion
This study investigated the use of a multifaceted tailored behaviour change intervention programme, including audit and feedback, education and training, use of prompts and cues and resources, to increase the proportion of patients with stroke receiving best practice screening, assessment and treatment. Practice improvements of up to 100% were achieved over a 4-year period. The majority of practice areas improved by at least 10%, which we consider to be clinically worthwhile.
Changes in screening, assessment and intervention: large changes were recorded in several practice areas including sensation screening (+75%) and rehabilitation (+100%); neglect screening (+92%) and assessment (100%). Several factors may have enabled changes in screening and assessment practices, compared with intervention. First, screening and assessment resources were often freely available online and easy to obtain, administer and implement, requiring no special training (e.g., the Catherine Bergego Scale for neglect screening). Second, access to free education and training by senior therapists helped to address the skills and knowledge barrier (e.g., visual scanning for neglect rehabilitation). Third, assessments and interventions that showed the greatest improvement required minimal or easily accessible equipment. For example, the Nottingham Sensory Assessment uses everyday objects such as a comb, scissors and coins, making implementation easier. Another interesting observation is that most areas showing the greatest improvement had very low compliance rates at baseline (mostly 0%).
While some practice areas improved, others were resistant to change. There was a drop in compliance with anxiety and depression screening (−27%), and little or no change was recorded in the delivery of education to patients with stroke regarding the nature of stroke (6%) or notifying local doctors regarding return to driving (0%). Some areas showed improvement after the second audit but by the third or fourth audits, there was a drop in compliance, for example, standardised aphasia assessment (0%); and standardised sensation assessment (0%). This drop in compliance has been observed in other stroke implementation research and is described as a ‘decay effect’.30
While a process evaluation was not conducted concurrently, there are some factors which may explain the lack of behaviour change in some areas, or sustainability of change. First, while we did not formally monitor staffing levels during the study period, we observed some staff turnover across all disciplines. Despite the staffing changes, we were still able to achieve practice improvements. Second, the dose of behaviour change intervention delivered may have been too small to effect change (e.g., annual audit feedback and education and training primarily delivered in phase I). Third, we only explored and addressed local barriers once, in 2009. The barriers facing staff may have changed after 2009. In addition, staff and managers changed during the subsequent audit periods and review of perceived barriers may need to have been acknowledged and addressed. Lastly, a lack of familiarity or agreement with some of the recommended assessments or interventions may explain the lack of uptake among some disciplines and has been observed in other implementation trials.13
Strengths of this study include use of a systematic approach to design and implement a multifaceted tailored intervention which targeted identified barriers. Use of the TDF and BCW is another strength, providing a framework for selecting and matching interventions to key barriers for each discipline. This is the first study to use such an approach when targeting the behaviour of clinicians and stroke rehabilitation practice.
Study limitations include the use of medical record audits as a measure of practice. While audits are widely used in implementation research, written medical records may not always accurately represent actual practice (e.g., they may under-report the use of screening, assessment and intervention). An alternative to auditing is observation of practice, which is more resource intensive and costly. This was not considered feasible for this study. Other study limitations include the small number of files audited (n=15 per cycle), use of a single site and lack of a control or comparison site.
Advantages of a theory-informed behaviour change programme: while tailored behaviour change interventions continue to be used widely in implementation research to improve practice, the results of such studies are variable and the best approach is still not clear.1 This study used the TDF and BCW as frameworks to identify, address and match identified barriers to appropriate behaviour change interventions. This theory-informed process resulted in a multifaceted tailored intervention programme aimed at changing professions’ and individual clinicians’ behaviour and practice. The use of the TDF and BCW is a newer approach to designing tailored interventions in implementation research. Other studies have used this approach to increase implementation of an early mobilisation intervention in acute hospital setting,31 improve management of the risk of falls after hospital discharge,32 improve the management of blood pressure33 and improve management of medication in people with multiple morbidities.34
Conclusions
The use of a multifaceted tailored behaviour change intervention programme including audit, feedback and education improved some practices of allied health, nursing and medical professionals over a 4-year period. Overall, more patients received best practice stroke rehabilitation. The quality improvement process needs to continue to sustain the gains made. The stroke service planned to conduct their own audits periodically using the same number of files and select criteria. These audits will focus on monitoring improvements made in practice areas targeted as well as target areas in need of further improvement, for example, patient and carer education. Discussions are ongoing with stroke unit staff on other ways to make the implementation process sustainable into the future.
Acknowledgments
The authors would like to thank the stroke unit staff for their participation and to Brad Lloyd who assisted with physiotherapy education sessions.
References
Footnotes
Contributors AV, AM and KS did the conception, design, analysis and interpretation of data.
Funding This study was supported by an infrastructure grant from the Ingham Institute for Applied Medical Research. The funder had no role in the design, conduct, analyses or reporting of the study or in the decision to submit the manuscript for publication.
Competing interests AM was supported between 2007 and 2011 by fellowships from the National Institute of Clinical Studies - National Health and Medical Research Council, the National Stroke Foundation, the University of Sydney, Royal Rehabilitation Centre Sydney.
Ethics approval This study was approved by the SWSLHD Human Research and Ethics Committee (HRECNo. QA 2008/046).
Provenance and peer review Not commissioned; internally peer reviewed.
Data sharing statement Full copies of the resources, that is, audit tool, audit reports and audit presentations are available by request to angela.vratsistas@sydney.edu.au.