Article Text
PDF
Abstract
Background To create a theory-informed survey that quality improvement (QI) teams can use to understand stakeholder perceptions of an intervention.
Method We created the survey then performed a cross-sectional survey of QI stakeholders of three QI projects. The projects sought to: (1) reduce unplanned extubations in a neonatal intensive care unit; (2) maintain normothermia during colorectal surgery and (3) reduce specimen processing errors for ambulatory gastroenterology procedures. We report frequencies of responses to survey items, results of exploratory factor analysis, and how QI team leaders used the results.
Results Overall we received surveys from 319 out of 386 eligible stakeholders (83% response rate, range for the three QI projects 57%–86%). The QI teams found that the survey results confirmed existing concerns (eg, the intervention would not make work easier) and revealed unforeseen concerns such as lack of consensus about the overall purpose of the intervention and its importance. The results of our factor analysis indicate that one 7-item scale (Cronbach’s alpha 0.9) can efficiently measure important aspects of stakeholder perceptions, and that two additional Likert-type items could add valuable information for leaders. Two QI team leaders made changes to their project based on survey responses that indicated the intervention made stakeholders’ jobs harder, and that there was no consensus about the purpose of the intervention.
Conclusions The Stakeholder Quality Improvement Perspectives Survey was feasible for QI teams to use, and identified stakeholder perspectives about QI interventions that leaders used to alter their QI interventions to potentially increase the likelihood of stakeholder acceptance of the intervention.
- quality improvement
- patient safety
- PDSA
- surveys
Data availability statement
Data are available on reasonable request. Data available on request from corresponding author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Quality improvement (QI) projects are an important and common activity in healthcare organisations. There is a need to evaluate the effectiveness of the QI methodologies such as Six Sigma that have been adapted from other industries.1 Six Sigma, for example, includes tools for QI teams to use such as fishbone diagrams, brainstorming tools, affinity diagrams and statistical process control charts.2 A critical but largely unstudied tool of successful QI is stakeholder analysis, by which QI teams obtain a thorough understanding of the perspectives of all the individuals who need to support the project and whose work will be impacted by it. Without this understanding a QI project may fail because it does not have support from key leaders and frontline clinicians, or because improvement interventions are incompatible with existing work requirements and processes.3 Although there is a maturing literature in business and management, including a large number of stakeholder analysis tools,4 these tools have not been adapted for healthcare QI, are not based on current theories about healthcare improvement, and are not rigorously evaluated.
Normalisation process theory (NPT) ‘identifies factors that promote and inhibit the routine incorporation of complex interventions into everyday practice.’5 Because of this focus on the adoption of interventions by a critical group of stakeholders, we believed some of its specific components could be used to inform the development of a survey for QI leaders to use to conduct stakeholder analysis for QI projects. Therefore, we created and tested an NPT-informed survey, the Stakeholder Quality Improvement Perspectives Survey (SQuIPS), that QI teams can use to understand stakeholder perceptions of an intervention.
Methods
Survey creation and pilot testing
Several authors (EF, ET, MO, JE, ES and AT-C) drafted initial survey items based on NPT components5 applicable to stakeholders of QI projects of the type routinely conducted in hospitals (eg, projects in specific clinical areas trying to improve compliance with evidence-based guidelines, to meet publicly reported quality metrics, improve safety or improve timeliness of care). Language specific to any QI project’s intervention was avoided so that the survey would be applicable to most QI projects. We did not attempt to create items to measure every component of the NPT model, but rather created a brief, actionable survey to inform QI project leaders about stakeholder perspectives. The authors, two other QI leaders in our organisation, and four frontline clinicians who were representative of future survey respondents then revised the survey by providing feedback on a shared document to improve readability and content validity.
The final survey contained 12 items based on the NPT components of coherence, cognitive participation, and collective action. We did not incorporate elements of reflexive monitoring in our items because that component was too difficult to translate into meaningful and actionable survey items (ie, participants would need to speculate about the future). The 12 items used a combination of Likert-type, multiple-choice and open-ended questions to understand aspects of coherence (five items total; four Likert type, one multiple choice), cognitive participation (three items total; all Likert type) and collective action (four items total; two Likert type, one multiple choice and one open-ended). We believed that a mixture of response options (Likert type, multiple choice and open-ended) were required to accurately measure the NPT components while also keeping the survey brief, relevant to QI project stakeholders and leaders, and actionable. We also added an additional open-ended question for respondents to list barriers that they thought would impact the intervention. The translation of NPT domains to our survey is demonstrated in table 1.
Relationships between NPT components and items on the Stakeholder Quality Improvement Perspectives Survey
After we tabulated results of the surveys and they had been reviewed by the three QI team leaders, we asked them how they used the survey results. We asked two questions: (1) Do you believe the survey gave insight into the perspectives of stakeholders that you would not have known without the survey? and (2) Did you change anything about the intervention(s) after you saw the survey results? If yes, what did you change?
Setting
We conducted the study in three settings affiliated with a single academic medical centre. We invited three QI teams by purposely sampling from a large pool of QI projects being conducted at the centre. Our goal was to survey QI stakeholders from different clinical settings and different size QI teams, and who experienced different types of QI interventions. The first three QI team leaders we asked agreed to participate. Their QI projects were conducted in a neonatal intensive care unit (NICU) in a large tertiary care teaching hospital (project 1), the operating rooms of a county safety net hospital (project 2) and gastroenterology procedure suite in an academic multispecialty outpatient practice (project 3).
Project 1 sought to reduce unplanned extubations (UPEs) in the NICU. The intervention began with a training session including 50 min of instructional videos and skills stations followed by a 1-hour hands-on skills check-off. After training, a short knowledge quiz was administered along with the SQuIPS. Participation in the training and testing was mandatory; responding to the survey was optional.
Project 2 sought to reduce surgical site infections (SSIs) after colorectal surgeries by improving maintenance of normothermia in the operating room. Interventions included new warming techniques, reinforcement of current procedures and an improved data collection system to track maintenance of normothermia. These changes impacted the equipment and personnel across the preoperative, perioperative and postoperative settings.
Project 3 sought to reduce specimen handling errors in the gastroenterology procedure suite. The project consisted of a retrospective data review to categorise errors, process map of the specimen handling process, and an anonymous survey of stakeholders’ opinions on the current process. Then the team implemented specific interventions targeting each objective error type, such as standardised labelling and a debriefing and time out after each procedure but before specimens were sent to pathology.
Survey administration
We administered the survey using Qualtrics to all clinicians of various professions who were stakeholders in the intervention and knowledgeable about each QI team’s intervention. We considered stakeholders to be anyone who might be affected by the QI project, as determined by the leader and their team. Stakeholders from each project took the survey once. Project 1 stakeholders took the survey prior to implementation of the intervention, and projects 2 and 3 stakeholders took the survey after the interventions had been in place for approximately 3 months. The survey was sent via email or completed on a device (phone, tablet, desktop) during a team meeting, and it was anonymous and optional for all stakeholders.
Analysis
One goal of our study was to understand the psychometric strength of the Likert-type items we created to measure QI stakeholder perspectives. Therefore, we examined how items loaded into related factors via an exploratory factor analysis. This analysis required that all items are on the same response scale. In our case, 9 of our 12 items based on NPT components (4 for coherence, 3 for cognitive participation and 2 for collective action) were on the same Likert-type response scale, so we submitted those 9 items for all projects to an exploratory factor analysis to see whether one or more factors best depicted the data.
For the two survey items that were free-text responses, we performed content analysis. In Project 1, the number of eligible stakeholders (N=298) and response rate (84% and 87% for each item) was sufficiently large to allow analysis of free text. In contrast, projects 2 and 3 had a much smaller number of stakeholder responses (N=8, N=20, respectively) and few free-text responses. Project 1 was so much larger because the intervention required training of all bedside nurses and respiratory therapists in a very large NICU, while projects 2 and 3 involved much smaller clinical areas (fewer numbers of beds, patients, and stakeholders). Analysis of the free-text responses for projects 2 and 3 is therefore not included in the analysis or discussion. Two of the authors (EF and ES) were responsible for coding the free-text responses for Project 1. First, these authors coded a small sample of responses together to establish preliminary codes (eg, ‘not enough nursing staff’ and ‘need more nurses to reach goal’ were both coded as ‘insufficient staff’). These authors each coded half the dataset and then reviewed each other’s responses to unify categorisation. Codes with the lowest frequency (less than 3%) were collapsed into larger codes (eg, ‘resistance of caregivers,’ ‘if people are unwilling to accept change,’ and ‘people not willing to change practice’ were all collapsed into the unified response ‘resistance to change’).
For survey questions with Likert-type scale responses we report simple frequencies for each response option.
Results
Reducing UPEs (project 1)
Response rates were good: 259 out of 272 registered nurses (95%), and 39 out of 75 respiratory therapists (52%). The overall response rate was 86%. The vast majority of respondents either slightly agreed or strongly agreed to all survey items, indicating an overall favourable impression of the UPE intervention (table 2). Agreement was slightly lower for the items asking if the intervention would make work easier and if it was compatible with current standard operating procedures.
Stakeholder Quality Improvement Perspectives Survey responses for each of the three QI projects, for the 9 Likert-type response items
Coding of the open-ended questions for the UPE respondents found that approximately two-thirds of stakeholders identified at least one barrier that would impact the intervention (table 3). ‘Insufficient staffing’ and expected ‘lack of compliance’ with the new policies and interventions were most often noted. The second open-ended question asked how the intervention would affect the respondent’s work (table 4). Over two-thirds of the responses identified one of four items: improved care/outcomes (27%), slower workflow due to more steps to follow (16%), improved safety (13%) and interruption of existing workflow (10%). Of note, improved care/outcomes included ancillary benefits of the intervention beyond safety.
Unplanned extubation project stakeholder responses to question ‘what barriers do you believe will impact intervention success?’
Unplanned extubation team responses to the open ended question ‘How will the intervention affect your work?
Fourteen stakeholders were identified to receive the survey and eight responded (three MDs, four RNs and one CRNA; 57% response rate). The group of stakeholders for this project was much smaller and they had greater variation in responses than project 1 (table 2). Two respondents (25%) slightly disagreed that there was consensus about the purpose of the intervention; 1 (13%) respondent did not see the point of the intervention (strongly disagreed) and 4 (50%) slightly disagreed that the intervention made work easier.
Reducing specimen handling errors in the gastrointestinal lab (project 3)
Twenty-five stakeholders were identified, 20 attended a meeting where the survey was administered, and all 20 responded (80% response rate). Twelve of 20 respondents indicated their professional role (1 MD, 8 RNs, 3 technicians). Two respondents (10%) disagreed (one slightly and one strongly) that there was consensus about the purpose of the intervention, 1 (5%) slightly disagreed that it made work easier, and 1 (5%) slightly disagreed that it was compatible with standard operating procedures (table 2).
Factor analysis
The exploratory factor analysis examined factor loadings for the 9 Likert-type items we created to measure coherence, cognitive participation and collective action. The results yielded one eigenvalue at 6.55, with the next highest eigenvalue at 0.80, which provides support for only one factor best depicting the items we examined. Additional support for this interpretation comes from the factor loadings, with the first seven items (corresponding to items we focused on coherence and cognitive participation) loading together, with loadings ranging from 0.81 to 0.93. The last two items we created based on collective action (The new practices/procedures will make my work easier and The intervention and new practices/procedures are compatible with existing work practices) cross-loaded with the first factor and the second factor, meaning that they were not empirically similar nor divergent enough from the first factor to warrant a second factor (this is also supported by the second eigenvalue being below 1.0). Cronbach’s alpha for the one factor solution with our first seven items is 0.97, which suggests a high level of intercorrelation between all items in the factor.
There were two items that did not have Likert-type scale response options (tables 5 and 6). The first asked stakeholders who would benefit from the intervention (patients, providers, families, the clinical area of the QI project and the hospital). The stakeholders from all three projects overwhelmingly said patients would benefit. Almost all stakeholders also thought providers would benefit from the UPE and gastrointestinal (GI) lab specimen handling projects (projects 1 and 3), but only 50% of the SSI project stakeholders thought providers would benefit (project 2). In addition, fewer of this project’s stakeholders thought benefits would be seen by the families, the clinical area or the hospital. The last item asked if the intervention would impact the division of labour, resources or responsibility between different professional groups. The UPE project stakeholders unanimously thought the intervention would impact division of labour (75%), resources (73%) and responsibility (84%) between themselves and other professional groups. The other projects’ stakeholders did not have such high percentages expecting such impacts, except that 85% of the GI specimen handling stakeholders thought it would impact division of responsibility.
Stakeholder Quality Improvement Perspectives survey responses to the question ‘The intervention will benefit the following (check all that apply): patients, providers, parents/families, my unit or clinical area, the hospital
Stakeholder Quality Improvement Perspectives Survey responses to the question ‘The intervention will impact the division of (check all that apply): labour, resources, and responsibility
All three QI project leaders said the survey results gave them new insights into the perspectives of stakeholders. Two of the three made changes to the interventions based on the results. The project 2 leader (maintaining intraoperative normothermia) noted that 50% of respondents said the intervention did not make their job easier and this made her realise ‘that it (the interventions) didn’t make anyone’s job easier and it helped me understand why implementing an intervention can be met with resistance. So in that sense, I did use the info to continue to help educate on its importance and with time the interventions are sticking and becoming more widespread.’ For project 3 (reducing specimen handling errors in the GI lab) one of the interventions was a postprocedure huddle and debriefing before specimens were sent to the laboratory. The survey results, especially lack of perceived consensus about the intervention, prompted the leader to reassess stakeholder understanding about the purpose and method of doing the huddle and debriefing. She recognised that the initial training on debriefing was ‘just implemented by word of mouth/emails. [I] realized there was no standardization and everyone was just doing random stuff. So we implemented the in person training and standardized the debriefing. This improved compliance and worked.’
Discussion
Statement of principal findings
After surveying stakeholders of three QI teams, we found that the SQuIPS was feasible for QI teams to use, resulted in good response rates, was sensitive to differences among QI projects, and that leaders used the results to alter their QI interventions. The QI teams found that the survey confirmed existing concerns (eg, the intervention would not make work easier) and revealed unforeseen concerns such as lack of consensus about the overall purpose of the intervention and its importance. The results of our factor analysis indicate that one seven-item scale can efficiently measure important aspects of stakeholder perceptions. In addition, we recommend retaining two additional Likert-type items (makes work easier and compatible with existing work practices). Even though they did not load on a factor, QI team leaders may use these two items to detect important and addressable stakeholder concerns.
In addition to results from the Likert-type items that QI project leaders used to make changes in the intervention, future QI project leaders may find the multiple-choice questions useful as well. A question about who would benefit can help leaders understand how stakeholders perceive the impacts of an intervention, and a question about division of labour, resources and responsibility can highlight whether additional attention should be paid to those important issues. For example, the project 1 (UPE) team leaders should expect major impacts on division of labour, resources and responsibility.
Strengths and limitations
We surveyed three very different QI teams (different clinical areas, number of stakeholders, types of interventions), which increases the generalisability of our findings to multiple QI projects and settings. Responses from the different teams varied, indicating that the survey is sensitive to differences among QI interventions and stakeholders. A limitation is that, as currently worded, some items may have a ceiling effect in which most responses tend to be very positive. These findings might also reflect accurately the perception of stakeholders, as it is possible that they viewed the QI interventions very positively. Other limitations include the small sample of stakeholders for two of the teams, and given that this is first administration of this new survey, the results need to be replicated and the survey may need to be refined.
Interpretation within the context of the wider literature
The NPT creators developed a 23-item survey in 2015 to measure all NPT components.6 We developed our survey based on NPT,5 but we did not work from their survey, and most importantly we did not seek to create an NPT survey that measured all of NPTs components. Instead, our goal was to start with NPT as the conceptual foundation, and then create a stakeholder perspectives survey for leaders of typical QI projects. This led us to create a shorter survey (11 items plus two open ended questions), and we tried to create items that would yield actionable information. To our knowledge, there are no published studies of similar stakeholders surveys.
Implications for policy, practice and research
Our survey can be important for the practice of QI because QI leaders can use survey results to alter interventions to improve the likelihood of adoption. For example, survey results may indicate that the intervention makes stakeholders’ work harder, or is incompatible with current work processes. The survey can also detect lack of consensus about the importance or overall purpose of an intervention. The open-ended questions can be used to identify stakeholders’ opinions about which barriers will impact the intervention’s success and how the intervention will impact their work. Future research should test the survey with other QI teams in other settings to see if the factor structure remains the same, and to confirm the feasibility and usefulness of the survey.
Another important question is when to administer the survey to QI project stakeholders, especially when QI interventions often change over time as they are refined by plan–do–study–act (PDSA) cycles. For simple interventions that require few resources and have little or no potential for unintended consequences, the survey could be administered after stakeholders have experience with the intervention. Then their responses will be based on actual experience. For more resource-intensive interventions with broad impact and possible unintended consequences, it would be best to administer the survey before any implementation, or perhaps after a small pilot or PDSA cycle.
Conclusions
Our NPT-based stakeholder survey measures important concerns that stakeholders have about QI interventions. If future research confirms our findings the survey can be an important tool for stakeholder analysis that may increase the chances of success for QI projects in healthcare.
Data availability statement
Data are available on reasonable request. Data available on request from corresponding author.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by The University of Texas Health Science Center Houston human subjects committee approved the work as quality improvement.
Footnotes
Contributors EF, ES, JE, AT-C, EJT and MO contributed to the development of the survey tool, and all authors approved the final version. EF, MO, RP and EJT facilitated the distribution of the survey for Project 1; EF, TS and EJT facilitated the distribution of the survey for Project 2; and EF, TM and EJT facilitated the distribution of the survey for Project 3. EF, ES, JE, AT-C and EJT performed data analysis. All authors contributed to the writing of the manuscript and approved the final version. EJT is guarantor and accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding This work was supported by grant # 5 P30 HS024459-03 from the Agency for Healthcare Research and Quality.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.