Background Despite significant attention to safety and quality in healthcare over two decades, patient harm in hospitals remains a challenge. There is now growing emphasis on continuous quality improvement, with approaches that engage front-line staff. Our objective was to determine whether a novel approach to reviewing routine clinical practice through structured conversations—map-enabled experiential review—could improve engagement of front-line staff in quality improvement activities and drive improvements in indicators of patient harm.
Methods Once a week over a 10-month period, front-line staff were engaged in 35 min team-based conversations about routine practices relating to five national safety standards. Structure for the conversations was provided by interactive graphical logic maps representing each standard. Staff awareness of—and attitudes to—quality improvement, as well as their perceptions of the intervention and its impact, were canvassed through surveys. The impact of the intervention on measures of patient safety was determined through analysis of selected incident data reported in the hospital’s risk management system.
Results The map-enabled experiential review approach was well received by staff, who reported increased awareness and understanding of national standards and related hospital policies and protocols, as well as increased interest in quality issues and improvement. The data also indicate an improvement in quality and safety in the two participating units, with a 34% statistically significant decrease in the recorded incident rates of the participating units relative to the rest of the hospital for a set of independently recorded incidents relating to patient identification.
Discussion This exploratory study provided promising initial results on the feasibility and effectiveness of map-enabled experiential review as a quality improvement approach in an acute clinical setting.
- quality improvement
- continuous quality improvement
- process mapping
- healthcare quality improvement
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- quality improvement
- continuous quality improvement
- process mapping
- healthcare quality improvement
Patient harm in hospitals is an ongoing worldwide issue, despite significant emphasis on safety and quality in healthcare over the past two decades.1–3 In Australia, one in nine patients will experience a complication during a hospital stay.1
To address this problem, in 2011 the Australian Commission on Safety and Quality in Health Care introduced mandatory National Safety and Quality in Health Service (NSQHS) standards against which hospitals are periodically accredited, together with indicators for benchmarking performance. The standards and indicators have generally been well received by health services.4 However, there are now many calls for a shift away from compliance-driven regimes towards concepts of continuous quality improvement (CQI).1 5–9 This includes: a focus on processes rather than on individuals; engaging and valuing the contribution of all staff; adopting team-based, systematic and ongoing approaches and promoting a culture in which quality is everyone’s business.5 8 10–12
One approach that aligns well with CQI principles is structured conversations involving front-line clinical staff. Structured conversations have been used in healthcare for some time, as a method for informing and debriefing staff (eg, huddling13) and for team-based training (eg, in situ simulation14). Increasingly, structured conversations are being seen as a CQI tool because they provide a mechanism to engage front-line staff through team-based reflections on clinical processes. The potential of debriefing to change team behaviours and improve clinical outcomes15 has resulted in the development of various tools to provide structure to debriefing sessions.16 17 In situ simulation enables identification of latent safety threats through immersion, practice, reflection and feedback,14 18 19 although the driver for the structured conversation is a simulated scenario, rather than an actual clinical event.
While debriefing tools and in situ simulation are highly effective, they tend to be focussed on specific clinical cases or events. There is growing interest in strategies to tap into team knowledge to examine routine clinical practices more broadly, although there are few descriptions of successful tools that systematically capture staff knowledge to improve clinical processes.20 One recent example, HEADS-UP, showed promising results in terms of improved safety and teamwork attitudes, although the structured conversations were primarily focussed on problems and issues from the previous day.20
Recently, a novel approach to structured conversations has been described that encourages review of all routine practices, not just those relating to a particular case or simulated scenario, or where a problem or adverse event recently occurred. The approach, termed map-enabled experiential review (MEER),21 uses diagrammatic process models to facilitate structured conversations among front-line staff about their routine practices. MEER has already been used successfully in health service settings in the context of a statewide quality improvement initiative in clinical education and training.22 Quality managers that participated in that initiative saw the potential for the same approach to be applied to improving quality and safety in a clinical context.
This study set out to determine whether the MEER approach increases engagement of front-line clinical staff in quality improvement activities and whether regular conduct of MEER sessions that address routine clinical practice leads to improvements in indicators of patient harm.
This exploratory pre-intervention/post-intervention study was conducted at the Epworth HealthCare Richmond hospital campus, a 600-bed not-for-profit private hospital in metropolitan Melbourne. Two clinical units participated in the study, the emergency department (ED) and an inpatient oncology ward (4Gray). The 35-bed ED employs roughly 80 nursing staff and has approximately 29 000 attendances per year; 4Gray is a 40-bed unit that employs about 60 nursing staff with a bed occupancy of ~13 000 patient bed days per year.
All nursing staff in ED and 4Gray, as well as ED medical staff and clerical staff, were invited to participate in the study. Participants were allocated a Personal ID Number (PIN) for use in the online surveys. Other staff, including pharmacy, allied health and support staff, as well as middle and senior managers, were encouraged to take part in the discussion sessions, although they were not allocated a PIN or invited to complete the surveys.
The intervention involved application of the MEER approach in each participating clinical unit once a week during the afternoon handover period. On the designated day, staff from the morning shift in ED or 4Gray met for ~35 min for a team-based, structured conversation about their routine practices relating to five specific NSQHS standards. The structure for the conversations was provided by graphical process logic maps that show the inputs, activities, outputs and outcomes through which the objectives of a particular standard are achieved. Further information on the MEER approach, including one of the process models used in this study, is provided in online supplementary file 1. Discussions were conducted in each unit’s meeting room with the maps projected onto a wall visible to participants.
Discussion sessions involved rating each item in the map—termed nodes—in a logical sequential order, with team members initially polled for their individual rating based on their own knowledge and experience, and then the team arriving at a consensus rating. Nodes identified as problematic were nominated for inclusion in an improvement action plan and tasks to address the issues were identified and assigned to individuals or designated teams. Discussion sessions were facilitated by individual team members and observed by the study investigators.
Sessions were conducted using the online application MEERQAT (https://meerqat.com.au/), which includes maps corresponding to the NSQHS Standards (see online supplementary file 1). This tool allows the session facilitator to capture team member ratings and comments directly into an interactive version of each map and to graphically display the group’s consensus ratings. The MEERQAT application also links the node rating step (termed an assessment) directly into a Kanban-style action planning tool. The application requires little training to use, which allowed session facilitation responsibilities to be shared among participating staff.
A total of five NSQHS Standards (Edition 1) were reviewed: Standard 3 (Preventing and controlling healthcare associated infections), Standard 4 (Medication safety), Standard 5 (Patient identification and procedure matching), Standard 6 (Clinical handover) and Standard 10 (Preventing falls and harm from falls). The maps for these standards range in size from 42 to 51 rateable nodes and each map was completed over one calendar month (ie, four or five 35 min sessions), with 8 to 12 nodes rated in each 35 min session. The five standards were assessed sequentially over five consecutive months and then repeated in the same order over the next 5 months, over the period January 2018 to October 2018.
Online surveys that canvassed participants’ knowledge and perceptions of quality and safety were created using SurveyGizmo (https://www.surveygizmo.com/). The pre-intervention version (administered in the 2 weeks immediately before the sessions commenced) collected baseline data, while the post-intervention version (administered at the 2.5, 5 and 10-month time points) included additional questions on participant perceptions about the intervention and its impact. Information about the reliability of the survey instrument is presented in online supplementary file 2. Participants were required to enter their assigned PIN to complete the survey, enabling alignment of de-identified responses from individuals over the course of the study. Surveys remained open for 2 to 4 weeks.
Analysis of adverse incident data
A primary hypothesis of this study was that the MEER intervention would improve the quality and safety of clinical care delivered in the participating units. We sought to test this hypothesis by analysing adverse incident counts logged in the hospital’s incident management system (IMS; RiskMan; http://www.riskman.net.au) during the MEER intervention (January 2018 to October 2018) compared with the same period the previous year. Incident records contain descriptive fields about the incident, including the relevant NSQHS standard(s).
Analysis of adverse incident report counts is potentially problematic since the number of these reports recorded in the IMS reflects reporting behaviours of staff as much as—if not more than—it reflects the number of incidents that actually occurred.23 Thus, an improvement intervention might impact on clinical practice, but it also might impact on reporting behaviours and the relative contribution of both impacts is difficult to measure.
Incidents for standards 3, 4, 6 and 10 were generally reported by staff who work in the unit where the incident occurred. We therefore expected interpretation of data for those incidents would be confounded by potential impacts of the MEER intervention on reporting behaviours of staff. For instance, incident reporting could be increased by a greater awareness of the requirements of the standards imparted through the MEER sessions, or conversely reduced by subject expectancy effects.24
Incidents relating to Standard 5 differ in that 95% of these incidents in ED and 4Gray were logged by staff who work in centralised service units such as hospital records, radiology and pathology and who log such incidents against units across the whole hospital. Standard 5 incidents include: missing or incorrect patient information; incorrect label use; unlabelled patient forms, samples, referrals and test requests; use of wrong patient identification (ID) and mismatched patient IDs (see online supplementary file 1, appendix S1.1). Importantly, the individuals that logged Standard 5 incidents were unaware of, and hence their reporting behaviours were uninfluenced by, the MEER intervention. They also would have applied consistent reporting behaviours to Standard 5-related incidents across all units of the hospital. Therefore, this data was deemed suitable for analysis, as it permits a change in clinical practice to be observed without any such change (potentially) being obscured by changes in reporting behaviour.
Standard 5 incident records, which identified the hospital unit where the incident occurred as well as the incident date, were grouped by month. Using monthly patient activity data for each unit, monthly incident rates (monthly incidents per 1000 patients or equivalent ED attendances) were calculated. The data was partitioned between incidents attributed to the intervention units (ED+4Gray) and those for the rest of the hospital (Other). Monthly incident rate ratios were obtained by dividing the combined ED+4Gray incident rates by the combined incident rate for the Other hospital units.
Patient and public involvement
Patients and members of the public were not involved in this study.
MEER sessions were predominantly attended by nursing staff. In total, 65 ED nurses and 46 4Gray nurses attended the sessions (81% and 79% of the nursing rosters, respectively). Nurse Unit Managers (NUMs) were frequent participants, while most sessions were attended by at least one Associate NUM. The ED Medical Director attended 25% of ED sessions and several members of the hospital’s senior management team attended two to four sessions each. Sessions were also attended infrequently by ED medical staff, clerical staff, orderlies and allied health staff. Further analysis of staff participation in MEER sessions and surveys is presented in online supplementary file 4.
Conduct of MEER-related activities
All weekly MEER sessions were attended by at least one of the investigators, which allowed all aspects of the intervention to be observed.
While the MEER approach was initially new to participants, they readily grasped the concept and quickly became proficient in using the online tool. From the first session, staff engaged collegiately in the process and all staff contributed to the discussions. The groups developed their own approach for reaching consensus about the overall node rating and whether to include the node in the action plan. Nodes that were rated poorly were usually included, but nodes that were rated well were also sometimes included if further improvements had been identified during the discussion. Nearly half of all tasks added to action plans during the intervention were completed by the end of the project.
The occasional attendance of middle and senior managers in the sessions appeared to add value to the discussions. Senior staff were able to clarify issues raised by front-line staff and provide a higher level perspective on policies and practices. Front-line staff commented that the presence of senior managers confirmed the importance of quality-related activities and allowed senior staff to hear directly from front-line staff about issues of importance to them. Similarly, the inclusion of other healthcare workers for particular discussions—such as pharmacists for Standard 4 and physiotherapists for Standard 10—was well received by regular participants.
Findings from the staff surveys
Seventy-five per cent of staff that had been assigned PINs responded to at least one of the four surveys conducted over the course of the project. Response rates for individual surveys varied between 45% to 59% of staff with a PIN who had attended at least one MEER session (see online supplementary file 4).
Table 1 reveals that, before the project commenced, the majority of staff were familiar with the detail and understood the relevance to their work of the five standards included in the project (row A). A similar majority of respondents rated the implementation of those five standards in their unit as good or very good (row C). When baseline survey respondents were asked the same question in post-intervention surveys, there was a considerable increase (8 to 15 percentage points per standard) in the proportion of respondents that indicated they were familiar with the detail and understood the relevance to their work of these standards (row B). A two-way repeated ordinal regression analysis on the ordered Likert scores26 indicated a highly significant change in the Likert score with time (p=0.0013) and non-significant changes between standards (p=0.23) (see online supplementary file 5).
A larger increase (19 to 32 percentage points per standard) was observed in the proportion of respondents that rated the implementation of each standard in their unit as good or very good (row D). Once again, there was a highly significant change in rating scores with time (p=2.9×10-6) and non-significant differences between standards (p=0.61). Consistent with this, by the end of the intervention, for each standard included in the project, 75% to 94% of respondents believed that implementation of the standard in their unit had improved over the course of the project (row E). This was consistent with the results of the MEER assessments: for most of the standards, the proportion of nodes with an above-average consensus rating increased between the two assessments (rows F and G).
The survey data also corroborated the investigators’ observations in relation to how well the MEER approach was received by staff. Table 2 reveals that a substantial majority of survey respondents enjoyed the MEER approach, responding favourably about both the team-based (row A) and the map-based (row B) approach to reviewing their activities. Respondents enjoyed the opportunity to reflect on their own practice (row C), felt comfortable expressing their own opinions (row D) and saw merit in hearing the perspectives of their colleagues (row E). The high level of positive responses was recorded in all post-intervention surveys. While some differences in the relative levels of ‘positiveness’ could be seen between the respondent groups for the three post-intervention surveys, no significant differences were seen between participants who only answered the first post-intervention survey and those who answered multiple surveys (see online supplementary file 4).
Respondents also saw the experience as an educational one, learning new information about both the NSQHS standards (row F) and their hospital’s policies and protocols (row G). The proportion of respondents that indicated they had learnt something through their participation continued to increase throughout the course of the intervention.
Respondents reported moderately high levels of involvement in the quality improvement activities arising from the action plans (row H), in contrast to the proportion that indicated they had been involved in any quality improvement activities in the 12 months prior to commencement of this project (row H, results shown in parentheses). At the same time, a substantial proportion of respondents indicated their level of interest in quality issues and quality improvement had somewhat or greatly increased since they commenced involvement in the MEER sessions (row I).
Staff were also asked about other impacts of the intervention. At the 2.5 month time point, two-thirds of respondents indicated that staff in their unit were generally more aware of quality (row J) and there were more informal discussions about the quality standards (row K), while 41% indicated there had been some notable improvements in practice in their unit (row L). By the end of the intervention, these proportions had increased substantially.
Analysis of Standard 5 incident data
A total of 1810 RiskMan records relating to Standard 5 incidents were identified across all hospital units that occurred during the intervention period (January 2018 to October 2018) and the comparable pre-intervention period (January 2017 to October 2017).
Figure 1 compares the incident rate for Standard 5 in the combined ED+4Gray units versus all other units in the hospital. The process control u-chart27 in panel A indicates a −33.3% reduction in the incident rate for ED+4Gray from the 2017 baseline period to the 2018 MEER intervention, whereas the Other units experienced a +2.7% increase (panel B).
To gauge the statistical significance of these changes, the data was tested as an interrupted time series (ITS).28 Segmented regression analyses were conducted on the incident count data for the combined ED+4Gray units using the generalised linear model function (glm) in the R statistical programming language.25 Due to moderate overdispersion in the data, a negative binomial modelling formula was applied with the default logarithmic link function.29 The glm model incorporated a term for level change between the 2017 baseline year and the 2018 MEER intervention year, and a term for seasonality effects. It also included a log offset for patient activity level in the ED+4Gray intervention units, as well as for the incident rate for all other units to account for other hospital-wide influences (see online supplementary file 3).
Panel C in figure 1 displays an ITS plot of the incident rate ratio of ED+4Gray versus other Epworth units for the recorded data and the model curve predicted by the segmented regression analysis. The level change between the 2017 baseline year and the 2018 MEER intervention was highly significant (p=0.00056) with a relative risk (RR) of 0.66 (95% CI 0.52 to 0.84), representing a 34% reduction in the ED+4Gray recorded incident rate for Standard 5 relative to the rest of the hospital.
This exploratory study set out to determine the feasibility and effectiveness of the MEER approach as a quality improvement tool in an acute clinical setting.
Data collected through observations and surveys revealed that the MEER approach was well received by staff. The interactive, graphical tool was an effective way to both impart and collect information and staff found the maps to be useful in relating their routine practices to the principles set out in each NSQHS standard. Staff valued the opportunity to contribute their own knowledge and insights to the discussions and the direct linkage between assessment and action planning in the online tool ensured that conversations about improving practice were immediately translated into action plan tasks. Thus, the weekly discussions were productive and enabled collective ownership of both issue diagnosis and issue resolution.
By their own assessment, staff reported increased interest in quality issues and quality improvement through their participation in MEER sessions. This was also evidenced by a substantial increase in involvement in quality improvement activities despite no additional time being allocated for such involvement.
The data also indicate that use of the MEER approach was associated with improvements in the two participating units, with three pieces of evidence supporting this conclusion. First, the survey responses revealed significant increases in the proportion of respondents that perceived staff in their unit were generally more aware of quality, more informal discussions about quality and notable improvements in practice. Second, the MEER assessments revealed an increase in the proportion of nodes rated above-average between the first and the second assessment. Third, analysis of the Standard 5 adverse incident data revealed a highly statistically significant decrease of 34% in the recorded incident rates of the two participating units relative to the rest of the hospital.
Therefore, the results of this study indicate that the MEER approach to regular, structured conversations with front-line clinical staff about their routine practices is an effective tool for engaging staff in quality improvement over an extended period of time and is associated with improvements in a key indicator of patient safety.
This exploratory study did not seek to identify the mechanisms through which improvements were achieved although, like other techniques based on structured conversations, MEER probably works through a range of direct and indirect mechanisms. Direct mechanisms reflect the content of the structured conversations, while indirect mechanisms reflect the nature of the conversations and the attributes those conversations help to develop in participants.
The most obvious direct mechanism is the implementation of action plan tasks to address issues identified during the discussions (see online supplementary file 1). Another direct mechanism is education: the discussions informed or reminded staff about best practice and hospital policies/protocols and they used this information to improve their own practice. They also reported passing their learnings onto other team members who were not present during the discussions, thereby amplifying the educational impact.
Indirect mechanisms include enhanced reflective practice, with the regular conversations encouraging a more mindful approach to routine activities, and the development of collective competence,30 whereby team members develop a shared understanding of the activities they all contribute to.
MEER in the context of other evidence
The promising results obtained in this exploratory study are consistent with current theories relating to quality improvement. For example, there is growing recognition that the knowledge and experience of front-line staff is a valuable resource for quality improvement that is often overlooked.6 10–12 31–36 Not only can front-line staff provide insights about ‘work-as-done’,6 37 but engaging staff in issue diagnosis and development of solutions is important to embedding the solutions as sustained changes to practice.9 10 33 35 37 38
Moreover, structured conversations10 13 14 34 36 that encourage both individual and team-based reflection6 18 31 39 are among the most effective mechanisms for tapping into staff knowledge and experience. With MEER, the structure for conversations is provided by the map, with the node rating questions providing an initial focus for discussion (see online supplementary file 1, appendices S1.2 and S1.3 for an example). The process of rating each node is inherently reflective and by conducting the exercise as a group, there is a greater likelihood of effectively reflecting on practice, rather than simply rationalising practice.40
MEER is also consistent with current thinking on the need for approaches that emphasise continuous improvement, rather than a focus on compliance.6 8 31 37 41 42 The MEER sessions were conducted on a regular weekly basis and while audit results or RiskMan incident counts were often referred to during discussions, these data were not the driver for the discussion. This proved beneficial in three ways. First, since the starting point for the conversation was ‘what we routinely do’ rather than ‘what went wrong’, staff didn’t feel defensive about reflecting on their practice. Second, issues were identified before patient harm actually occurred. Third, the sessions provided a mechanism for identifying positive deviance,9 43 as well as latent safety threats.
Feasibility and sustainability
No significant operational issues were encountered in either participating unit. Team discussions were held in existing facilities and used commonly available information technology resources. Sessions were conducted during a handover period when a scheduled staff overlap occurs and were conducted every week over a 10-month period, demonstrating the long-term sustainability of the approach.
The limited time and resources available for this study meant the intervention was conducted in only 2 of the hospital’s 25 clinical units, addressed only 5 of the 10 first Edition NSQHS Standards and was targeted primarily to nursing staff. As this was a research project, participation in MEER sessions and surveys was voluntary. Moreover, participation in MEER sessions was effectively limited to those staff rostered on the morning shift of the day of the session, thereby restricting the survey respondents to those staff who were able and willing to attend the MEER sessions and willing to complete surveys. While other clinical and non-clinical staff occasionally attended MEER sessions, their participation in the surveys was not sought. These facts, combined with a response rate for each survey below 60%, reduced the power of the survey data analysis and limits the possible conclusions.
For example, it is possible the staff that attended MEER sessions and completed surveys represent a particularly keen and engaged subsection of staff who might be more inclined to rate the intervention positively. Further analysis of staff attitudes to the approach will be needed in circumstances where MEER sessions are part of regular hospital practice and all available staff are expected to attend sessions and provide feedback.
The small scale of the intervention may also have limited potential benefits. The study was conducted with the knowledge of the Quality Manager and various senior managers but with no additional support or time allocation, which limited the number of action plan tasks that could be completed. Issues that required an organisation-level intervention were not addressed.
As this was an exploratory study to determine whether the MEER approach could impact positively on indicators of patient harm, no attempt was made to determine the mechanisms responsible for any such impact. Therefore, it is not possible to definitively identify which particular aspects of clinical practice were improved, or the relative contribution of direct versus indirect mechanisms through which the MEER approach is thought to impact outcomes.
Further studies will compare MEER to other structured conversation-based strategies and seek to determine the extent to which implementation of the action plan contributes to positive impacts of the intervention. Other work will investigate organisation-wide implementation of the MEER approach.
This exploratory study has provided promising initial results on the feasibility and effectiveness of map-enabled experiential review as a quality improvement approach in an acute clinical setting. MEER represents a novel, team-based approach to utilising the knowledge and experience of front-line staff that is engaging, educative and engenders ownership of quality concepts and activities for participants.
We wish to thank Professor Nikolajs Zeps for his advice on project design and ethical review and Associate Professor Dean McKenzie for advice on statistical analyses. We particularly wish to thank the frontline staff of Epworth Richmond that participated in the project and the Nurse Unit Managers of the participating units that supported the conduct of the project in their area. We also wish to thank Jane Lynch for championing this project.
Contributors AC: contributed to initial conceptualisation and study design; collected observations at almost all intervention sessions; contributed to data analysis and interpretation, as well as drafting and review of the article. VA: contributed to initial study design; contributed to data analysis and performed all statistical tests; prepared figures for publication; contributed to drafting and review of the article. FB: contributed to initial study design; provided support and advice on project conduct during the intervention; reviewed data and data analysis; reviewed the article. DRC: provided initial conceptualisation of the study and study design; collected observations at some intervention sessions; contributed to data analysis and interpretation; prepared tables for publication; drafted the article.
Funding AC, VA and FB received a grant from the Epworth Research Institute (ERI) to cover costs associated with the conduct of the project. ERI had no involvement in study design; collection, analysis and interpretation of the data or preparation and submission of this article.
Competing interests AC and VA have a financial interest in MEERQAT Pty Ltd; DRC is a Director of MEERQAT Pty Ltd.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Patient consent for publication Not required.
Ethics approval The project was approved by the Epworth Human Research Ethics Committee (Protocol # EH 2017-261). All participants provided informed consent prior to taking part in the study.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. Survey data: De-identified individual participant data that underlie the results reported in this article, as well as the statistical analysis plan, will be available immediately following publication for any researchers whose proposed use of the data has been approved by an independent review committee. RiskMan data: The incident report data used in this study are not publicly available; researchers wishing to access de-identified incident report data or tabulated incident count data will need to submit a proposal to Epworth Hospital.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.