Article Text
PDF
Abstract
Objectives To profile the aims and characteristics of quality improvement (QI) initiatives conducted in Ireland, to review the quality of their reporting and to assess outcomes and costs.
Design Scoping review.
Data sources Systematic searches were conducted in PubMed, Web of Science, Embase, Google Scholar, Lenus and rian.ie. Two researchers independently screened abstracts (n=379) and separately reviewed 43 studies identified for inclusion using a 70-item critique tool. The tool was based on the Quality Improvement Minimum Quality Criteria Set (QI-MQCS), an appraisal instrument for QI intervention publications, and health economics reporting criteria. After reaching consensus, the final dataset was analysed using descriptive statistics. To support interpretations, findings were presented at a national stakeholder workshop.
Eligibility criteria QI studies implemented and evaluated in Ireland and published between January 2015 and April 2020.
Results The 43 studies represented various QI interventions. Most studies were peer-reviewed publications (n=37), conducted in hospitals (n=38). Studies mainly aimed to improve the ‘effectiveness’ (65%), ‘efficiency’ (53%), ‘timeliness’ (47%) and ‘safety’ (44%) of care. Fewer aimed to improve ‘patient-centredness’ (30%), ‘value for money’ (23%) or ‘staff well-being’ (9%). No study aimed to increase ‘equity’. Seventy per cent of studies described 14 of 16 QI-MQCS dimensions. Least often studies reported the ‘penetration/reach’ of an initiative and only 35% reported health outcomes. While 53% of studies expressed awareness of costs, only eight provided at least one quantifiable figure for costs or savings. No studies assessed the cost-effectiveness of the QI.
Conclusion Irish QI studies included in our review demonstrate varied aims and high reporting standards. Strategies are needed to support greater stimulation and dissemination of QI beyond the hospital sector and awareness of equity issues as QI work. Systematic measurement and reporting of costs and outcomes can be facilitated by integrating principles of health economics in QI education and guidelines.
- quality improvement
- quality measurement
- cost-effectiveness
- quality improvement methodologies
- evaluation methodology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- quality improvement
- quality measurement
- cost-effectiveness
- quality improvement methodologies
- evaluation methodology
Introduction
Quality improvement (QI) is an intrinsic part of healthcare and functions to support better patient experience and outcomes, better professional development and better system performance.1 Using clearly defined methodologies, the intention of QI is to make systematic, data-based, iterative improvements, to enhance healthcare delivery and outcomes.2 In 2001, the US Institute of Medicine (IOM) identified six goals of quality in healthcare: safety, timeliness, effectiveness, efficiency, equity and patient-centredness.3 Worldwide, organisations have followed these aspirations to chart QI plans. Due to the growing costs of healthcare globally, it is becoming increasingly obvious that the explicit aims of healthcare systems will no longer be to provide ‘quality’ exclusively, but to deliver ‘value’, that is quality relative to cost.4 To support the achievement of high-value healthcare, in 2008, the US Institute for Healthcare Improvement suggested health systems pursue the ‘Triple Aim of Healthcare’; to improve the individual experience of care, improve the health of populations and reduce the per capita costs of care for populations.5 In recognition of the foundational role of staff well-being in achieving these aims, the improvement of the experience of providing care was added in 2014, to advocate the ‘Quadruple Aim of Healthcare’.6
In Ireland and other countries, evidence is limited about the practice of QI and whether it supports better-value healthcare.7 Accordingly, only a fraction of QI projects implemented in practice are reported in peer-reviewed journals.8 There are some online QI reporting repositories to disseminate learning, however, variation in the classification of projects makes comparisons difficult.8 Furthermore, while the Standards for Quality Improvement Reporting Excellence guidelines advise to report costs,9 barriers to this include disparate and limited formal guidance for improvement teams on the measurement of costs associated with QI, and the accessibility of organisational and patient cost data.7 10–15
To support effective decision-making, robust information is required about the content and context of QI initiatives, the expected outcomes, initial costs of implementation and the subsequent impact on long-term costs for the health service.10 16 Accordingly, several international reviews have identified the scope of QI practice17 and good practice evaluation methods within different clinical areas.18–20 In Ireland, in recent years, QI has been strategically led and increasingly integrated in Irish health services.21 However to date, there has been no formal review of the characteristics and volume of Irish QI studies reported in the scientific literature. It is unclear how Irish QI studies align with recognised international quality goals3 and adhere to established reporting standards.
Therefore, this study aimed to map available reports of QI initiatives in Ireland and interpret their impact on patient experience, provider experience and health system performance. Specific objectives were to: (1) profile the aims and characteristics of QI initiatives conducted in Ireland, (2) review the quality of their methodological reporting and (3) assess the cost-effectiveness of the QI initiatives by comparing their outcomes and costs.
Methods
A scoping review was performed according to the Johanna Biggs Institute, Guidance for Conducting Scoping Reviews.22
Search strategy
We searched PubMed, Web of Science, Embase, Google Scholar and the two national databases Lenus and rian.ie for peer-reviewed articles and grey literature published from 1 January 2015 to 8 April 2020, using the search term “quality improvement”, with the addition of [“Irish” OR “Ireland”] for international databases. The search was adjusted slightly for each database, given the differences in how their search tools are constructed (see online supplemental file 1 for details). Full texts were searched for cross-references of Irish QI studies that had not been retrieved through the original searches. Reports of QI initiatives known to the study team were included to maximise reach.
Supplemental material
Study selection
All abstracts were reviewed independently by two researchers with a QI or health economic background. Inclusion criteria were that the study met the definition of QI (to support better patient experience/outcomes, professional development or system performance)1 and was implemented and evaluated in the healthcare sector in Ireland. After the individual review of abstracts, both researchers discussed their assessment and formed consensus on inclusion for full-text review.
Data abstraction and quality assessment
Two researchers independently documented the reported characteristics, outcomes and costs of each QI study and assessed the study reporting standards using a novel 70-item assessment tool. To construct our tool, we used the Quality Improvement Minimum Quality Criteria Set (QI-MQCS)23 as a basis. The QI-MQCS enables reviewers of QI studies to report whether 16 QI reporting standards have been ‘met’, ‘not met’ or ‘partially met’.23 We added measurement of the aims and characteristics of the QI studies. Namely, where appropriate, we added quantifiable study details to the QI-MQCS domains (9 of the 16). For example, we added each of the IOM goals of quality3 and the goals of ‘staff well-being’ and ‘value for money’ (binary ‘yes’/‘no’ items) to the QI-MQCS domain of ‘intervention description’ to characterise study aims similarly to other review studies.17 24 Furthermore, we incorporated basic tenets of health economic evaluation in our tool.10 We included items to assess reporting of various types of costs, the perspective (societal, healthcare services or public healthcare), costing approach (top-down, bottom-up or mixed), incremental analysis of cost and outcomes (cost-effectiveness and cost–utility analysis), discount rates of future costs/outcomes and the potential for sensitivity analysis. Finally, we added items assessing whether the QI had met its stated aims and had enhanced the patient experience, provider experience and system performance. For each study, we assessed whether it met each of the 16 QI-MQCS criteria and we recorded the additional quantifiable items detailed previously. No critical appraisal of methodological quality was conducted as this was not part of our study aims and is not standard for a scoping review.25 The tool was embedded in Microsoft Excel and was tested on 10 studies and extended following the initial use. An overview of the tool is provided in online supplemental file 2.
Supplemental material
Descriptive statistics were performed to profile the characteristics and reporting standard of QI initiatives. We held a stakeholder (n=40) engagement workshop in October 2020 to share and contextualise the findings with invited national and international QI leaders: policy-makers, practitioners, educators, health economists and patient and family representatives. This informed our discussion of the findings presented in this review.
Results
Data synthesis
Of 379 references retrieved, 275 remained after removing duplicates. Eighty references were identified for full review, 43 of which satisfied the specified inclusion criteria,26–66 including two unpublished reports known to the authors. Search results and reasons for exclusion of studies are detailed in figure 1. Of the 43 studies, the majority (n=37; 86%) were peer-reviewed journal articles and the remainder (n=6; 14%) formed grey literature. One in three journal articles featured in quality-themed journals and since January 2015, there is a trend towards an increasing number of studies published each year (see details in online supplemental file 3).
Supplemental material
Study selection process. QI, quality improvement.
Reported characteristics of QI studies
QI location
Most studies (n=33, 77%) reported a QI implemented in a single organisational site. Fewer were implemented across organisations (n=7, 16%) or at national or regional level (n=3, 7%). The majority of QIs were conducted in the hospital sector (n=38, 88%), mainly in acute hospitals (n=31, 72%) (see details in online supplemental file 4). Most (n=40, 93%) did not mention whether healthcare services were public, private or mixed public–private institutions. On investigation of the institutional names reported in studies, we identified 89% as public, 2% as private and 2% as mixed public–private institutions. For the remaining 7%, no information could be retrieved.
Supplemental material
QI aims and change ideas
The 43 QI studies are characterised by study aim, methodology and design in table 1 and an extended description of the aims of studies is provided in online supplemental file 5.
Supplemental material
Aim categorisation and key characteristics of QI studies
Most studies aimed to improve more than one domain of quality. Two in three studies aimed to improve the ‘effectiveness’ (65%) of care while approximately half aimed to improve ‘efficiency’ (53%), ‘timeliness’ (47%) and ‘safety’ (44%) (figure 2). Fewer aimed to improve ‘patient-centredness’ (30%), ‘value for money’ (23%) or ‘staff well-being’ (9%). No study aimed to increase ‘equity’ of care provision.
Number of quality improvement (QI) studies which aimed to enhance Institute of Medicine (IOM) quality goals+ and frequency of studies that indicated achievement of these goals.
The 43 studies also reported on a variety of themes for QI change ideas, with little overlap across studies. Noteworthy themes included testing the effect of technology,28 29 50 61 time to care,32 33 65 health surveillance,42 43 45 47 education48 49 58 59 and antimicrobial use30 44 55 interventions on healthcare quality.
QI methodology
Two-thirds of studies (65%, n=28) reported the use of an established QI method. Of these, approximately half (n=15) used the ‘Plan-Do-Study-Act Cycle’ (n=9) or ‘Model for Improvement’ (n=6), 12 used a form of Lean (‘Lean Six Sigma, Define Measure Analyse Improve Control’ (n=6); ‘Lean Six Sigma’ (n=2) or ‘Lean’ (n=4)) and one study used Action Research (table 1). While 15 studies did not report the use of a formal QI method, the authors, however, labelled these studies as ‘QI’ and reported the use of common QI practices. For example, the utilisation of quality tools to diagnose, measure and enhance quality.
QI study designs and data sources
Nearly all studies (97%, n=42) named the study design (table 1). The majority (n=26, 62%) were pre–post designs; studies that compared the same parameters before and after QI implementation. Of these, six (23%) also collected time series data to track iterative changes. A further one-quarter of studies, without establishing pre–post measures, collected time series data to track iterative change (n=11, 26%). The remaining studies collected post implementation data only (n=1, 2%) or used parallel control group designs (n=4, 10%). Furthermore, the majority of studies described the existing standard of care before implementation of the QI intervention (the ‘comparator’; n=36, 84%) and mechanisms for ‘fidelity and adherence’ (n=36, 84%), such as compliance with intervention components.
Most studies (n=28, 65%) used routine healthcare data (eg, patient records, prescriptions charts) solely (n=11) or in conjunction with other data sources (n=17). In 70% of studies (n=30), non-routine data were collected. Through use of surveys or interviews, approximately one in four studies (n=11) incorporated patients’ views, one in five (n=9) incorporated staff views and few incorporated relatives’ or carers’ views (n=3, 2%). Data collection mainly focused on care processes and one in three studies (n=15, 35%) reported on health outcomes.
QI study time frame
Study time frames varied across the 40 studies reporting this detail (table 1). The average total study duration was 16.8 months, the minimum 1 month and the maximum 48 months. Fewer studies (n=29, 67%) reported the duration of the QI implementation, which was 8.7 months on average, 1 week at a minimum and maximum 24 months.
Health outcomes
Fifteen studies (35%) reported patient health outcomes (28%), for example, pain or infection, or proxies for patient health outcomes (7%), for example, length of stay or hospital admission rate (table 1). Studies reported that health outcomes were positively affected by the QI. No study examined health outcomes for staff.
QI costs
As displayed in table 1, 23 studies (53%) discussed or alluded to costs associated with the QI initiative. Tangney and seven other studies28 32 37 51 56 61 63 (n=8, 19%), provided at least one quantifiable figure for a cost or cost saving. As shown in table 2, the 23 studies mostly considered ‘staff costs’ (57%), followed by ‘overhead costs’ (39%), ‘capital costs’ (35%) and ‘indirect healthcare costs’ (22%). A single study included ‘direct costs to the healthcare user’. Five studies (22%) did not break down the types of costs considered.
Frequency of the discussion of types of costs in quality improvement studies (n=23)
Number and percentage of studies that met reporting standards of the Quality Improvement Minimum Quality Criteria Set
An example of detailed cost data was provided in one study28 that compared the cost of staff and phone/texting charges before and after implementation, and mentioned two technology purchases for implementation; while a full cost analysis, details on the perspective (healthcare/public health/societal), costing approach (top-down or bottom-up) or incremental analysis was not provided. Considering all 43 studies, performing a full economic assessment is not possible based on the data reported.
Achievement of QI aims
Ninety-eight per cent of studies were interpreted by the researchers to have achieved their intended aims, either fully (70%) or partially (28%). These studies most frequently conveyed ‘effectiveness’ (64%), ‘efficiency’ (48%), ‘timeliness’ (38%) and ‘person-centredness’ (36%) as the elements of healthcare quality improved. Furthermore, one in four studies conveyed improved ‘safety’ (n=10) and ‘value for money’ (n=11). Few reported improvements to ‘staff well-being’ (2%) and the ‘equity’ (2%) domain (figure 2).
Sustainability and spread of QI initiatives
Over 90% of studies reported on the sustainability of the QI. Specifically, 88% of studies (n=38) reported evidence of enduring improvement and 60% (n=26) reported policy changes implemented or needed to support the change. The spread or the requirements for spread were discussed in 86% of studies (n=37).
Assessment of QI impact
Over 90% of studies were interpreted to have improved the system performance (91%). Less were interpreted to have improved the patient experience (n=28, 65%) and provider experience (n=20, 47%).
Assessment of the reporting standard of QI studies
Seventy per cent of studies (n=30) met the minimum standard for reporting 14 of 16 QI criteria (either fully or partially) as described by the QI-MQCS23 (table 3).
Studies least often reported the ‘penetration/reach’ (53%) of an initiative such as the number of units or sites participating in the intervention compared with those available or eligible and ‘health outcomes’ (35%) such as health-related outcomes of patients or non-professional carers.
Discussion
An increasing number of QI studies in the Irish Health Service were published over the past 5 years, most of which focused on improving the effectiveness, efficiency, timeliness and safety of care. Most of the studies were single-site hospital-based projects focused on ‘better disease management’. This phenomenon has recently been termed as Quality 2.0, an advancement on the field’s historical focus on compliance to minimum standards, Quality 1.0.67 Examples from our review included initiatives to reduce adverse events, to increase capacity and to release time to care. While our review found that patient-centredness, staff well-being and value for money were less often the focus of improvements, the equity dimension of quality was not a focus at all. Recent research has indicated that standards to help organisations monitor and improve their ability to provide equitable care are less mainstream than other quality standards24 and are at pilot stage in numerous countries.68
All studies shared important learning. Our assessment that over 90% of studies achieved their aims (fully or partially) and improved the health system performance presents a good indicator of the impact of QI approaches taken. Studies demonstrated very good coverage of international minimum standards for QI reporting across 14 of 16 criteria.23 Similar to other review studies,19 we identified opportunities for improvement in relation the ‘penetration/reach’’ of the initiative.
Our study also identified that the health outcomes and costs of QIs were understudied. Only one in three studies reported health outcomes. While 53% of studies discussed or quantified costs, it was not feasible to ascertain the cost-effectiveness of QI initiatives due to limited measurement of health outcomes and absence of comprehensive cost data. Other studies have similarly found that the costs of staff time and of ongoing data monitoring are often poorly elucidated in improvement work.19 Yet, as over half of studies discussed costs without adequately quantifying them, our findings suggest that knowledge of, or confidence in, performing cost assessments may be low among individuals engaged in QI.
In the context of the Quadruple Aim of Healthcare,6 together, these findings indicate that the QI studies were often focused on enhancing the quality of care patients receive and less often on measuring associated changes in health outcomes, costs, or staff well-being.
Implications
The profile of studies in our review implies there is strong engagement in QI project work in local settings yet insufficient measurement of cost and outcomes. Reflecting stakeholder discussions, adopting a value-based approach to programmes of QI69–72 may support large-scale service enhancement and better health (Quality 3.0).67 Clear guidelines exist for the assessment of resource use and cost in QI studies10–14 and in healthcare more generally.73–76 These should be further explored and tailored to support documentation and reporting of costs in QI. The development of QI reporting checklists that include explicit health economic items would also be of benefit.
Additionally, adopting equity standards for healthcare in the future, may support greater awareness of equity issues, and foster equity measures and improvements.68 Routine data collection on outcomes and costs is important to assure that health gains in one subpopulation are not achieved at the expense of another.5 Further, to support policies aimed at integrating care in the community in Ireland,77 increased visibility of QI work beyond the acute sector is needed.
Finally, for QI practitioners, this scoping review may help inform QI practice and reporting. A systematic review of QI studies in specific clinical contexts could follow on to identify best practices in these areas.
Strengths and limitations
To our knowledge, this is the first study to profile QI studies focused on a range of QI interventions on a country level. A key strength was that we used a robust scoping review approach and published critique tool adapted to context. Our use of two researchers to independently screen abstracts, review QI studies and build consensus reduces the potential for bias in our findings. Our workshop with QI stakeholders helped with interpretation and contextualisation of the findings. Yet, as our review was based on QI studies in the public domain, the results may not give a full representation of QI work conducted in Ireland over the past 5 years. Therefore, for non-acute sectors, it is difficult to conclude what activities are needed most: stimulus to support QI work or increased support for dissemination activities. Publication bias may have led to a proportionally higher level of QI studies of high reporting standard. Additionally, our study results likely reflect to some extent self-reporting bias in QI studies. However, our rigorous approach to the interpretation of results may have off-set this somewhat.
Conclusion
Studies included in our review demonstrated a variety of QI interventions and high reporting standards. Strategies are needed to support stimulation and dissemination of QI beyond the acute sector and awareness of equity issues as QI work. While it was not possible to assess the cost-effectiveness of QI interventions, it is clear that QI practitioners need to consider and report health outcomes and costs, routinely. This achievable goal may better support decision-making about resource allocation to maximise healthcare quality and health outcomes.
Ethics statements
Acknowledgments
Dr Philip Crowley, National Director for Quality Improvement, Health Service Executive for commissioning the review and supporting the process. Gethin White, HSE Librarian for advice on search process. Workshop participants. Dr Peter Lachman, International Society for Quality in Healthcare and Ray Healy, Department of Health for contributing their perspective.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @SiobhanMCarthy, @SamiraJabakhanj
Contributors Idea for research: JS, JM. Research design: SMC, SBJ, JS, JM, MAF. Tool development, search and critique: SMC, SBJ. Descriptive statistics and inclusion of manuscript references: SMC. Plan and deliver national stakeholder engagement workshop: MAF, JM, SMC, SBJ, JS. Manuscript development: SMC, SBJ and JS. Manuscript edits: JM, MAF. All authors have approved the submitted version of the paper.
Funding This study was supported by HSE National Quality Improvement Team.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.