Article Text
PDF
Statistics from Altmetric.com
- healthcare quality improvement
- quality improvement
- quality improvement methodologies
- health services research
Background
In healthcare, quality improvement (QI) aims to improve patient outcomes and fight inefficiencies.1 2 Inefficiency is associated with limited access to care and premature death.3 QI is not a costless endeavour,1 so healthcare providers and decision-makers must quantify both potential health and economic impacts that result from QI projects.
Economic analysis supports decision-makers by estimating the value associated with a programme. In QI, this includes studying the costs of implementing a project as well as evaluating any incremental changes in healthcare costs that occur as a result of the project. In tandem with an evaluation of the project’s impact on patient and provider outcomes, economic analysis provides another value dimension. Demonstrating added value can further incentivise resourcing for the implementation of a QI project to decision-makers. Despite the potential benefits of performing economic analyses on QI projects, little guidance exists on how to empirically evaluate their potential costs.4
Primary and administrative data each have unique and complementary strengths. Primary data, collected during a QI project, measures processes and outcomes which are important for evaluation, and are often not captured in administrative data.5 In contrast, administrative data represents a secondary data source which, although routinely collected for purposes other than research, can serve as a source of readily available information that lends itself to further analyses. Primary data are collected immediately, without the delays inherent in accessing and analysing administrative data. Administrative data offers linkable, comprehensive records of health system resource utilisation at both the population and the individual level,6 and provides a way to follow patients over time and outside the facility conducting the project. Measures of performance (eg, changes in the utilisation of healthcare services) can be operationalised to quantify the impact of the project. The objective of this paper is to describe a methodology, using administrative and primary data to inform measures of economic impact using three QI project case studies.
Setting and case studies
The IDEAS programme
The IDEAS (Improving and Driving Excellence Across Sectors) programme is an evidence-based QI training programme for healthcare professionals.5 The programme teaches clinicians and administrators QI methods so that they can improve the quality of patient care. An essential component is a team-based applied learning project. Project teams identify an issue, develop and implement a QI project in their own organisation.
The case studies
Three IDEAS projects were selected to illustrate the potential for combining primary with administrative data to evaluate their impact from the healthcare system perspective. Additional information describing each case study can be found elsewhere.7–9
The administrative databases
Health administrative databases cover services received by all residents insured under Ontario’s single payer system, including inpatient, long-term and ambulatory care.8 These databases allow patients to be followed over time and across the continuum of care. Table 1 contains a list of databases used in the case study examples.10 These data sets were linked using unique, encoded identifiers and analysed at ICES.10 The use of these data is authorised under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board.
Administrative data sources for case study examples
Methods
Five steps were crucial to informing and carrying out the economic analyses:
Step 1: identify the research question
Identifying the research question requires collaboration among all relevant stakeholders (ie, QI specialists, health economists, front-line healthcare workers and managers). The QI project team informs researchers of the problems the project addressed, project design and the interventions provided. Using this information, a structured approach, (eg, the PICO (Population Intervention Comparator Outcome) framework) can be used to develop a focussed research question.11
Step 2: identify the appropriate study design
Choosing the study design best suited to answer the research question is closely linked to identifying the research question. The study design should be appropriate and feasible in its ability to assess incremental changes following the implementation of a QI project.
For the purpose of economic analysis, two scenarios were defined to determine the impact of the QI project: (1) a comparable contemporary cohort in which the project was not implemented or a historical cohort from the same facility; and (2) the cohort exposed to the QI project implementation. Population cohorts can be used to extrapolate the incremental changes attributed to the QI project to measure the potential effect of project spread. The proper study design should be able to assess the differences between these two scenarios.
Step 3: clarify available data
In cases where primary data was not collected for the purpose of economic evaluation, administrative data can serve as a potential secondary data source to detect changes in healthcare resource utilisation and quantify the associated change in expenditures associated with a QI project. A combination of primary data collected by the QI teams and routinely collected health administrative data held at ICES were used to evaluate the health and economic impact associated with each QI project.
Step 4: identify outcomes relevant to health system impact
The project’s performance outcomes were expressed in terms of health system impact. Ideal performance outcomes include measures available in both the pre-project and post-project scenarios identified at either the population-level or patient-level.4 Outcomes shown to be associated with health service utilisation are ideal targets for quantifying the health system impact of QI projects.
Step 5: quantify the impact in dollar values
As most administrative databases record only service delivery, the value of each resource consumed needs to be estimated to obtain an estimate of healthcare payer-borne costs. Resource-level costing data derived from primary data, administrative data or literature estimates can all be used.
For the case studies, information in the Ontario administrative databases allowed estimation of patient-level costs.12 The Canadian Institute for Health Information developed a case mix methodology which assigns a relative measure of average resource use, the Resource Intensity Weights (RIW), to each hospital discharge record, based on clinical characteristics of a hospital stay.12 The cost of hospital stays was estimated by multiplying the RIW by the cost per weighted case for the year in which the hospitalisation took place.12 Similar case mix methods were used to assign costs to inpatient rehabilitation and long-term care stays. Micro-level cost data for prescription drugs and physician services was obtained from the Ontario Drug Benefit Plan (ODB) database and the Ontario Health Insurance Plan (OHIP) physician claims database, respectively (table 1).10 12
Primary data and literature estimates can also be used to derive unit or patient-level costs. Direct costs/revenues can be estimated from price lists, fee schedules or records of items purchased during a project’s implementation.4 Literature estimates may also provide costs associated with healthcare resources or treatment modalities. These costs are year specific and should be inflation-adjusted using relevant price indices.4 13 For the case studies, measures of health expenditure are reported in Canadian dollars (CAD). Between 2015 and 2017, $1 CAD ≈ $0.77 USD or 0.69 EUR.14
Patient and public involvement
The case study projects outlined below were developed with QI coaches, contributors, staff and patient participants.
Results: case studies
Case study 1: St Joseph’s Health Centre (Toronto, Ontario) Patient-Centred Falls Prevention project
Step 1: identify the research question
Falls are among the most frequently reported inpatient event; estimates of the proportion of patients who experience a fall range as high as 15%.15 Approximately one-third of falls can be prevented by managing patients’ underlying fall risk factors and optimising the hospital environment.16
The St Joseph’s Health Centre (SJHC) Patient-Centred Falls Prevention project focussed on fall prevention in their transitional care unit (TCU) through a revamping of fall risk assessment screening, creation of patient-centred fall prevention plans and coordination of care rounds specific to raising awareness around patients at higher risk of falling.
The identified outcomes pertinent to understanding the potential economic impact of the QI project were: (1) hospital resources attributed to an in-hospital fall resulting in serious injury; and (2) the reduction in falls resulting in serious injury following project implementation.
Step 2: identify the appropriate study design
A matched-cohort design was used to compare two scenarios to evaluate the economic impact of the project: (1) the SJHC TCU prior to project implementation; and (2) the SJHC TCU following implementation. Similar scenarios were used to evaluate the potential impact of the project being spread across the province of Ontario.
Step 3: clarify available data
The project team used data routinely collected by SJHC, consisting of medical records of in-hospital falls resulting in serious injury. The team supplemented this by documenting all falls, regardless of severity, over a 9-month period prior to project implementation and an 18-month period post-implementation.
Step 4: identify outcomes relevant to health system impact
The SJHC’s TCU data recorded a 50% reduction in the rate of falls following project implementation. Using administrative data, the impact on healthcare resource usage was estimated by matching patients with no fall to those who experienced fall resulting in serious injury on age, sex, comorbidities and reason for hospital admission. Matching results indicated an average incremental RIW of approximately 3.70 per patient, reflective of healthcare resource usage attributed to the fall. Administrative data also provided a province-wide estimate of in-hospital falls resulting in serious injury. From 1 April 2010 to 31 March 2016, 7769 patients in Ontario had an in-hospital fall recorded; 65% resulted in a serious injury.
Step 5: quantify the impact in dollar values
Scenario 1 translated to a cost of $11 696 per month pre-project implementation. Scenario 2 translated to a cost of $5848 per month post-project implementation. Assessing the net cost between scenarios estimated that the QI project resulted in an average savings of $5848 per month for the TCU. Extrapolating to the province-wide estimate for falls resulting in serious injury, the project may have the potential to avert $8 859 804 annually.
Case study 2: Kipling Acres Convalescent Care Length of Stay project
Step 1: identify the research question
Individuals in alternative level of care (ALC) no longer require the service intensity provided in the acute care setting and are waiting for placement in a more appropriate setting. ALC hospital days are considered an inefficient use of hospital resources that could otherwise be used for those with acute needs.17 ALC patients have created ‘gridlock’ in the system by occupying beds that could otherwise be used to admit patients from emergency departments.18 Timely availability of post-discharge services, including beds in long-term care facilities, would help address these problems.
The Kipling Acres Convalescent Care Length of Stay project focussed on improving resource utilisation and efficiency in two areas: (1) streamlining the application approval process for convalescent care admission; and (2) effective discharge planning with residents, families and community partners.
The potential economic impact was predicted using outcome measures pertinent to the project’s objectives of: (1) initiating a more efficient transfer of patients from acute care to convalescent care rather than from acute care to inpatient rehabilitation; and (2) reducing costs by treating eligible individuals in the convalescent care setting rather than in inpatient rehabilitation.
Step 2: identify the appropriate study design
A observational matched-cohort design was used to evaluate the economic impact of the project. Differences were compared for two scenarios: (1) a current care scenario where patients are discharged from ALC to inpatient rehabilitation; and (2) patients being discharged from ALC to convalescent care.
Step 3: clarify available data
Data regarding the reduction in ALC days associated with improved patient intake to convalescent care was not collected by the project team and was not available in health administrative databases. Therefore, routinely collected administrative data was used to measure the change in hospital resource utilisation associated with usage of convalescent care when compared with inpatient rehabilitation. The analysis included patients who were candidates for treatment in either care setting. Convalescent care patients were matched to inpatient rehabilitation recipients on age, sex, comorbidities and reason for hospitalisation.
Step 4: identify outcomes relevant to health system impact
Patients discharged to convalescent care spent an average of 8.5 days longer in ALC than those transferred to inpatient rehabilitation. Further, convalescent care patients had an incremental RIW of 0.67 per patient compared with matched inpatient rehabilitation patients. The difference in RIW likely reflected the increased burden of ALC stays.
Step 5: quantify the impact in dollar values
Costs of prescription claims and physician visits were added to the estimated daily cost of care for patients in convalescent care. For inpatient rehabilitation, these costs are already included in the hospital’s cost of care. Despite being associated with longer patient stays, convalescent care cost, on average, $11 100 less than inpatient rehabilitation. Considering the incremental change in preceding hospital RIW between convalescent care and inpatient rehabilitation, this suggests that by improving the efficiency and likelihood of transfer from acute care to convalescent care, the project has the potential to achieve in-hospital cost-savings estimated at $3800 per patient.
Case study 3: London Health Sciences Centre: Renal Program Body Access and Independent Dialysis project
Step 1: identify the research question
In Ontario, the population incidence rate of end-stage renal disease (ESRD) increased from 182.0 per million in 2006 to 219.4 per million in 2015, an increase of 20% over a 10-year period.19 Two types of dialysis are utilised by persons with ESRD: peritoneal dialysis and haemodialysis. In Ontario, in 2015, 75.9% of ESRD patients were initially treated using home-based haemodialysis (down from 80.4% in 2012), while 21.4% were initially treated using peritoneal dialysis (up from 16.8% in 2012).19
The London Health Sciences Centre: Renal Program Body Access and Independent Dialysis project focussed on improving peritoneal and vascular access for dialysis patients and increasing the uptake of independent (ie, home-based) dialysis. The project targeted suboptimal starting dialysis patients which included new dialysis patients who did not initially start on a home-based dialysis modality and implemented a new process for referral to vascular surgery, peritoneal dialysis and home haemodialysis consultations.
To assess the potential economic impact, we estimated the project’s impact on the uptake of independent dialysis modalities.
Step 2: identify the appropriate study design
An observational cohort study design evaluating patients from the following two scenarios was used to evaluate the impact of the QI project: (1) prior to project implementation patients would likely remain on hospital or clinic-based haemodialysis; and (2) post project implementation when patients were able to switch from hospital or clinic-based haemodialysis to home-based dialysis treatment (either home-based haemodialysis or peritoneal dialysis), if clinically appropriate.
Step 3: clarify available data
The research team collected primary data first-hand from January 2015 to September 2016, documenting patients who switched from hospital or clinic-based haemodialysis to independent dialysis.
Step 4: identify outcomes relevant to health system impact
From January 2015 to September 2016, the interventions were provided to 210 individuals who had initiated treatment on hospital or clinic-based haemodialysis. Of those, nine switched to independent dialysis.
Step 5: quantify the impact in dollar values
A study conducted by the Toronto Health Economics and Technology Assessment (THETA) Collaborative examined healthcare costs from the perspective of the Ministry of Health and Long-Term Care for patients newly initiating long-term dialysis in Ontario.20 Using administrative databases held at ICES, the study estimated the cost of hospital or clinic-based haemodialysis to be $132 889 per patient in the first year of dialysis. Compared with the cost of hospital-based haemodialysis, independent dialysis was associated with an estimated cost-savings of $56 068 and $48 071 per patient for peritoneal dialysis and home-based haemodialysis, respectively.
Prior to project implementation, all 210 individuals would have likely remained on hospital or clinic-based haemodialysis resulting in a healthcare cost of $27 906 690. Following project implementation, those switching to home-based modalities had an estimated healthcare cost of $27 434 066. Therefore, the incremental net cost associated with the QI project is an estimated savings of $472 642 in the first year of dialysis.
Discussion
The three case studies represent examples of estimating the economic impact of a QI project using administrative and primary data. In each case, the evaluation was tailored to the project’s specific health system impact and the data available to measure relevant health system performance outcomes.
For the SJHC and Kipling Acres case studies, two different methods were used to estimate the economic impact of the project using administrative data. In the SJHC case study, administrative data allowed for the matching of patients to estimate the direct hospital resource utilisation costs potentially averted per patient. The per patient cost was combined with programme-specific performance outcomes to estimate the impact of averting falls resulting serious injury. In the Kipling Acres case study, administrative data was used to measure the cost of care associated with hospital resource utilisation pre-discharge and post-discharge treatment in the convalescent care and inpatient rehabilitation settings, generating estimates of the potential impact of the project in improving the transition of patient to a different and appropriate care setting.
The London Health Sciences Centre (LHSC) case study used primary data and literature estimates (although literature estimates obtained using Ontario’s administrative databases) to evaluate the project’s potential economic impact. Project-specific data estimating frequency of patient transfers to different treatment modalities were costed using literature estimates to obtain a measure of the economic impact of the project.
Strengths and limitations
Strengths of this methodology are in the flexibility of using readily available administrative and primary data to estimate the economic impact of QI projects. This method can also lend itself to estimating the economic impact accompanying the spread and expansion of a project. Since implementation the SJHC fall-prevention project, the programme has spread to other departments, including SJHC’s Medicine Department where a large proportion of patients are at a high risk for falls.21 The Kipling Acres team is considering wider-scale adoption in other Long-Term Care Homes providing similar services throughout the city of Toronto.22 The LHSC Renal Program Body Access and Independent Dialysis project has already spread to other hospital sites in Ontario.23
Using administrative and primary data to derive estimates of economic impact has potential limitations. The described methods are limited by information availability. As overhead and implementation costs associated with the QI projects were not considered, the economic impact may be overestimated. In contrast, because changes in productivity associated with improved efficiency were not included in the analysis, it is possible that the full impact of the projects were underestimated.24 For example, an additional improvement achieved by the Kipling Acres project was a reduction in the cost associated with patient intake, freeing up approximately 240 staff hours each year.8 This was not included in the economic evaluation as staff hours could not be tracked using administrative databases. A further limitation is that the information available in administrative databases only allows the estimation of costs from the perspective of the payer. QI may also reduce costs and improve care from the perspective of the patient and caregivers; or, conversely, cost savings from the perspective of the insurer may involve imposing additional costs on patients and their caregivers.
Collection and usage of either primary data or secondary data such as administrative data may introduce their own associated biasses into a study. Primary data can be prone to bias associated with collecting data for a predetermined purpose.25 In comparison, administrative data may introduce bias due to misclassification, environmental influences and data collection processes as a result of the information not being specially collected for research purposes.25 26 It is important to note that even when considering outcomes relevant to economic analyses, it is difficult to assume that the healthcare resource usage being quantified and the calculated changes in healthcare expenditure are representative of each patient being impacted by the QI project.27
As this approach focusses on studying a potential economic impact, outcomes of interest were those which are quantifiable in monetary values such as health service utilisation. Other clinical outcomes are important and should be included in further research. Moreover, this approach does not represent a comprehensive economic analysis but rather a high-level economic analysis to highlight a potential economic impact. Future work should be planned to execute a comprehensive economic evaluation.
Conclusions
These case studies demonstrate how the use of administrative data and primary data can potentially inform a QI project’s economic impact. Economic impact can be weighed against a QI project’s implementation and operating costs, adding estimates of economic value to considerations of improved efficiency, and quality of care, thereby emphasising another dimension of a QI project’s value to the healthcare system.
Acknowledgments
This work was conducted through access to QI project and administrative data that was supported by IDEAS Ontario. IDEAS is a collaboration between Health Quality Ontario, ICES, the Institute of Health Policy, Management and Evaluation (IHPME) at the Dalla Lana School of Public Health, University of Toronto, and the Faculties of Medicine at seven Ontario universities. The authors would also like to thank the QI coaches (Shauna Cunningham, Joe Mauti, Ajibike Oyewumi and Alice Strachan), contributors, staff and patient participants of the original QI projects used as case study examples.
Footnotes
Contributors AM, RC, LM, GR and WI were responsible for study conceptualisation. AM, RC, LM and WI were responsible for analyses and interpretation of the data. RC and LM were involved in the acquisition of the data. AM wrote the first version of the manuscript. Data discussion and manuscript review was performed by AM, RC, LM, GR and WI. All authors contributed to the drafting of the final manuscript.
Funding Funding for this project evaluation study was provided by St. Michael's Hospital, Unity Health Toronto and the Institute of Health Policy, Management and Evaluation (IHPME) at the Dalla Lana School of Public Health, University of Toronto. This study was also supported by ICES, which is funded by an annual grant from the Ontario MOHLTC.
Disclaimer The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.