Article Text
PDF
Abstract
There are many instances of perceived or real inefficiencies in health service delivery. Both healthcare providers and policy makers need to know the impact and cost of applying strategies to change the behaviour of individuals or organisations. Quality improvement or implementation research is concerned with evaluating the methods of behavioural change. Addressing inefficiencies in healthcare services raises a series of issues, beginning with how inefficiency itself should be defined. The basic concepts of cost analysis and economic evaluations are explained and a model for working through the economic issues of quality improvement is discussed. This model combines the costs and benefits of corrected inefficiency with the costs and degree of behavioural change achieved by a quality improvement method in the policy maker’s locality. It shows why it may not always be cost effective for policy makers to address suboptimal behaviour. Both the interpretation of quality improvement research findings and their local application need careful consideration. The limited availability of applicable quality improvement research may make it difficult to provide robust advice on the value for money of many behavioural quality improvement strategies.
- quality improvement research
- health policy
- cost effectiveness analysis
Statistics from Altmetric.com
There are a number of formal definitions of “inefficiency”, but broadly we can think of it as a wasteful use of resources for no (or very little) benefit or a failure to use resources on clearly beneficial activities. Inefficiency may arise because of apparently inappropriate, irrational, or misinformed decisions by individuals (carers, patients, or clinicians) or organisations. The impact of non-compliance on healthcare efficiency is determined by its effects on both outcome and costs.1 Quality improvement and implementation methods seek to change the behaviour of individuals or organisations in response to inefficiencies. Behavioural change comes at a certain cost and achieves a certain level of change. It is never costless—for example, the pharmaceutical industry markets and supports its products to influence uptake and these costs are offset against increased profit. From a social perspective, we need to assess health gains which we commonly think of in physical rather than monetary terms. A more complex framework than simple net profit is therefore needed. The economics of quality improvement provide a way of thinking about inefficiency and identify, for policy makers and practitioners, the best use of scarce resources to achieve quality improvement goals. There are many instances of apparent inefficiencies and variation in our healthcare systems, resulting in considerable interest from policy makers about the scope for using quality improvement and change management methods. Examples include important research findings that do not translate consistently into practice, older but cost effective treatments that fall out of fashion, or equivocal but well marketed products that achieve considerable uptake.2–4 This paper describes an introduction to economic evaluations and presents models and examples of economic evaluations of quality improvement and implementation strategies in health care.
BASICS OF ECONOMIC EVALUATION OF HEALTHCARE INTERVENTIONS
Economic evaluations are a specific form of evaluation research which focus on making the relationship explicit between the amount of benefit achieved and the required investment related to a healthcare intervention. Before an evaluation can be considered a complete and valuable economic evaluation, two criteria must be met. First, there needs to be a problem of choice. In quality improvement research this problem of choice consists of a comparison of different quality improvement strategies or the comparison of such a strategy and “doing nothing” or “usual health care”. Second, in economic evaluations an explicit relationship is made between the efforts (use of people and resources), on the one hand, and the related consequences or actual outcomes on the other. The use of people and resources is usually expressed in monetary units (euros, dollars) so that they can be considered expenses. Relating costs to outcome results in the efficiency or relative cost effectiveness of a quality improvement strategy which can be expressed as a cost effectiveness ratio.5 An example is shown in box 1.
Box 1 Example: A randomised study of three training/support strategies for physicians to use when screening and intervening in cases of alcohol abuse5
The three training/support strategies were intended to motivate physicians to introduce screening and a brief intervention targeted towards alcohol abuse. The strategies concerned were (1) distribution of guidelines, (2) distribution of guidelines plus training, and (3) distribution of guidelines plus training and telephone support. Strategy 3 was the most cost effective in terms of cost per patient screened and cost per patient on whom an intervention had been performed. The cost effectiveness for each strategy was determined separately. Costs per patient screened were: trained and supported GPs (£1.05); trained GPs (£1.08); and controls (£1.47). Costs per patient in whom an intervention had been performed were: trained and supported GPs (£5.43); trained GPs (£6.02); and controls (£8.19). The cost effectiveness reported therefore concerned the relative cost effectiveness compared with doing nothing, an alternative which was not included in the study.
Incremental analysis of these figures indicated that the relative cost effectiveness of distribution of the guidelines plus training versus guideline distribution only is more positive than the relative cost effectiveness of distribution of the guidelines plus training and telephone support versus guideline distribution only. The relative cost effectiveness of the most intensive form of implementation (distribution of guidelines plus training and telephone support) compared with distribution of guidelines plus training turned out to be less preferable.
Cost analysis
The execution of cost analyses is the main part of each economic evaluation of quality improvement strategies. Within a cost analysis a distinction is made between fixed and variable costs. Fixed costs are costs that show no link to the scale of the actual use of the specific (healthcare) provision. For example, if consensus meetings are organised to formulate a clinical guideline, the costs will remain the same whether the guideline is later used by one or 100 primary care physicians/general practitioners (GPs) or whether it is applicable to 100 or 1000 patients. Attribution of fixed costs to an individual caregiver or an individual patient occurs on the basis of a division. If we assume the fixed costs to amount to €10 000 for the consensus meetings and there are 10 physicians with 10 relevant patients each, the fixed costs for that guideline are €1000 per healthcare provider and €100 per patient.
On the one hand, the variable costs of a quality improvement strategy are dependent on the intensity. For example, extra education that lasts more than one day is more expensive than extra education that lasts only a few hours. On the other hand, the variable costs are related to the degree to which the guideline or change proposal is followed. Let us assume that a guideline advises that patients with a high risk for cardiovascular disease should be recalled on a regular basis for check ups to measure blood pressure. In this case, the number of patients affected by the guideline determines the costs involved. If the number of patients is zero, then the number of clinical actions based on the guidelines is zero, and thus there are zero costs. Variable costs will therefore always have to be measured empirically because they cannot be calculated through a simple division per measuring unit (practice, primary care physician, or patient), as can be done with fixed costs.
Different types of costs
In the economic evaluation of quality improvement interventions, costs can be subdivided into different phases of the quality improvement process.6 Firstly, there are costs related to the task of collecting evidence to identify best practices or to the task of developing new or optimal care procedures. Patient related research can be performed, for example, so that best care procedures can be defined and consensus meetings can be organised in order to collect the opinions of experts in the specific area of attention. Basically, these developmental costs (fixed costs) should be part of a cost analysis regarding quality improvement strategies in case a change and improvement intervention is not available and has to be developed or adjusted.
Secondly, there are costs associated with organising a specific quality improvement or change intervention. For example, when implementing a clinical guideline that uses outreach visitors who are to visit primary care physicians, their training would be desirable. Such costs are basically one-time costs and can therefore be considered fixed costs, unless the intervention used after the experience that is gained is subject to change. In that case, the efforts associated with a revision of the strategy must be considered execution costs.
On the other hand, the costs of the actual execution of the quality improvement strategy (such as sending out guidelines or outreach visitors who spend time visiting GP practices) are not relevant until the moment the strategy is executed.7 Such costs can be considered fixed or variable, depending on the amount of detail included in the cost study. If researchers perform a general cost analysis and use a fixed cost approach, it will sufficient to know the total scale of the outreach visitor formation. Consequently, division calculation can be made that is based on the number of GPs or patients reached (depending on the level of cost effectiveness measuring which is discussed later). It is then, however, not possible to determine cost variation per GP or patient. If the researchers want to determine such a variation to obtain more detail, each outreach visitor will have to record the time spent per visit. With that information, the variable costs can be determined per visit, per practice, per physician, and possibly even per patient. Obviously, a combination of the fixed and variable costs approximation is possible: the visitors’ time that is not directly related to a physician’s clinical work (such as work meetings and reading of literature) are considered fixed (overhead) costs and attributed to each specific visitor by using a division calculation. The time that is directly related to the visit of a specific practice is expressed in minutes and converted into monetary units. Examples are shown in boxes 2 and 3.
Box 2 Example: Costs and cost reductions of a quality strategy to improve test ordering in general practice6
This multicentre randomised controlled trial compared the costs and cost reductions of an innovative strategy aimed at improving the test ordering routines of GPs with a traditional strategy. It included 27 local GP groups in the Netherlands with diagnostic centres and comprised a total of 194 GPs. The test ordering strategy was systematically developed and combined feedback, guideline dissemination, and quality improvement sessions in small groups. In 13 experimental local groups, GPs discussed their feedback report at regular quality meetings, related them to guidelines, and made plans for change. In 14 control groups only feedback was provided. The main outcome measures were costs, which were divided into running costs (costs of the feedback reports), development costs (activities for the continuation of the project—administration, organisation, development and updating of concise guideline information), and research costs (activities for the scientific development of the feedback system). In addition, costs (reductions) of the laboratory tests and all tests per GP and per 6 months were determined by assessing the difference between the follow up period and the baseline period in the costs of tests ordered; this difference was compared between the two arms.
When only running costs were included, the total strategy was found to cost €93 per GP per 6 months compared with €17 per GP per 6 months in the feedback arm. The GPs in the total strategy arm achieved a mean reduction in the costs of tests of €301 per GP per 6 months compared with €161 per GP per 6 months in the feedback arm. On the basis of this cost analysis, the authors recommended the implementation of the quality strategy on a larger scale.
This study clearly shows that not only developmental and execution costs are relevant in economic evaluation, but also the costs related to the change in healthcare provision. Of course, limiting the analysis of the costs of healthcare provision to diagnostic activities and not including the costs of therapeutic actions is based on the assumption that this would increase the positive cost difference found.
Box 3 Example: Process evaluation of a multifaceted intervention to improve cardiovascular disease prevention in general practice7
The Dutch CARPE study was a randomised controlled before and after study which evaluated quality improvement resulting from the use of trained consultants for patients with cardiovascular risk indicators or cardiovascular disease in 60 experimental and 60 control GP practices. Before and after the quality improvement strategy was applied, measurements were taken of the organisational characteristics (teamwork, special office hours, administrative and patient follow up systems), the healthcare process (working according to GP standards), and clinical parameters (blood pressure, etc). The implementation strategy consisted of regular visits by trained consultants to the experimental practices during an 18 month period. The consultants’ task was to encourage the active use of seven national primary care physician/medical guidelines (hypertension, cholesterol, diabetes mellitus II, peripheral arteriosclerosis, angina pectoris, heart failure, and cerebrovascular accident or TIA). The cost of this implementation strategy was compared with the cost of no active implementation strategy over an 18 month period from the healthcare perspective. The cost analysis was limited to the execution of the implementation strategy, so possible changes in the healthcare process (such as longer consultations) were not taken into consideration. It was therefore not necessary to perform cost research in the control practices since the cost of no implementation strategy would equal zero by definition.
During the study the costs for each visit to the GP practices were prospectively recorded by the consultant. This included the number of visits by the consultant to each practice; the preparation, travel, and consultation time per visit; the preparation and execution time spent by the GP(s) and practice assistant(s) during each consultant visit; and the number of miles the consultant had to travel. The recorded activities were compared with the actual cost prices. Data were collected on 934 consultation visits to 62 GP practices. Even though the protocol required a fixed number of consultation visits per practice, the actual number of visits per practice ranged from 3 to 17 (mean 14.8). Partly because of this, the costs per practice varied. It also turned out that the number of primary care physicians and assistants who actively participated in the implementation strategy by preparing and attending consultation visits varied for each practice from one to four primary care physicians and zero to five assistants per practice. In particular, the costs of time investment of the primary care physicians largely determined the variation in costs of the quality improvement strategy.
The above mentioned cost categories are the cost per practice or caregiver of a quality improvement strategy (as reflected by Δci in the model described by Mason et al8). In addition, costs are sometimes associated with a change in healthcare provision as a result of application of a quality improvement strategy. As a result of the application of a clinical guideline, for instance, physicians may be able to see patients more frequently or consultations may last longer because of more elaborate physical examinations. Benefits may include the ability to perform more diagnostic tests and the possibility of a quicker referral of a patient to the specialist. This also includes a change in the way a patient makes use of medical care, which can be partly attributed to effects on health. Non-medical costs, such as patients’ cost for time and travel and costs resulting from absence from work, can also be analysed on this level. These changes in healthcare provision costs are always considered variable costs. In essence, they are part of the cost of care (as reflected by Δct in the model of Mason et al8).
Different types of economic evaluations
The process or patient outcome (consequences, benefits) of alternative strategies studied in quality improvement projects are sometimes similar, in which case a cost minimisation analysis is useful (box 4).9 However, in most quality improvement strategies process or patient outcomes are also relevant, in addition to costs, and a full economic evaluation should be performed. The expression of the ratio of costs versus consequences into one unit (the cost effectiveness ratio) is often neglected when evaluating quality improvement strategies. Such studies can be called “cost consequence analyses” and simply give an overview of the costs and consequences without connecting them into one single unit. They are then presented to policy makers who have to make a decision about the choice of intervention based on a list of pros and cons without indicating a value or preference.10 McIntosh et al11 introduced the so called “balance sheet approach” in which positive and negative consequences are simply stated in a table. Although useful, these methods do not give a clear insight into the question of efficiency, which is the aim of other types of economic evaluation.
Box 4 Example: Individual feedback to physicians about diagnostic test requests9
In this study the influence of individual feedback on physicians’ requests for diagnostic tests was examined. Based on an extensive retrospective study, it was shown that the individual feedback led to a significant reduction in costs. In order to estimate the relative impact of the implementation strategy, data were used from a comparable laboratory in another part of The Netherlands. The researchers concluded that, based on both retrospective comparison and on the comparison with data from the other laboratory, routine individual feedback can be considered economically valuable. Data on patient outcomes were not available. Possible undesired side effects of the decline in diagnostic test requests were described on the basis of the number of hospital referrals in which no unexpected changes had been observed. Since no explicit connection was made between costs and outcomes, this study can be defined as a cost minimisation analysis. Although not stated, it was implied that cheaper health care would not equal worse health care.
Cost effectiveness analyses express effects in natural quantitative parameters which can be process parameters12 as well as patient outcome measures. In quality improvement research this may include, among others, the number of practices reached by the quality improvement strategy (e.g. mailing of guidelines); the number of practices, departments, or professionals working in accordance with a specific clinical guideline or proposal care pathway; the number of patients receiving treatment in accordance with such a guideline; the health condition of the patients concerned; and their satisfaction with the health care provided. An example of cost effectiveness analysis in thrombosis is shown in box 5.
Box 5 Example: Cost effectiveness of audit in thrombosis12
A study that examined the cost effectiveness of audit in improving the care of patients suspected of having an acute myocardial infarction provides an example of the problems concerning process parameters versus patient outcomes as cost effectiveness parameters. This study examined the cost of each additional patient treated for thrombosis. Instead of patient outcomes, a process parameter was used which resulted in an estimate of £101–395 per additional patient treated. The authors rightly based their choice for such an outcome measure on the fact that there is overwhelming evidence that the clinical actions encouraged are effective and pragmatic. This legitimised the assumption that the increase in thrombosis treatment would lead to better patient outcome, although this was not explicitly analysed.
In cost utility analysis the physical health of the patient as an outcome measure is central. In these analyses the patient’s eventual physical health is rated through the use of a utility. This physical health rating is indicated by a number between 0 and 1, where 1 equals perfect health and 0 the worst imaginable condition (death). Utilities can be used as the basis for the calculation of so called quality adjusted life years (QALYs), a measure that uses societal rating of a patient’s health and relates it to life span (box 6). In general, cost utility analyses require a lot of work because patients are required to fill out extensive questionnaires. This method of analysis is applied frequently in clinical evaluation studies but only occasionally in quality improvement or implementation research.13
Box 6 Example: Randomised controlled economic evaluation of asthma self-management in primary health care13
In this randomised controlled economic evaluation, guided self-management of asthma was compared with usual asthma care according to guidelines for Dutch family physicians. Nineteen family practices were randomised and 193 adults with stable asthma (98 self-management, 95 usual care) were included and monitored for 2 years. Patient specific cost data were collected, preference based utilities were assessed, and incremental cost per quality adjusted life year (QALY) and incremental cost per successfully treated week gained was calculated. Self-management patients gained 0.039 QALY and experienced 81 successfully treated weeks during the 2 year period; the corresponding figures for usual care were 0.024 QALY and 75 weeks. Total costs were 1084 euros for self-management and 1097 euros for usual care. Self-management patients consumed 1680 puffs of budesonide compared with 1897 by those in the usual care group. When all costs were included, self-management was cost effective on all outcomes. It was concluded that guided self-management is a safe and efficient alternative approach compared with the asthma treatment usually provided in Dutch primary care.
Finally, cost benefit analyses are distinguished from other types of economic evaluations because they measure, not only the costs, but also the consequences in financial terms. For example, a patient’s survival or quality of life is expressed in guilders or dollars. There are many issues involved in this and therefore cost benefit analyses are seldom used in health care. The term “cost benefit” is often used incorrectly because financial savings that are characterised as benefits are, in fact, lower costs.
The efficiency or cost effectiveness of a quality improvement strategy expresses how the costs relate to the results obtained. These can then be formulated as the intervention costs per optimally treated patient (process parameters as outcome unit) or as change or implementation costs plus treatment costs per successfully treated patient (outcome parameter as outcome unit). By comparing the effects and costs of alternative quality improvement strategies with each other, the incremental cost effectiveness ratio can be determined. Based on the comparable alternative in the study, the average amount of investment needed when applying the experimental alternative can be determined.
MODEL FOR ASSESSING THE OVERALL POLICY IMPACT OF A QUALITY IMPROVEMENT METHOD
From the above description of the different types of economic evaluations it is clear that the ultimate goal of health economists is to express efficiency of a quality improvement strategy in terms of patient outcome. However, to date, this goal has seldom been attained in economic research on quality improvement. An important step forward to overcome this has been made by Mason and colleagues8 who have developed a more advanced approach to the economic evaluation of quality improvement strategies. Pursuing a policy to change suboptimal patterns of care combines quality improvement cost effectiveness (net cost and magnitude of impact upon behaviour when using a quality improvement strategy) with treatment cost effectiveness (incremental costs and benefits of more optimal behaviour). An example of the estimation of overall policy costs and benefits is shown in the following equation:
Policy cost effectiveness:
where Δbt, Δct and ΔCEt are the net health gain, cost of care and treatment cost effectiveness per patient (Δct /Δbt); Δci, Δbi and ΔCEi are the net cost, proportion of patient care changed, and quality improvement cost effectiveness per practice (Δci /Δbi); d is the duration of effect of the quality improvement method; np and pd are the average practice size and population prevalence of the condition targeted; and LCE is the loading factor on treatment cost effectiveness.
This equation shows the case when a change in health care is valued as a cost effectiveness ratio (cost per year of life gained) and the performance marker is the simple proportion of patients receiving appropriate care. This model was used, for example, to analyse data in a study of the apparent underutilisation of ACE inhibitors for heart failure in primary care.14,15 This so called EBOR trial examined the effectiveness of outreach visits to primary care by community pharmacists using recommendations derived from evidence based clinical practice guidelines. The numbers from the EBOR trial can be put into the equation as follows8:
Investing resources to change clinician behaviour imposes an addition (or loading) on treatment cost effectiveness. The loading is small (£300/life year gained) in the example and does not substantially diminish the attractiveness of the intervention to policy makers. Policy cost effectiveness is most likely to remain attractive in those treatments that are highly cost effective, and most likely to become unattractive when the cost effectiveness of treatment is borderline. A potentially complex interplay of factors determines whether quality improvement by a particular method is worthwhile. Cheaper methods achieving greater levels of change reduce the loading effect. Similarly, larger health gains per patient, higher prevalence of disease, larger practice size, or longer duration of behavioural change all reduce the loading, all other things being equal. Where the loading is small, treatment and policy cost effectiveness are very similar. Where the loading is large, further use of a cost effective treatment may not be worthwhile encouraging as a policy goal using available behavioural change methods.
DISCUSSION
Compared with simple treatment interventions, economic evaluations regarding quality improvement strategies are few. Because it is hardly ever possible to have one empirical study that gathers all the data needed to study cost effectiveness of a quality improvement strategy, the model described by Mason et al8 may provide the solution. In this model a distinction is made between treatment cost effectiveness (the net costs and benefits of a treatment when provided) and policy cost effectiveness (which combines treatment cost effectiveness with the cost and magnitude of change achieved by a quality improvement method). The example presented illustrates the case of a level of efficiency determined by a simple proportion—that is, the percentage of patients receiving an ACE inhibitor. In other instances it may be necessary to consider a mean level of exposure or a combination of dose and proportion. The key is to have robust linkage with (change in) an efficiency measure and treatment costs and health benefit. In addition, with the knowledge of the relative cost effectiveness of following a guideline compared with not following it, it is possible to determine the maximum financial resources that can be used neutrally to encourage the use of a guideline.16
A number of assumptions are often implicit when planning quality improvement. Commonly, local patients are assumed to be typical of those enrolled in trials and are treated in the same way. The findings of quality improvement studies are assumed to be transferable to different settings and possibly also to different evidence based messages and diseases. Costs of treatment and quality improvement will vary with the setting, hence the value of studies that report units of component resources disaggregated from their unit costs. The local cost of quality improvement needs to reflect local resource utilisation patterns and prices. The magnitude of behavioural change is unlikely to remain constant over time,17 and a decision needs to be taken as to whether quality improvement is a “one off” or whether periodic reimplementation needs to be costed.
Key messages
-
Economic evaluations of quality improvement strategies are based on the comparison of alternative methods of introducing desirable changes (or doing nothing) in health care. When performing a complete economic evaluation, the costs incurred by people and resources are related to the (health) outcomes obtained.
-
In contrast to economic evaluations of medical interventions, both process and patient outcomes can be used as cost effectiveness measures in economic evaluations of quality improvement strategies.
-
Economic evaluation research of quality improvement strategies is most beneficial if the cost effectiveness of the desired professional behaviours or healthcare processes are known and found to be acceptable.
-
To determine the cost effectiveness of a quality improvement strategy, the costs and benefits of corrected inefficiency can be combined with the costs and degree of behavioural change in the policy maker’s locality.
Prerequisites for worthwhile behavioural change are evidence of local suboptimal care and a locally deliverable cost effective alternative, but the local demography and characteristics of the audience to be targeted may also be important. Failure to understand these facets beforehand risks investment in quality improvement for a poor return. To understand how any piece of quality improvement research may be generalisable to a different context, it is necessary to consider its setting, its message, the method used, facilitators, barriers, and motivations both of those receiving the intervention and those designing it. A trial of educational outreach by community pharmacists to improve general practice prescribing of non-steroidal anti-inflammatory drugs failed to show a significant impact because participating practices already exhibited good baseline prescribing.18 It is important to address actual rather than perceived problems.
Given their limitations, findings of current quality improvement studies should be applied cautiously. It would be ideal for any condition with known prevalence and method of behavioural change to estimate loading adjustments for treatment cost effectiveness and thus determine the best quality improvement method to meet policy aims. Better research into economic aspects of quality improvement methods is needed to achieve this goal.
Acknowledgments
The author is grateful to Professor James Mason, Centre for Health Services Research, University of Newcastle upon Tyne, UK who supported this work substantially, especially by providing the example of the model for assessing the overall policy impact of a quality improvement method, and Dr Michel Wensing, Centre for Quality of Care Research, University Medical Centre Nijmegen, The Netherlands who commented and made suggestions on a previous draft of the paper.