Article Text
PDF
Abstract
Objective Our aim was to see if the reporting of better experiences by elective surgical patients was associated with better outcomes (effectiveness and safety). The objectives were to: describe the distribution of experience scores and any association with patients’ characteristics; determine the relationship of experience with effectiveness and with safety; and explore the influence of patient characteristics, year and provider on the relationship between experience and effectiveness.
Methods Patients undergoing one of three procedures from 2010 to 2012 in England who completed a patient reported outcome measure (PROM) questionnaire before and after surgery and a patient reported experience measure (PREM) questionnaire. Data on 4089 hip replacement patients, 4501 knee replacements and 1793 groin hernia repairs. Regression analysis was used to examine associations between disease-specific and generic PROMs and PREMs.
Results There was a weak positive association between experience and effectiveness for all three procedures (correlation coefficient with disease-specific PROMs for hip and knee replacements 0.2 and with EQ-5D 0.1 for all three procedures). The aspect of experience most strongly associated with a better outcome was the level of communication with and trust in their doctor. A higher experience score of 1 SD (about 1.5 on a 10-point scale) was associated with about 30% less likelihood of the patient reporting a complication. There was no difference between the eight dimensions of experience.
All the relationships observed were consistent over time, between different types of patients (age, sex, socioeconomic status) and between providers.
Conclusions Patients distinguish between the three domains of quality when reporting their experience and outcome. If the weak positive associations between domains were shown to be causal, there would be implications for maximising performance measures for providers.
- Patient Safety
- Health Services Research
- Patient-Centred Care
- Quality Measurement
- Surgery
Statistics from Altmetric.com
Introduction
In England, quality of care is defined as having three domains: patient safety, clinical effectiveness and patient experience (compassion, dignity, respect, etc).1 There is also increasing requirement to measure not only clinicians' views of the quality of care but also those of patients. While considering and measuring each domain separately has practical advantages, the three are inter-related.2 This may have implications for any attempts to improve the quality of care. For example, it may be possible to improve patients’ reports of the effectiveness of care by improving patients’ experiences. The increasing availability of data in England on the quality of inpatient care based on patient reported outcome measures (PROMs, covering effectiveness and safety) and patient reported experience measures (PREMs) provides a means of exploring the relationship between the three domains for hospital care.3
Although there have been many studies of the relationships between PROMs and PREMs in primary and ambulatory care,4 only eight studies have been carried out with hospital inpatients, six conducted in the USA and two in Taiwan. Most found a weak positive relationship. Three US studies5–7 were limited to analysing and comparing aggregated data. They considered all admissions for a large number of providers or health plans and reported better patient experience was associated with more effective care (eg hospitals in the top quartile for patient experience had outcomes 2–4% better than the lowest quartile)6 and with safer care (eg fewer complications).7 The other five studies explored associations using patient-level data (ie linked PROMs and PREMs for individual patients). The two earliest studies looked at patients admitted following an acute myocardial infarction and found that patients reporting good communication by clinicians were associated with better postdischarge health related quality of life (HRQL) (correlation coefficient 0.33).8 The other study found patients reporting a better experience were more likely to report better physical health (68% compared with 60%).9 Similar relationships have also been reported for patients admitted with chronic obstructive pulmonary disease (OR of reporting good health outcome if also reporting a good experience was 1.19)10 and for diabetes (greater trust in doctor and better support for self-management associated with better mental health (correlation coefficient 0.24 and 0.18, respectively)) but less so with better physical health (0.11 and 0.10).11 ,12 None of these studies was able to determine whether the relationship is causal and, if it is, the direction of causality.
The availability of PROM and PREM data for patients undergoing one of three elective operations (hip and knee replacement, groin hernia repair) in the east Midlands region provides an opportunity to explore the relationship between patient experience and outcome (effectiveness and safety) for surgical patients for the first time. Patients were invited to complete PROM questionnaires before and after surgery and a PREM questionnaire after surgery.
Our aim was to see if the reporting of better experiences was associated with better health outcomes. The objectives were to: describe the distribution of experience scores and any association with patients’ sociodemographic characteristics; determine the relationship of experience (overall and for component dimensions) with effectiveness and with safety; and the influence of patient characteristics, year and provider on relationship between experience and effectiveness.
Methods
Sample
All patients resident in the East Midlands who underwent one of the three operations between April 2010 and March 2012 were eligible for inclusion. East Midlands was defined as the area covered by the nine Primary Care Trusts that participated in the East Midlands Patient Experience Survey covering a population of 4.3 million in the counties of Derby, Leicester, Lincoln, Northants and Nottingham.13
PROMs
As part of the National PROMs Programme, patients were invited to complete a PROMs questionnaire before surgery and a second questionnaire was mailed to them either 3 months (hernia repair) or 6 months (joint replacement) afterwards.14 One reminder was mailed to non-responders.
Preoperative questionnaires included information on patients’ age, sex, general health, whether primary or revision surgery, and self-reported comorbidities (using a validated question15 ,16 that minimises the use of medical terminology.17 Data were linked to patients’ administrative records in Hospital Episode Statistics to obtain information on ethnicity and socioeconomic status (derived from their postcode and based on the Index of Multiple Deprivation using 2007 rankings).18
Joint replacement patients completed a disease-specific PROM that assessed symptoms and functional status (disability): Oxford Hip Score (OHS)19 or Oxford Knee Score (OKS).20 There was no disease-specific measure for patients undergoing hernia repair available. The OHS and OKS include 12 items, each scored from 0 to 4 and summated to provide an overall score of between 0 (severe symptoms and disability) and 48 (no problem). A typical patient has a preoperative score of 18 reflecting moderate pain, extreme trouble carrying out activities of daily living, walking with a limp and sudden severe pain on some days. The questionnaires also included a widely used generic PROM (EQ-5D index score) that assessed HRQL.21 It is based on patients’ assessment of five dimensions of health (mobility, self-care, daily activities, pain and anxiety/depression). Each question has three levels of response and answers are transformed into a HRQL score using utilities from the UK-Time Trade-Off value set.22 Scores range from –0.59 (worse than death), through 0.00 (dead) to 1.00 (perfect health).
The postoperative questionnaires included the same PROMs to enable improvement in scores to be assessed (our principal measure of effectiveness). In addition, a single transitional item was used to determine the extent of improvement: ‘Overall, how are the problems now in the hip/knee/groin on which you had surgery compared to before your operation?’ with five response categories from ‘much worse’ to ‘much better’.
For each surgical procedure we derived three measures of effectiveness: mean change in disease-specific PROM; mean change in generic PROM; and proportion reporting being ‘much better’ on a single transitional item. For hernia repair only two measures were possible as there was no disease-specific PROM.
Postoperative questionnaires also sought patient’s reports of the occurrence of four complications: wound problem; urinary problem; bleeding; allergy or reaction to drug. These had been selected by the National PROMs Programme as they needed to be generic and relevant to a wide range of surgical operations. The incidence of any one complication was our measure of safety.
PREMs
PREM questionnaires were mailed 6 weeks after surgery to all patients who had completed a preoperative PROM questionnaire. The questionnaire included 32 items concerned with experience (plus seven sociodemographic questions) based on those used in National Inpatient Surveys run by the Picker Institute.23 Responses to each item scored: 0 (always poor experience); 5 (sometimes poor experience); or 10 (always good experience).
Although the questionnaire is structured around the patient’s pathway from admission to discharge (see online supplementary appendix A), we analysed the data according to the eight dimensions of experience that have previously been identified by the Picker Institute. To do this, we mapped 24 of the 32 items (excluding 4 items on mixed sex facilities, 3 on discharge instructions and 1 on privacy discussing treatment) onto the dimensions which resulted in different numbers of items in each: consistency and coordination of care (1 item); treated with respect and dignity (1 item); adequacy of pain control (1 item); sufficient explanation and involvement (6 items); communication with and trust in doctors (6 items); communication with and trust in nurses (4 items); cleanliness of facilities and staff hand hygiene (4 items); sufficient discharge information (3 items). As there was only one item on trust and it did not distinguish between doctors and nurses, we decided to include it in both dimensions.
We also created an overall score based on a simple summation of all 24 items (scoring from 0–240). (An alternative overall score in which each of the eight dimensions contributed equal weighting regardless of the number of constituent items produced similar results and so is not presented).
The internal consistency of the multi-item dimensions was assessed using Cronbach’s α: sufficient explanation and involvement 0.79; communication with and trust in doctors 0.83; communication with and trust in nurses 0.78; cleanliness of facilities and staff hand hygiene 0.71; sufficient discharge information 0.57; overall score 0.92. Exclusion of the item on ‘trust in staff’ did not change the consistency of the two dimensions which it included.
Data
The number of providers of each procedure differed: hip replacement 31 providers; knee replacement 26; and hernia repair 20. This included 20 hospitals within 11 acute National Health Service (NHS) Trusts located within the East Midlands. The other 11 providers were either independent sector treatment centres, private hospitals or located outside the East Midlands region. The number of patients who completed a preoperative PROM for each procedure were: hip replacement 7037; knee replacement 7889; hernia repair 3829. Of these, the number who went on after surgery to complete a PREM and PROM were: hip replacement 4089 (58%); knee replacement 4501 (57%); hernia repair 1793 (47%).
There was little missing data within completed PREM questionnaires (1–4% for any single item). No attempt was made to impute missing data. Overall PREM scores were based on the items for which data were available with adjustment to take the smaller denominator into account.
Analysis
Analysis was performed using STATA. Associations between PREM scores and socioeconomic characteristics (age, sex, deprivation) were tested using linear regression for continuous outcomes and logistic regression for binary outcome measures. The outcome metric for the disease-specific and generic PROM scores was the difference in the scores before and after surgery. We did not adjust the analysis for patient characteristics as we were interested in health gain and safety regardless of patient characteristics.
The analysis was done using linear regression and repeated using random effects models adjusting for clustering within providers. The best fitting models for the data were found to be the linear regression models, hence the estimates obtained from these models have been reported.
To explore the effects of patient characteristics and of year on the relationship between PROMs and PREMs, an interaction term was introduced in the regression models. Case-mix adjustment was performed for the assessment of the impact of PREMs on the outcome rating of providers using previously published methods.24
The effects of patient characteristics and of year on the relationship between PROMs and PREMs, and differences in the PROM:PREM relationship between hospitals were tested using an interaction term in the regression models.
Results
Distribution of overall PREM and PROM scores
Most patients reported a good overall experience with the distribution of scores skewed (figure 1). While about 20% reported a ‘perfect’ experience, about 15% of patients undergoing hip or knee replacement had scores of less than 7 out of 10. Slightly fewer hernia repair patients (12%) reported less than 7.
Distribution of patients’ overall PREM scores. PREM, patient reported experience measure.
The change in PROM scores (not shown) was normally distributed for all three procedures: mean change in OHS 19.96 (SD 10.34); OKS 15.57 (9.83); EQ-5D for hip replacement 0.42 (0.34); for knee replacement 0.31 (0.33); for hernia repair 0.08 (0.22).
Association between patient socioeconomic characteristics and experience
The mean overall PREM score was higher among those aged 61–70 years undergoing joint replacement than among younger patients (table 1). There was no such association for hernia repair. Men reported a better experience than women for all three procedures but there was no significant association with socioeconomic status. There were insufficient non-white patients who underwent these procedures (1.9% hip replacement, 6.1% knee replacement, 5.5% hernia repair according to Hospital Episode Statistics data) to analyse the association with ethnicity.
Association (95% CI) between patient sociodemographic characteristics and overall mean PREM score
Association between experience and effectiveness
There was a significant positive association between a patient’s overall PREM score and their PROM change score for all three procedures. For example, for hip replacement a difference in the overall PREM score of 1 SD (1.47 on a 10-point scale) was associated with an improvement of 2.23 in the Oxford Hip Score and 0.038 in EQ-5D score (table 2). This association was also apparent in the scatter plot which revealed a weak positive correlation coefficient of 0.2 for hip and knee replacement (figure 2). Correlations with the generic PROM (EQ-5D) were weaker (hip replacement 0.1; knee replacement 0.14; hernia repair 0.1).
Change in PROM score associated with 1 SD difference in PREM score (overall and for each dimension)
Association between outcome (change in disease-specific PROM score) and overall PREM score. PREM, patient reported experience measure, PROM, patient reported outcome measure.
Analysis of the eight component dimensions of experience revealed that effectiveness was associated with all eight dimensions but that the strongest association was with ‘Communication with and trust in doctors’ followed by ‘Communication with and trust in nurses’ and ‘Sufficient explanation and involvement’ (table 2).
A consistent positive association between experience and effectiveness was seen when the latter was assessed using a single transitional item (figure 3). Patients undergoing hip replacement reporting that their condition was ‘much better’ reported a better experience (overall score 8.8) than those reporting it was ‘much worse’ (6.9). A similar pattern was apparent for the other two procedures. It is notable that even those patients who reported their condition to be ‘much worse’ still reported quite a high level of satisfaction with their experience.
Overall mean PREM score for response to single transitional item (‘Overall, how are the problems now…compared with before your operation?’ 1=Much better; 2=Little better; 3=Same; 4=Little worse; 5=Much worse). PREM, patient reported experience measure.
Association between experience and safety
The proportion of patients reporting at least one postoperative problem differed between procedures: hip replacement 31%; knee replacement 34%; hernia repair 23%. There was a significant negative association between patient experience and the reporting of postoperative complications (table 3). The likelihood of reporting a complication following hip replacement was lower (OR 0.72) if their overall experience score was 1 SD higher. The strength of the association was similar for knee replacement (0.71) and for hernia repair (0.64). There was little difference in the strength of the association between safety and each of the eight dimensions of experience.
OR of reporting a complication (any) associated with 1 SD change in PREM (overall and each dimension)
Influence of patient characteristics, year and provider on relationship between experience and effectiveness
There was no significant difference in the observed relationship between experience (overall PREM score) and effectiveness (disease-specific and generic PROM scores) for different age groups (<61 years, 61–70 years, >70 years), sexes or deprivation quintiles for all three procedures (see online supplementary appendix B). There was also no significant difference between 2010/2011 and 2011/2012 for all procedures (see online supplementary appendix C) or between the 11 highest volume providers (see online supplementary appendix D).
Adjustment of the main measure of providers’ effectiveness (mean risk adjusted PROM score) for their reported experience (overall PREM score) made little difference to their relative performance. For the 11 main providers of hip replacement there was little change in their position on a funnel plot (see online supplementary appendix E), although one provider shifted from the ‘alert’ section (greater than 2 SDs worse than average) to within the ‘normal’ range.
Discussion
Main findings
Most patients reported a good experience with 85% scoring over 7 out of 10. Men were more likely to report a good experience than women, as were those in their 60s (compared with younger patients). No association with socioeconomic status was seen. There was a weak positive association between patients’ experience and their reports of effectiveness for all three procedures (correlation coefficient with disease-specific PROMs for hip and knee replacements 0.2 and with EQ-5D 0.1 for all three procedures). A weak positive relationship was also apparent when effectiveness was based on patients’ reports of the extent of improvement in their health. What this demonstrates is that the two domains of quality are related and that patients distinguish between effectiveness and experience. The aspect of experience most strongly associated with a better outcome was the level of communication with and trust in their doctor.
Patient reported experience was also associated with safety. An experience score 1 SD (about 1.5 on a 10-point scale) higher was associated with about 30% less likelihood of the patient reporting a complication. Similar to that seen for experience and effectiveness, the dimensions most strongly associated with safety were patients’ views of communication with and trust in doctors and, to a lesser extent, nurses.
All the relationships observed were consistent over time, between different types of patients (age, sex, socioeconomic status) and between providers. Given the relatively slight differences in the mean reported experience between providers, adjusting provider comparisons of effectiveness for differences in experience had little impact, although one provider previously lying more than 2 SDs below average (and thus subject to an ‘alert’ by regulators) was no longer an outlier.
Relationship to previous research
This is the first study to consider the inter-relationship of different domains of quality in surgical patients, using patient-level data. Although the only previous studies to look at data on hospital patients linked at the individual patient level were restricted to medical conditions (acute myocardial infarction, diabetes, COPD), they reported remarkably similar results. They too found weak positive associations between experience and effectiveness (correlation coefficients of 0.1–0.33).8–12 Studies that looked at different dimensions of experience also reported that patients’ communication with and trust in their doctor was the most important factor. Previous studies have not considered the relationship between patient reported experience and safety.
Strengths and limitations
The principal strengths of this study are that patient reported data were available on large samples that could be linked at the level of individuals. Many previous studies exploring the relationship between domains of quality in healthcare have been limited to comparing aggregated data for institutions derived from different samples of patients. In addition, this study has considered effectiveness and safety.
A potential limitation is the extent to which the results are generalisable. The study was limited to three elective surgical procedures so some caution is needed in generalising to the whole of surgery, let alone non-surgical care. Being restricted to one region of England is not thought to be an important limitation as patients were recruited from a wide variety of providers in terms of size, teaching status and urban:rural setting. Their PROM scores were similar to the rest of England.
Failure to collect data from all eligible patients will have affected slightly the estimates of patient outcome and patient experience for each of the participating providers, as previously reported analyses of recruitment bias25 and response bias26 have shown. However, any such lack of representativeness of our study samples had little or no adverse impact on the internal validity of our analyses as we have focused on within individual comparisons.
The validity and reliability of the measures used for assessing patient reported outcomes of effectiveness are well established. As regards safety, the high incidence of patient-reported complications reflects patients’ inclusion of minor postoperative problems and does not provide a valid indication of the frequency of serious clinical problems. However, that does not invalidate its use in these analyses for comparisons with patient experience measures.
The patient experience questionnaire was based on thoroughly tested instruments widely used in the NHS with good face and content validity.23 The scale and subscales we created demonstrated good internal consistency, apart from that for ‘Information on discharge’.
Implications
First, our results have implications for the concerns of some clinicians who have questioned whether the effectiveness and safety of their practice (measured using PROMs) may instead reflect patients’ views of their level of satisfaction with the experience of care.27 Some have even expressed concerns that surgeons and hospitals providing care of poor effectiveness and safety may pass undetected if their patients are content with their experience. This study provides no support for these concerns as we have shown that patients make a clear distinction between the different domains of quality. In addition, some managers, regulators and policy makers believe that the assessment of a patient’s experience is sufficient for judging the quality of care and that outcome measurement is an inessential luxury. This study confirms that patient experience cannot be used as a proxy for outcome.
The second implication concerns whether or not these findings have any practical significance and utility for providers of surgery. The answer depends on which of three possible explanations may account for the weak positive relationship detection: the association is not causal; experience influences outcome reports; outcome influences experience reports. It may be that there is no causal link and that the same factors are associated with experience and outcome (eg surgeons who are poor communicators are also poor operators). Alternatively there may be a causal relationship. It could be that a better experience is causally related to reporting a better outcome. The plausibility of this is strengthened by the finding that the dimension of experience most strongly associated with assessments of effectiveness is communication and trust in the doctors. This could be explained by good communication instilling a more realistic expectation of the likely outcome which in turn will increase the likelihood of patients regarding their outcome as satisfactory. It is well recognised that the better a patient's preoperative understanding of the postoperative processes and likely recovery pathway, the better the patient’s reports of their experience and outcome.28 If true, the implication is that hospitals providing a better experience may reap rewards from a better rating of experience and in their outcome assessment.
However, it is also possible that a better outcome will make a patient more inclined to overlook any minor concerns about their experience and more likely to report a good experience. In the absence of an intervention study (eg to see if an improvement in doctor communication is associated with better patient-reported outcomes), it is impossible to determine if there is a causal relationship and its direction.
Third, although 1 of the 11 providers studied shifted from ‘alert’ status to average, adjustment for experience had little impact on comparisons of providers’ effectiveness. This should reassure providers that their patients are not unduly influenced by their experiences when reporting on the effectiveness and safety of care. However, if providers want to maximise their effectiveness, improvements in patient experience may (if causally related) enhance their patient-reported outcome slightly.
Finally our findings suggest that further enquiry is needed to understand more about those patients who report a good outcome but a bad experience (and vice versa). These may be the very cases that could best inform providers as to where to concentrate their quality improvement activities.
Acknowledgments
The authors thank: Elaine Moss and Gemma Riley (Nottinghamshire County PCT) for providing access to their data; Reg Race, Daniel Ratchford, Mandy Moore, Lee Towndrow and Richard Gosling (QualityHealth) for preparing and providing the data; East Midlands Patient Experience Service and all the providers for collecting the data and giving us permission to analyse it; and the many thousands of patients for completing questionnaires.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Footnotes
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Editorial