• accreditation
• anaesthesia
• attitudes

All healthcare systems show variation in the quality of care provided, whether that means access to primary care services,1 ambulance response times,2 Accident & Emergency waiting times3 or treatment processes and outcomes.4–6 Monitoring this variation in quality can serve multiple purposes: informing patients about where best to seek care;7 allowing clinicians to compare their performance with that of their peers and thus identify targets for local-level quality improvement efforts, and supporting the development of national policy. Though, what all these have in common is a trust in the reliability of the data to adequately reflect healthcare quality—sometimes a questionable assumption.

In BMJ Quality and Safety, Hofstede et al 8 have addressed a common situation where providers (such as hospitals, general practices or community teams) are ranked according to their performance on a quality indicator. Rankings are used often to make direct performance comparisons between providers and used to identify positive or negative outliers. Yet, one of the downsides of this approach is that the ranks of providers can be susceptible to chance fluctuations in the indicators. The precision of rankings, that is their reliability, therefore has to be carefully assessed when developing these kinds of approaches to reporting the quality of care. This is particularly the case when payment is linked to performance,9 or when ongoing quality improvement efforts might be undermined by measurement errors.

Performance measures are driven by patient case-mix, differences in care provided and chance variation, with their accuracy to reflect on real quality variation determined by two components.10 The first is the reliability of the indicator for each healthcare provider. This component (the ‘within-provider uncertainty’) is highly dependent on the number of patients receiving the type of care in question at each provider and is likely to be affected by random variation especially in smaller population groups. The second component is the variance in the indicators between providers. This ‘between-provider uncertainty’ relates to the true variation in the indicators between providers, setting aside chance variation within the individual providers. These distinctions are relevant because the reliability of the ranking system will depend on both the within-provider and between-provider uncertainty.

One way to combine these two sources of uncertainty is to measure the ‘rankability’, defined as the ratio of between-hospital variation and the sum of between-hospital variation and within-hospital variation multiplied by 100.10 This calculated percentage describes the level of variation due to true hospital differences, as opposed to random noise. Low values for this percentage imply that variation in performance across hospitals largely reflects chance, not true differences in performance. Referring to this situation as having low rankability conveys the idea that hospital rankings are unstable: chance variation could just as easily have produced quite different rankings. By contrast, high values for rankability mean that most observed variation in performance reflects real differences between hospitals – any given ranking is thus quite stable.

Hofstede et al 8 examine whether it is possible to improve the reliability of rankings based on quality measures. Two strategies are assessed: combining indicator data across several years to increase the number of events (eg, reporting readmission rates based on the number of admissions occurring over a multi-year period rather than a single year) or generating a composite measure by combining information from two or more quality indicators. Both approaches might improve the rankability – yet with some downsides in terms of the usefulness of the quality indicators, as we will return to discuss.

Hofstede and colleagues make use of Dutch National Medical Registration data for over half a million patients treated in 95 hospitals, containing indicators for in-hospital mortality, length of stay and 30-day readmission rates across 12 years. The authors considered a rankability ratio below 50% as low, between 50% and 75% as moderate and above 75% as high. Findings from the analysis show that both strategies--collecting individual indicators over multiple years or combining multiple indicators into a single composite--can significantly improve rankability compared with the use of any single outcome measure. Yet, composite measures showed greatest reliability of rankings in this study, and the authors conclude that composite measures provide more information and more reliable rankings than combining multiple years of individual indicators. But of course, there are other considerations which we now address.