Health in childhood contributes to adult health,1^–3 and improving the quality of healthcare for UK and US children is now viewed as critical.4 5 An essential component of any improvement effort is the identification of specific, measurable indicators of quality. Quality indicators, also known as performance indicators and review criteria, are systematically developed statements that can be used to assess the appropriateness of specific healthcare decisions, services and outcomes.6 These indicators are developed as a first step in a quality improvement effort and are ideally drawn from the available scientific evidence.

There is evidence that the healthcare system is underperforming for children and there is large variation in care. A study conducted by the RAND Corporation showed that US children received less than 50% of overall indicated care in the outpatient setting.7 A national survey also found that paediatricians used more than 100 different practice guidelines, but no single guideline except for asthma was used by more than 27% of paediatricians.8 Recent reports have found widespread variation in hospital services available to children9 and the availability of paediatric cardiology services10 in the UK.

In this review, we present examples of widely available quality indicators and their use in quality measurement efforts in the UK and USA. It is not our intent to comprehensively review all indicators; rather, we will use examples from prominent agencies to illustrate the current state of quality indicators and quality measurement. We will also discuss the role of health information technology (HIT) in the measurement of quality indicators. Finally, we will outline the areas requiring additional work in indicator development in order to improve healthcare provided to all children.

THE QUALITY MOVEMENT IN INDUSTRY AND HEALTHCARE

The quality movement in the 20th century stemmed from efforts to produce high quality products from increasingly complex industrial processes. Three theorists are credited with the early development and dissemination of quality assurance and improvement methodologies in manufacturing: Walter A. Shewart, William E. Deming and Joseph M. Juran. In the 1920s and 1930s, Shewart and Deming developed the concept of statistical control of processes (ie, common cause and special cause variations), the statistical process control chart to manage and improve processes, and the Plan–Do–Study–Act (or plan–do–check–act) cycle, a simple method to test information before making a major decision or change.11 12 Juran overlapped briefly with Shewart and Deming at Western Electric in the 1920s and is credited with expanding the Pareto principle through its application to quality (ie, 80% of a problem is caused by 20% of the causes) and the development of the quality improvement method, “Total Quality Management”.13 Many have applied these concepts to the medical field, including Donald Berwick and the Institute for Healthcare Improvement.14 15

The Institute of Medicine defines healthcare quality as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge”.16 To achieve high quality healthcare, there must be a framework to structure quality improvement efforts, a scientifically-based system to define best practices, demonstration of variation in the quality of care provided, and the means to monitor the effectiveness of interventions—all of which exist today. In 1966, Donabedian published his seminal work outlining the “structure/process/outcome” quality improvement paradigm which is used routinely today.17 Evidence-based medicine (EBM) provides the scientific basis for defining high quality care and is produced through efforts from groups such as the Cochrane Collaboration and the National Institute of Health and Clinical Excellence (NICE) in the UK, and the Agency for Healthcare Research and Quality (AHRQ) and specialty societies in the USA. Improved health services research methodologies have revealed wide variations in the care provided to children in the UK9 10 and the USA.18 Finally, advances in HIT have given researchers the ability to measure and monitor quality.

DEFINITION OF QUALITY INDICATORS

Quality indicators are explicitly defined and measurable items referring to the structures (eg, the environment care was provided), processes (eg, did the patient receive care) or outcomes of care (eg, mortality).19 20 Desirable characteristics of quality indicators include: unambiguous descriptions and clear definitions of variables to be measured; explicit definition of the population to be included and the setting to which they apply; and links between process measures to health outcomes for the care being reviewed.21 Targets monitored by these indicators should be specific, measurable, achievable, relevant and time-specific.22

Development of quality indicators

Whenever possible, indicators should be based on strongest scientific evidence available (eg, randomised controlled trials).21 23 However, many areas of healthcare have a limited evidence base, therefore systematic methods have been developed that combine evidence and expert opinion. These methods include the consensus development conference (eg, use of hydroxyurea in sickle cell disease24), guidelines developed by iterated consensus rating procedure (eg, NICE guidelines for diabetes25), the Delphi technique (eg, UK prescribing practices in general practice26) and the RAND/UCLA Appropriateness Method (eg, very low birth weight follow-up care27).

Quality indicators versus clinical practice guidelines

Clinical practice guidelines differ from quality indicators in that they are systematically developed statements designed to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances.28 Guidelines are often less specific than quality indicators and may not provide sufficient detail to support the actual measurement of recommendations. They are also used prospectively whereas quality indicators often assess care retrospectively.29 For example, the National Heart, Lung and Blood Institute and National Asthma Education and Prevention Program released guidelines in 2007 for the diagnosis and management of asthma, which recommended the identification of asthma triggers to educate patients to avoid unnecessary exposures or alert them to exposures that might require increased treatment.30 In contrast, a quality indicator for asthma developed by RAND recommended that all patients diagnosed with asthma should have an evaluation of possible triggers within 6 months of diagnosis.31 The quality indicator clearly defines the population, timeframe and the specific goal to be measured.

Process–outcome relationship

Process data often provide the basis of quality indicators because they are more easily measured. Processes are also more sensitive measures of quality than outcome data because outcomes are only partially produced by health services and are frequently influenced more by other factors (eg, natural history of the disease, patient physiological reserve, or patient age).32 Processes are likely to be under providers’ control and thus can be improved through individual training or systems change. Finally, a poor outcome does not occur every time there is an error in the provision of care.20

To date, studies attempting to show that improved processes lead to better outcomes have produced mixed results. Better performance on process quality measures was strongly associated with better survival among community-dwelling vulnerable adults,33 and 100% adherence to a set of quality indicators was significantly associated with better overall survival with breast cancer.34 However, acute care processes for stroke were not associated with functional outcome at 12 months,35 and mixed results have been reported on the relationship between HbA_1c levels and mortality.36^–38 These studies highlight the difficulty and complexity in selecting appropriate processes to measure and the need for additional studies to examine this relationship further, especially in children.

EXAMPLES OF QUALITY INDICATORS DEVELOPED FOR CHILDREN

For the purposes of this review, we searched governmental healthcare and accreditation agencies in both the UK and USA (eg, National Health Service (NHS), AHRQ) as well as private foundations (eg, RAND, Cystic Fibrosis Foundation) to identify examples of quality indicators developed for children.

Quality indicators developed by UK and US government agencies

In the UK, the government introduced the Quality and Outcomes Framework (QOF) as part of the new General Medical Services contract in 2004.39 Currently, the QOF is organised into four domains: clinical, organisational, patient experience, and other services. The clinical domain consists of 80 indicators covering 19 different clinical areas; less than 20% of these indicators apply to children. The National Service Framework for Children was released in 2005 and provides standards of care rather than quality indicators.4

In the USA, AHRQ developed 18 indicators for inpatient care provided to children, including asthma, nosocomial infections, and selected postoperative complications.40 41 In addition, the Joint Commission on Accreditation of Healthcare Organizations has worked with the Centers for Medicare and Medicaid (CMS) and the National Quality Forum to develop quality indicators to provide data about the best treatments or practices for hospital-based care, and mandates annual reporting by healthcare organisations for accreditation. The Joint Commission, in conjunction with the Pediatric Data Quality System Collaborative Measure Workgroup (Pedi-QS), developed indicators to review the delivery of inpatient asthma care42 and care provided in the paediatric intensive care unit.43 Finally, CMS established the Physician Quality and Reporting Initiative (PQRI) in 2007.44 This programme is a voluntary quality reporting programme that provides financial incentives to eligible professionals to provide data on over 100 different indicators and delivers electronic feedback reports to those that participate. Few of the indicators in this programme are specific to children.

Quality indicators developed by private foundations/organisations

In 1993, the National Committee for Quality Assurance (NCQA) introduced the Health Plan Employer Data and Information Set (HEDIS), which contained performance measures that were used in the assessment and licensure of managed care plans. Currently, NCQA publishes annual “report cards” on over 90% of US health plans for outpatient care using performance measures included in HEDIS. The 2008 dataset consists of 71 measures in 8 domains of care, with 25% of indicators specific to children.45

The RAND Corporation has been instrumental in the development of specific methodologies to develop quality indicators, namely the RAND/UCLA Appropriateness Method (or the Modified Delphi Method).46 In 2000, RAND published a comprehensive set of over 400 outpatient indicators for children and adolescents in the USA,31 and used them to analyse the state of healthcare delivered to children.7 In addition, they developed a set of 70 indicators for the follow-up care of very low birth weight infants (<1500 g) after discharge from the neonatal intensive care unit.27

The Institute of Healthcare Improvement (IHI) is a leader in the development and implementation of quality indicators in collaborative quality improvement efforts at institutions across the USA and Europe. In 1995, IHI developed a series of collaborative projects called the Breakthrough Series to address issues such as reducing caesarean section rates while maintaining or improving maternal and fetal outcomes; reducing costs and improving outcomes in adult cardiac surgery; and improving prescribing practices.15 Presently, IHI has paediatric indicators for HIV and asthma care.47 48

Finally, private foundations have developed quality indicators to improve care for special diseases or conditions. For example, the Cystic Fibrosis (CF) Foundation has seven goals to improve cystic fibrosis care in their sponsored care centres, including 14 quality indicators.49 The CF Foundation efforts are unique in that they provide information systems and financial support to operationalise the collection of data and indicator measurement. There is interest in developing quality indicators for other common chronic conditions in paediatrics, such as type 1 diabetes and inflammatory bowel disease.

CLASSIFYING QUALITY INDICATORS FOR CHILDREN

Table 1 provides a listing of quality indicators described previously, organised by setting. From this list, less than 5% of indicators are devoted to inpatient care. Even though some are applicable to children, most of the indicators developed for the inpatient setting are not unique to children. For example, “foreign body left in during procedure” and “accidental puncture or laceration during surgery” are relevant to quality healthcare for adults as well as children.

Quality indicators can be further organised by function, such as screening, diagnosis and treatment (see table 2). Less than one-third of indicators assembled for this paper focus on preventative care (ie, screening and prevention), while nearly two-thirds are devoted to diagnosis and treatment of medical conditions. However, most of the care provided to children in clinical practice has a preventative focus.

Finally, we organised the indicators by Donabedian classification (see table 3). None of the paediatric indicators included in this paper measure structural data, while more than 95% focus on care processes. In addition, none of these indicators measure school achievement or consider the family context, both of which are important contributors to child health.4 50

QUALITY INDICATORS IN THE INFORMATION AGE

Data sources for quality indicators

The implementation and monitoring of quality through the use of indicators depends on the availability of discrete and accurate data. There are three primary sources from which data may be extracted for indicator measurement: paper charts, claims or administrative data, and electronic health record (EHR) data. While many have hoped for a seamless, automated approach to quality measurement through electronic systems, each of these systems has advantages and disadvantages.

Paper medical charts are still in widespread use in the USA and are capable of providing detailed information about the care and timing of treatment that a patient received. While it is possible to extract meaningful data from paper charts, extracting these data is costly and often requires specially trained individuals. The time required to gather data often limits the feasibility of performing large-scale projects. Missing data are also a concern, since the chart primarily reflects the documentation of care an individual receives from a certain provider or practice and may not contain a comprehensive set of data or data about care provided elsewhere.

Electronic claims data are widely available, particularly in the USA, and can be useful because they contain data on large numbers of patients and provide data on both the utilisation51 and cost52 of care provided. There is clearly an incentive to submit comprehensive claims data to maximise reimbursement. However, claims data are usually limited to diagnoses and procedures; do not provide longitudinal information as enrolees change insurance policies or employer; and only capture those services covered by the health plan or system, and therefore, may not truly reflect all care received by an individual. Claims databases can also be expensive for researchers due to fees charged for their use.

The EHR can be a very useful tool for monitoring quality indicators. Data from EHRs have the potential to most closely describe the actual care delivered to patients, and the use of automated systems has the potential to deliver much more information at lower cost than that available from manual review of paper charts. Currently, EHR systems are not yet optimised for quality improvement. For example, there are conflicting reports about the effectiveness of EHRs for quality measuring purposes. A recent review of HIT and EHRs demonstrated improved quality and efficiency among four healthcare institutions in the USA who developed their own systems,53 and use of EHRs was associated with improved quality of care for children in an urban ambulatory practice.54 However, a large retrospective cross-sectional analysis of National Ambulatory Medical Care Survey data in 2003 and 2004 showed that use of EHRs was not associated with better quality ambulatory care.55

CURRENT HIT SYSTEMS IN THE UK AND USA

The National Programme for Information Technology (NPfIT) is a £12 billion (US$21 billion) component of a ten-year investment in HIT commissioned by the NHS.56 A key element of the NPfIT is the NHS Care Records service, an integrated system that will provide important elements of a patient’s clinical record electronically throughout England.57 The programme’s infrastructure and broadband networks are scheduled to be fully operational by 2014.58 However, a 2007 report on the NPfIT found a two-year delay in the piloting and deployment of the electronic patient record; IT suppliers to the programme are struggling to deliver; clinicians are uncertain about the NHS’s ability to fulfil its promises; and for the local NHS, the total cost of the programme and the value of its benefits remain uncertain.59 There is also concern that those responsible for designing the new system are unaware of the potential research uses of routinely collected healthcare data.60

In the USA, information systems for healthcare are fragmented across insurers, healthcare organisations and providers. The USA lags as much as a dozen years behind other industrialised countries in HIT adoption58; a recent study showed that only 13% of US physicians in ambulatory care reported using a basic system, and 4% reported having an extensive, fully functional electronic-records system.61 In 2004, President Bush established the Office of the National Coordinator for Health Information Technology (ONCHIT) to promote HIT, but funds were not appropriated for it by Congress at that time.58 In 2009, Congress passed the American Recovery and Reinvestment Act which appropriated $19.2 billion to support widespread deployment and utilisation of HIT and the availability of an EHR for all US citizens by 2014.62 Included in this appropriation was $2 billion in funding for the ONCHIT.

PATIENT PRIVACY AND PROTECTION

Patient privacy is a key concern in any quality measurement effort. In the UK, legislation protecting patient privacy is complex. In a recent report, the Council for Science and Technology recommended a detailed legislative review and possibly new legislation to allow the use of personal data for researchers and statisticians, especially as the NHS Care Records Service becomes reality.63 Interestingly, the British public did not seem to consider confidential use of personal, identifiable data as an invasion of privacy in a recent survey.64

In the USA, the Health Insurance Portability and Accountability Act (HIPAA) of 1996 created the Privacy Rule, which permits healthcare provider organisations (“covered entities”) to disclose individually identifiable health information (called protected information) for research purposes only if the researcher has obtained written authorisation from each patient or has obtained a waiver of authorisation requirement from an institutional review board.65 HIPAA rules implicitly require that the amount of protected information released is the “minimum necessary” for a specific project.66 Since implementation, many researchers have expressed concerns that HIPAA has a negative influence on research and quality improvement efforts.65 67 68 The balance between patient privacy and research will continue to be an issue in the USA as systems evolve.

DISCUSSION

We are in the midst of a transformation in healthcare stemming from the convergence of the quality movement and use of quality indicators, use of EBM in routine practice, and the potential (USA) or implementation (UK) of interconnected health information systems. Advances in quality healthcare for children are evidenced by the growing library of quality indicators developed by prestigious institutions in the UK and USA.

However, there are gaps in quality measurement and quality indicator development in areas important for child health. First, the majority of indicators are devoted to routine care provided in the outpatient setting, yet nearly 40% of US healthcare spending for children in 2004 was on inpatient care compared to 28% for physician/clinic visits.69 Second, few indicators have been developed for children with special healthcare needs. In 2000, 15.6% of US children younger than 18 years had a special healthcare need yet they accounted for 34% of total healthcare costs.52 Quality indicators are needed for this population of children. Third, there are few quality indicators focused on educating parents about fundamental child-rearing topics such as safety and child development. Education of caregivers in these areas may prevent accidental injury and allow early identification of potential learning or behavioural problems. Finally, the indicators surveyed for this paper do not address the greater social context of childhood, including family functioning and school performance, which may be amenable to intervention and increase the likelihood that a child will be a productive member of society as an adult.

The likelihood that quality indicators will improve care delivery and health outcomes is dependent on many factors. Implementation of quality indicators often leads to increased adherence to those measures (ie, Hawthorne effect: what gets measured gets improved). If adherence to a quality indicator is also highly correlated with an increase in desired health outcome, then increased adherence to this indicator may result in improved health. However, the degree of improvement may vary across different patient populations due to issues such as general health, co-morbidities, and genetic and environmental factors. Therefore, proving the process–outcome relationship can be difficult.

At this juncture, we propose several recommendations to advance the quality indicator development agenda for children. First, the library of quality indicators for children needs to be expanded, particularly in inpatient care and in chronic care, and made available to the child health community in an integrated, easy-to-use format. Second, continued support of integrated, comprehensive HIT efforts from government and healthcare agencies is necessary to support quality measurement and ultimately provide evidence for the process–outcome relationship and the development of better quality indicators. Finally, the science and tools necessary to measure quality and develop quality indicators should be taught to more healthcare professionals to allow widespread integration of quality efforts into routine clinical practice.

The 20th century saw significant decreases in child mortality following the introduction of immunisations and the use of antibiotics for common childhood illnesses. The science of quality measurement and indicator development combined with a growing scientific evidence base and integrated information systems in healthcare may prove to be the next leap forward in improving child health in the 21st century.