Complexity

Complexity theory distinguishes between systems that are simple, complicated or complex.

• Simple systems show straight-forward cause and effect behaviour. Hitting a ball with a bat is a simple system.

• Complicated systems have many more parts but each part follows cause and effect. Sending a rocket to the moon is a complicated system.

• Complex systems are hard to model and the outcomes can be unpredictable. Healthcare systems are usually complex.24

The NASSS Framework was originally conceived to understand why many healthcare IT systems failed to achieve their objectives.21 NASSS has seven levels and uses the lens of complexity theory. This section shows the original NASSS name for each level, plus short comments about its relevance to PROMs and PREMs.

1. The condition or illness. The number of possible clinical conditions is very large, although in some specialist units, such as oncology and maternity, most patients have a single known condition. In others patients may have multiple diagnoses, or the diagnosis may not be known. When this is the case, generic measures are more appropriate.

2. The technology. The technology includes the questionnaire, data collection and reporting tools. Survey designers need to be clear about what they aim to achieve, what to measure and the end-to-end process of planning, data collection, analysis and use of results. The measures selected need to have been validated.

3. The value proposition for the supplier, purchaser and users differs. Some purchasers are fixated about choosing free measures, without understanding that these come with little or no support, training or software. Questionnaires, measures or the whole service may be charged at a flat rate (all you can eat), per response or be free. Purchasers must always consider the cost of the whole process, including hardware and software, license fees, support, training, and results analysis.

4. The adopter system comprises professional staff, patients and lay caregivers. At each stage in the process, it is important to be clear about who does what, when, where, how and why. Our experience is that patients will complete questionnaires if asked by people they respect (eg, their clinicians) and if the terms used are familiar. Also, staff will do what they are asked if the request comes from a line manager to whom they report. However, if patients or staff are not engaged, poor response rates are to be expected.

5. The organisation(s). Managers and leaders can motivate staff, provide resources and remove or mitigate barriers to use. They can ensure that the results are used to improve efficiency and the quality of care for all.

6. The wider context (institutional and societal). Most healthcare systems are managed top-down and people usually try to do what they are asked to. Sometimes, people at national or regional bodies try to pick winners and mandate their use, but this can have mixed success.

7. Interaction and mutual adaptation over time. As people, policies and technology change, so must the process. For example, a decade ago most questionnaires were completed on paper, but today most are online.

When all NASSS domains are simple (which is rare), the programme is likely to be easy to implement, on time and within budget. When many domains are complicated, the programme is achievable, but it will be difficult and likely to exceed both timescale and budget. When domains are complex, the chances of success are low. In all cases, the aim should be to work with the system, reduce complexity and avoid problems.25

Plan-Do-Study-Act

The Plan-Do-Study-Act (PDSA) approach helps implementers of any innovation think about who must be engaged at every stage.26 PDSA includes three questions to answer before starting:

• What are we trying to achieve?

• How will we measure improvement?

• What changes will we do?

The stages in the PDSA cycle are:

1. Plan. Staff at all levels and stages must be involved early on, so people know who is to do what, when, where, how and why. Support from a recognised leader or champion is invaluable, to make sure that everyone has the tools, training, time and resources required.

2. Do. Ultimately, responses are analysed by computer. Responses may be collected digitally or on paper, although paper responses have to be entered into a computer later. Ideally collect digitally at source.

3. Study. Examine results quickly and disseminate widely. Use thresholds as simple comparators. Using 0–100 scales, we use: under 40 bad; 40–59 poor, 60–79 moderate; over 80 good.

4. Act. Trust the results and act as quickly as possible. What you do depends on local circumstances, but do not ignore the findings. Our experience suggests that unexpected results are usually correct.

PDSA cycles usually run sequentially. This is helpful if a cycle suggests that the approach needs to be revised. PDSA cycles may also be run simultaneously, but this is harder to manage when changes are required.

Response rate is a success indicator which is easy to measure.27 A poor response rate usually means that potential respondents were not asked, or that they deemed the questionnaire to be irrelevant or burdensome. The following response rate bands are helpful: under 10% bad; 10–30% poor; 30–60% moderate; over 60% good.

Noise

Noise is scatter; it is always present in surveys.28 Noise is a serious problem, especially when the main result is the difference between two responses (two noisy numbers) for the same person (eg, before and after treatment).

There are three main types of noise29:

• Level noise happens when respondents are consistent in their differences. For example, some people always avoid the endpoints of scales, but others do not. Level noise also occurs when some respondents differ in their understanding of a question.

• Stable pattern noise happens when respondents prioritise different aspects of the problem. For example, some people prioritise physical symptoms and some mental symptoms.

• Occasion noise occurs when scores vary in ways that are hard to predict in advance. For example, some people respond differently when rushed or hungry.

One way to reduce noise is to limit the number of options. For example, most people are consistent about whether they agree or not with a statement. They can also say whether they agree strongly or weakly. Being non-committal does not mean agreement—it may just be a polite way of disagreeing. A short scale (eg, 4 or 5 options) creates less noise than a long scale (ten or more options).30 Always think about how you expect people to rate each question and aim for a broad spread of answers representing different views. It says little if everyone answers the same way.

When designing a survey, decompose complex, multidimensional judgements by taking one dimension at a time. It is hard to compare across different dimensions. Also, put overall assessments near the end, not at the start of a questionnaire.31

Interoperability

PROMs need to be interoperable with EHRs to help clinicians tailor care for individual patients. For example, in a randomised controlled trial of patients having chemotherapy, adding the use of PROMs between clinic visits (importing results into the EHR, reviewing and acting on the findings) led to an improvement in 5-year survival from 33% to 41%.32

There are two primary use cases for interoperability: (1) the clinician asks the survey engine to administer a specific questionnaire; (2) the EHR imports results data from the survey engine for use by clinicians and ideally patients.

Fast Health Interoperability Resources (FHIR) Release 4 (or later) is now the preferred way to do interoperability in most countries (including USA and UK). FHIR is built around resources. FHIR has two special resources for use with surveys: Questionnaire, which specifies the questionnaire itself, and QuestionnaireResponse, which specifies the answers of a specific subject (patient). QuestionnaireResponse is needed for most clinical purposes, along with resources to identify the patent, clinic and so on.

All fields in a resource are optional. Resources must be simplified (profiled) and combined with other resource profiles to create an Implementation Guide (IG) for each transaction. The best IGs have the minimum optionality and define the exact format, including what codes shall be used. Implementers use IGs, not the base standard.33

Information exchanged must be unambiguous, which means that concepts must be coded. The two main coding schemes used in many countries are the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) and Logical Observation Identifiers Names and Codes (LOINC). In 2022, it was agreed that all LOINC codes would have an exact SNOMED CT equivalent.34 Some of the concepts used in questionnaires have the codes required, but not all.

Questionnaire design

People will complete questionnaires when the following conditions are met:

1. They are asked by someone they respect.

2. They believe it will help themselves or others.

3. The questionnaires are relevant, quick and easy to complete.

When designing a questionnaire, think about the benefits to those who need to act on the results as well as to the respondents who complete them.

Scoring creates problems for people using the results when different measures use different scales, endpoints or directions. For example, the NHS PROMs scheme for Hip Replacement surgery uses measures with three different scales and endpoints:Oxford Hip Score: 0 to 48, EQ-5D-3L Index: −1.58 to 1.00, EQ-VAS (Visual Analogue Scale): 0 to 100.35

Good reporting practice is to:2

• Distinguish between cohort mean scores and individual scores.

• Use common scale endpoints such as 0 for bad and 100 for excellent.

• Avoid using more than two significant figures without good reason—say 67 not 66.7.

• Choose different items to measure different things. This can be tested using statistical tools such as Cronbach’s alpha and factor analysis.

• It may be easier to ask everyone than to pick out a representative sample. Statistical significance testing depends on the square root of the sample size.36

Range of measures

The range of useful patient-reported measures is broad. PROMs cover more than health status; PREMs cover more than satisfaction with staff. For example, the range of measures developed and used at R-Outcomes Ltd is shown in table 1.37 38

Published measures have usually been refined over several years before publication. Always make sure that you know what measures are already available before building your own questionnaire from scratch. Have your own checklist. We value brevity, readability and actionable reports.