Primary research

Primary research involves synthesising data, observations and predictions contributed from selected participants, based on their knowledge and perspectives of the context. This may be supported using methods, such as reflective practice,14 action learning15 16 and other forms of questioning. It is around a sequence of key questions that the OIL process described in this paper is structured.

Secondary research

Secondary research into outcomes, including product design, often involves designers adapting a priori lists of generic outcomes, such as the National Health Service (NHS) outcomes framework,8 or other standard outcome sets.10 As Nagendran et al9 highlight in their critique of this approach, “[this] framework will need considerable ‘deconstruction’ in its application locally into actionable structure and process measures that will support quality and outcomes improvement”. The tools presented in this paper support this local adaptation.

The OIL process for building impact models

Iteration of the OIL process

The OIL questions

The knowledge and understanding required to produce the impact model lie with those people who interact with it. In healthcare, this means patients, staff, managers and others. ‘Mapping out’ the system relies on asking participants the right questions in the right way, or, alternatively, asking these questions in the thematic analysis of transcript data, reviews and other sources of insight.18–20

The questions that form the outcome identification process described here have emerged from the analysis and abstraction of existing methods,21–37 that define systems in terms of outcomes, stakeholders and their analogues (these methods are reframed within the OIL process, as described later). The questions are:

1. The stakeholder identification question. Referring to an outcome:‘Who (else) might this affect?’.

2. The stakeholder impact question. Referring to a stakeholder:‘What (else) might affect them?’.

3. The system analysis/design question. Referring to an outcome:‘What (else) might affect this?’.

4. The consequence (intent/risk) question. Referring to an outcome:‘What (else) might this affect?’.

Rather than attempting to consider the system as a whole, these questions subdivide the process, focusing participants on each outcome or stakeholder in sequence. Focus can be further subdivided by emphasising analysis or intent in each question. For example, question 3 can be posed as either a design question: ‘How might we affect this?’ or as a system analysis query: ‘What do we know to affect this?’. Similarly, question 4 can elicit intended consequences alongside perceived risks. Both intent and analysis are valid in the impact diagram.

As these questions are answered, outcomes and relationships may be added to the impact model and stakeholders can be identified. While question 3 elicits antecedent outcomes (branching to the left on the impact model) and question 4 elicits consequent outcomes (drawn to the right), questions 1 and 2 help to identify accompanying outcomes. Figure 3 illustrates how these questions can be used to grow an impact diagram from any given starting point.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Impact model proliferation during questioning.

Concatenating these questions in iteration

At any point, at least one of these questions can be applied to the content of the nascent impact diagram. For a more balanced and coherent process, it is possible to concatenate the questions (as illustrated in figure 4). Akinluyi determined that by answering these questions in sequence, considerations of the system and its stakeholders could be interlaced, unifying multiple outcome-identification activities within one process.12

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

The outcome identification loop—Adapted from Akinluyi’s outcome identification process12 (with permission).

Using OIL to frame the use of other techniques

The OIL questions may simply be posed to the participant(s) as they are. To enrich the process, however, one can use them as a basis to select and integrate techniques that pose questions in different ways.

Questions 1 (‘Who (else) might this affect?’) and 2 (‘What (else) might affect them?’) are a paired abstraction of core questions from stakeholder identification and analysis techniques. Literature on this topic in several fields emphasises both its importance21 22 and its need for improvement.23 24 Methods to support these questions include participatory techniques, such as contextual inquiry,25 26 and visualisation techniques, such as user stories or personas.27 Stakeholders are otherwise commonly identified through ‘brainstorming’—a process facilitated here by the piecewise consideration of a system’s outcomes rather than the less-tractable picture of the overall system.

The checklist is another popular tool for stakeholder identification. A priori lists of stakeholder groups can be adopted or adapted to generate more specific lists. Some QI approaches are highly focussed on particular stakeholder groups—consider, for example, singular emphasis on ‘the customer’ in Lean thinking and analogously on ‘the patient’ in some value-based healthcare approaches.28 Leviton and Melichar,22 however, ‘urge QI evaluators to think more broadly about the stakeholders they consult’. Less-constrained approaches like theirs or that of Concannon et al29 support more complete system exploration.

Question 3 (‘What (else) might affect this?’) relates to system analysis and design. It performs a function analogous to that of the ‘(solution-neutral) problem statement’30 which seeds many design exercises. This question is also the basis of the ‘driver diagram’ and its analogues.31 32 Within an impact model, the content of these diagrams can be combined with chains of outcomes elicited using analysis methods, such as process modelling,33 and root cause analysis.34 35

Question 4 (‘What (else) might this affect?’) can be answered using analysis methods, including those intended for proactive hazard/risk analysis, such as Failure Modes and Effects Analysis and the ‘Structured What if Technique’.36 37 However, it is important to additionally consider desirable-yet-unintended consequences, which some of these techniques are less attuned to.

Filtering, converging, iterating and stopping

As figure 4’s loop implies, it is possible to continue elaborating the impact model almost indefinitely—stopping criteria are required. As a minimum, the process should ensure that stakeholders and outcomes of the greatest concern have been identified. The impact model should contain, on the left-hand side, those outcomes directly influenceable through QI intervention and on the right-hand side, outcomes experienced directly by stakeholders. Beyond this, OIL can be used to elaborate the impact model and test its validity until the end user judges it to be sufficient.

To focus the OIL process on the outcomes of greatest concern, it is important to prioritise the content of the impact model periodically, filtering out unnecessary detail. This means assessing the relative importance of outcomes as they are identified. Outcome valuation in complex systems is not a trivial task and it is subject to the limitations of ‘bounded rationality’.38 To compensate for this, it might be necessary to revisit and update impact models in multiple outcome-identification exercises or workshops. It is possible to accommodate impact model iterations within QI cycles, such as Plan-Do-Study-Act iterations.39

Completing the SIM: criteria for setting the system boundary

Healthcare is complex, with outcomes emerging in different contexts, to various stakeholders. In the UK NHS, for example, where services are free at the point of access: costs, clinical outcomes and patient experience might appear quite removed from one another, but they all constitute value.

While some outcomes emerge within a system as intermediate factors and others emerge coincidentally, the most meaningful outcomes are those which we purposefully deliver or avoid. They are located at the boundary of the system being designed in QI and collectively model the value of the system to a given stakeholder group. As shown in figure 5, boundary definition in an impact model literally involves connecting sets of these ‘boundary outcomes’ of interest. This process completes the SIM for a stakeholder group.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Flexibility in boundary outcome combination in system impact modelling. QI, quality improvement.

As described by Akinluyi,12 boundary outcomes are ideally:

• Directly experienced by stakeholders.

• Mutually exclusive; collectively exhaustive.40

• Quantifiable.

• Commensurable.

Boundary outcomes are directly experienced by particular stakeholder groups and different groups might require separate boundaries to reflect their interests. This underlines the importance of including and prioritising among stakeholder groups with an interest in the outcomes of a QI exercise.

Sets of boundary outcomes that are mutually exclusive are easier to resolve from one another—though this is not a strict requirement, it makes value models easier to understand. It is more important, however, that boundary outcomes should be collectively exhaustive, in the sense that they should fully account at some level for the impact of the QI initiative. It is worthwhile checking an impact diagram to ensure that meaningful outcomes have not been omitted. In figure 5, collective exhaustiveness is represented by boundaries that span the impact model from top to bottom, though this may depend on the arrangement of outcomes in the diagram.

For quantitative value modelling, or in QI monitoring, the outcomes that affect stakeholders are ideally directly quantifiable and also comparable with one another in terms of their desirability (ie, commensurability41). Boundary outcomes may not always be measurable directly, however, and this can be problematic in some QI approaches. An alternative approach, and one that is advocated here, is to strictly maintain that boundary outcomes are experienced by stakeholders, whether measurable or not, and that where an outcome cannot be measured directly, associated outcomes are used as a proxy. These proxy outcome measures can be seen as ‘indicators’ of the boundary outcome and can then be interpreted more honestly in the context of the SIM.

As a general rule, outcomes closer to the left-hand side of the impact diagram are more measurable. They are also relatively context-specific and less generalisable. Conversely, outcomes on the right-hand side are more relevant to stakeholders and support more general, transferrable models of value linked to corporate shared purpose; it is here that the system boundary lies. While it also helps if these right-hand side outcomes are mutually exclusive, quantifiable, measurable and commensurable, it is more important that the value model retains its overall integrity and relevance.

Applying the SIM

The primary purpose of the SIM is to identify outcomes that form objectives in QI. Subsequently, these outcomes form the building blocks of more sophisticated value models and QI evaluation criteria.12

The diagram also captures valuable strategic information that can be used when initiating QI. For example, if a diagram exposes several factors that systematically confound the delivery of value to patients or other stakeholders, this can trigger a QI intervention to mitigate this effect. Where QI projects are initiated to address a particular boundary outcome, such as ‘reducing number of attendances’,42 the SIM provides a repository of knowledge about current and future mechanisms for effecting this change.

Another major benefit of the SIM is that it contextualises operational activity. As well as giving participants an opportunity to be heard, the SIM engages staff by demonstrating their contribution to the system. This is the case in the example produced by a radiation safety team described in box 1 and shown in figure 6.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

System impact model produced by a radiation safety team.

Although only the example of box 1 is presented here, in subsequent adaptations of this method, similar benefits have been noted. SIMs have been produced in workshops at Guy’s and St Thomas’ NHS Foundation Trust to explore the impact of its clinical engineering service and, in other work, to develop a strategy to improve nursing staff retention. SIMs have also been produced in workshops with King’s Health Partners’ ‘Education Academy’ to explore the impact of training and development initiatives.

Workshop overview

The SIM was seeded using statements of core purpose that the participants had produced: they initially identified ‘(increase) image quality’, ‘(lower) patient dose’ and ‘(lower) staff dose’, alongside a list of activities that nominally fall within the remit of the radiation safety team (see left-hand side, figure 6).

Participants were then divided into small groups undertaking one of two tasks: some used driver diagrams and other methods to identify antecedent outcomes, and others used communication diagrams to identify consequent outcomes. Antecedent outcomes were elicited during a discussion about encounters with patients and staff.

Insights derived immediately from OIL and the SIM

‘Patient assurance’ was identified as a key outcome during the course of the workshop—and this triggered a discussion about how ‘quality of patient information’ might be the key to addressing patients’ concerns. This exchange demonstrated that, even in the initial draft, the SIM could be used to inspire new QI interventions.

The SIM also highlighted some dependencies that the team decided they would like to better understand or influence. For example, one senior physicist made the observation:

‘We would like to think that by adding quality control processes, we optimise patient (radiation) dose. Actually, it’s the radiographers that can apply or ignore our recommendations, and so there’s a big dependency there—it’s important that we let the radiographers know why changes are being made. The diagram perhaps shows the complexity that we have to deal with.’

This comment also reflects the benefit of considering a range of stakeholder outcome boundaries.

Analysis of the SIM

Figure 6’s SIM illustrates the fact that the team is driven, to a degree, by high-level organisational goals. It is also apparent that their service impacts patients in numerous ways that often depend on how they engage and interact with internal stakeholders, such as radiologists and radiographers. Reconciling stakeholder perspectives with an overarching mission can help to identify opportunities for QI through improved collaboration—as was exemplified here, in the discussion on patient information.

Reception of the SIM

The utility of the SIM was positively received:

‘This representation is like a Swiss-army knife—we can work backwards from the right side to understand how we can make changes … starting on the left, we can think about what we’re doing now’ (where ‘the right side’ corresponds to the organisational outcome boundary in figure 6 and ‘the left side’ corresponds to radiation safety interventions).

The process empowered the team to challenge what they were doing and to identify opportunities to strengthen the evidence base for their decisions and improve outcomes (by identifying poorly understood relationships within the SIM). Although it could stimulate reflection in this way, the SIM did, however, lack detail in its representation of causal relationships between outcomes:

‘It would have been useful to apply some sort of weighting system to know the strengths of these factors’.

The quantitative analysis of outcomes features in the wider value-modelling framework by Akinluyi,12 but was decidedly excluded from the scope of the SIM—however, it was acknowledged that it is possible and sensible to add more information to SIMs in future, by at least assigning +/− signs (to indicate whether causal relationships are incremental/decremental, respectively).

A major benefit perceived by participants was that the diagram linked procedure to purpose, moving their focus away from the operational task and desk/workshop, towards the interests of, and interaction with, other stakeholders. The process of constructing the SIM helped them to articulate their contribution.

All of this was achieved by asking the OIL questions in such as a way as to access and mobilise the considerable knowledge of the team.