Identifying value in healthcare transformation initiatives: an evaluation of an approach to benefits realisation

Problem

Benefits realisation management (BRM) was first introduced in Information Technology (IT) systems programmes in the early 1990s1 and has since become an important part of transformation within complex healthcare systems, in addition to standard programme management.2 National Health Service (NHS) Digital has defined BRM as ‘the activity of identifying, optimising and tracking expected benefits from the change initiative’.3

Improving service, quality and patient outcome is an important focus for healthcare providers.4 Over the last decade, healthcare systems have increasingly used quality improvement (QI) techniques within local service redesigns and transformation projects.5 However, the inability to identify and achieve intended benefits through change initiatives appears to be common and scarcely reported.6 7 The challenge is complex and has been attributed to a number of factors including a lack of, or even total absence in some cases of a robust benefits management system. Funding required to undertake complex programmes of projects is also typically awarded upfront, while the benefits will only accrue once the initiative is developed and embedded. Benefits measurement of initiatives that did not implement during the funded period of the project may be further challenged where further funding is required to sustain them.8 Additionally, healthcare transformation projects often comprise a large diversity of stakeholders, of who those responsible for delivering the benefits are often different to those overseeing the management of the transformation itself.9 The importance of identifying stakeholders integral to management of the measurement and realisation phase cannot therefore be understated.10

Various approaches, tools and templates of BRM have been developed since its inception. For example, the BRM Framework guidance outlines the purpose of benefits realisation and its key foundations.10 The approach is broadly structured in two phases. The first states to ensure that identification of key benefits to be measured is discussed with key stakeholders on the project team. This is based on the premise that higher levels of BRM can be achieved when all the key stakeholders associated with the project contribute to the benefits discussion. In addition to benefits, it outlines the need to define the outcomes and enablers required to achieve each benefit. The concept of a benefit itself has been described as ‘an outcome of change that is perceived as positive by a stakeholder’.11 The second phase of the framework relates to outlining the measurement and reporting requirements for each benefit. This includes detailing how the benefit will be measured once realised and reporting of baseline measurement.

Another important aspect of benefits management is identifying and managing disbenefits. That is, a measurable decline resulting from an outcome perceived as negative by one or more stakeholders. Disbenefits can inform understanding of what stakeholders perceive as negative consequences of change.11 However, disbenefits management is often overlooked by project teams and identified as they emerge later on in the project cycle. Guidance for benefits realisation states that disbenefits should be mapped earlier on in the project in the same way benefits are predicted based on the project objectives. Recognising disbenefits and the weighting on these in comparison to benefits is considered an important determinant in sustainability of the change implemented. A dynamic approach to benefits management that prioritises identification of benefits as well as disbenefits can also help to inform communication plans and realisation planning.12

Previously, project success was usually assessed using key performance indicators designed to evaluate the adherence to budgets, schedules and technical specifications.13 More recently, there is increasing support that data collected from patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) has a number of uses, including in transformation and QI projects. PROMs can be validated tools and questionnaires that aim to ascertain patients’ views of their symptoms, their functional status and their health related quality of life.4 PREMs on the other hand, capture the impact of the process of care on the patient’s experience. Such measures have been described as valuable in improvement and focus patient-centred clinical management.4

Value-based health services comprises achieving desirable health outcomes that are important to patients; using outcomes that are cocreated with patients and their caregivers. These outcomes would measure a full cycle of care, rather than reporting for a single service or intervention.14 Authors have also suggested that the concept of a benefit and the ability for projects to demonstrate value are related, but separate domains. Projects are often considered complete when their deliverables have been achieved.7 However, project benefits are typically realised over time and in some cases a benefit may only be achieved once the project initiatives have been implemented for a minimum period of time.15 A subsequent risk is the potential absence of responsibility to manage the realisation phase. Further, QI initiatives often lack a clear outline to link initiatives directly to intended outcomes. Defining these intended relationships (cause and effect) prioritises the list of these interventions and support for subsequent monitoring and effectiveness.16 Maximising the potential of the benefits delivered such as through utilisation of risk management and effective project planning may also increase the value of the benefits reported.17

There is a strong indication in the literature to the importance of effective BRM. Yet, there is a dearth of empirical research and reporting on applications of BRM, particularly in the healthcare sector. With that in mind, the aim of this QI paper is to report on the identification of benefits and metrics across seven transformation projects using a benefits realisation framework.

Background

Seven newly funded clinical-academic transformation projects (across cardiovascular, respiratory and critical care subspecialties) commenced across three UK-based NHS organisations in early November 2020. In order to demonstrate outcomes that support strategic objectives and justify awarded funds, there was a need to create a benefits realisation plan as early as possible. Prior to commencement of these transformation projects, a benefits mapping template (figure 1) was codesigned alongside clinicians, with a view to use across partnership projects. The purpose of the template was to help facilitate the identification of benefits and relevant metrics directly associated with the change initiatives of each transformation project. The development of the template was directly informed by the BRM evidence base in the literature. The template was critically appraised and iterated prior to utilisation with projects. Given the importance of benefits realisation within transformation projects, our aim was to complete a purpose-designed, benefits mapping template across seven transformation projects within the first 3 months of project initiation. The scope of this paper does not intend to evaluate the project change initiatives themselves.

Request permission

The full template that includes the template definitions can be found in online supplemental file 1. The template aims to outline the potential benefits and associated metrics to be derived from the project, directly linking these to the project enablers and change initiatives. Visualising these components on the map aimed to provide a clear line of sight between the project aims and the intended benefits.

Measurement

Measures to study the application of the benefits template include the engagement from project teams, as well as the time taken to create and complete the template. To capture this information, the application of the benefits map with each of the projects will be reported in a series of plan-do-study-act templates.18 Qualitative user feedback from teams regarding the ease of use and value of the template will also be collected. On completion of the templates with each team, key benefit themes will be drawn and collated.

Design

The application of the benefits mapping template was facilitated by the project managers (PM) overseeing the corresponding project. A realistic and simple approach was needed in order to maximise stakeholder engagement with the benefits template, and to achieve completed templates approved by the project leads. Our concept focused on organising a workshop-style meeting with key stakeholders from each transformation project. The scope of the session included an introduction to the benefits mapping template with progression on populating the template components. The application of this approach is described within each PDSA cycle. The engagement with key stakeholders and the aim to populate the templates with the project leads intended to yield a greater sense of ownership of the benefits to be derived from the respective projects. We recognised that it was crucial to be flexible in our approach with teams, given the clinical pressures faced during the challenging winter months, in addition to the impending second wave of COVID-19. Therefore, our expectations of participation in the workshops aimed to take this into account. Instead of aiming to engage the entire project team, we envisaged engaging at least two key stakeholders for the initial workshop, with subsequent review from the project leads and wider team. Prior to any targeted engagement, a webinar (benefits introductory session; 25 November 2020) on benefits realisation and improvement was delivered by the programme and open to attendance to all transformation project teams.

Strategy

A series of PDSA cycles were conducted in order to document the approach and reflections of introducing the benefits template to each transformation project. The application of the benefits mapping template for seven transformation projects is reported here. Where possible, changes based on the learnings from the preceding PDSA cycle were implemented in the next cycle of applying the template with another team. The PDSA cycle approach was deemed most appropriate in order to capture these small-step changes. The completion of the benefits template for each transformation project was approached by the overseeing project manager. These have been reported below for each transformation project.

PDSA 1: For the first transformation project, a 1-hour virtual workshop took place with the clinical project lead, project coordinator and two project managers. A blank version of the benefits mapping template was sent to both participants in advance of the session. In this case, the project coordinator had attended the benefits introductory session. During the workshop, it was necessary to talk through the template again with a worked example and to reiterate the context of the exercise. Prior to the session, the project manager had populated one row of the template as a starting point using one of the change initiatives stated in the project application form, which was then shared and discussed. Given there were several change initiatives associated with this project, the project manager proposed that each attendee attempt to complete a section of the template based on each of the change initiatives. This incorporated an interactive element to the session. Around ten minutes was allocated to this with a further 10 min to share initial thoughts. The 10 min window for interactive creation was relatively tight but did provoke discussion, and was intended to provide some sense of ownership to the team. Following the session, the completed template was circulated to the project coordinator and the lead for any iterations. The coordinator had also forwarded the template to the wider project team, comprising colleagues involved in delivering the project (respiratory consultant, physiotherapist leads and patient and public engagement managers), for comments ahead of finalising the template.

PDSA 2: The initial plan for the session for the second transformation project was to adapt the interactive component in PDSA 1, to be completed outside of the session. This modification to eliminate the interactive element was driven by the availability of the project leads following a challenging COVID-19 wave and learnings from PDSA 1. Instead, the benefits mapping template was introduced and discussed across 2-monthly project team meetings with the two clinical-project leads (respiratory consultants). In advance of the meeting, the project manager had completed a first draft of the benefits template using the project application form. In this case neither of the project leads had attended the benefits introductory session. The first meeting was used for context of the benefits mapping template and presentation of the drafted map. The project leads provided feedback on components of the completed template with additional suggestions to incorporate with very positive feedback overall.

PDSA 3: Given the approaches taken thus far with the transformation projects, we aimed to build on the cocreation element to populating the benefits template for the next transformation project. Again, due to COVID-19, it was not possible to engage the projects leads at this stage. Instead, two sessions were arranged with one of the clinicians on the project team. The first 30 min session was solely focused on introducing and explaining the template. There was also time allotted to start populating the template using one of the change initiatives stated in the project application form. The template was forwarded to the clinician to review in advance of the second 1-hour session dedicated to completing the template. The completed draft of the benefits map was then reviewed at a separate meeting with project leads.

PDSA 4: For this transformation project, the project manager populated the benefit mapping template as a first draft based on the project application form. Where there were queries or gaps, these were left blank or noted to query for the project leads to address. This draft was then circulated to project leads for comment prior to the Introductory Benefits Session. One project lead was able to attend the introductory session and present his initial reflections regarding the benefit map. Following this session, some momentum was lost due to the impact of the second COVID-19 wave on clinical duties among the project team (comprising critical care and cardiovascular consultant/medical colleagues). However, the project manager met with the project lead to review the benefits map. Given the project was a pilot randomised control trial, the project lead suggested the benefits map could be better organised into research, clinical and partnership benefits instead of outlining the project enablers. As such, this feedback was taken on board to simplify the process and to create additional slides which outlined (1) main outcome, (2) benefit, (3) metric/measure of success, (4) baseline data and (5) time frame (online supplemental file 2). This was then sent back to the project lead for final comments.

PDSA 5: Following the Benefits Introduction Session, this project team were introduced to the ‘model for improvement’ (MFI)19 to help establish the project aims, defining improvement in the context of the project initiatives, change measures and initiatives that will lead to longer term improvement. The project manager used this information and the project application form to draft an initial benefit map. The initial plan for the session was to have an interactive session and apply changes to the benefits map during the discussion. Instead, the session comprised of in-depth discussion on benefits realisation and project leads gave feedback on what looked correct and what needed to be amended. The project leads then iterated the map based on the MFI and discussion. This project team comprised pharmacists and project management support. The lead was also prompted for general feedback on the usability of the template. A total of four iterations of the benefit map were made prior to a final version.

PDSA 6: The team for this transformation project team were unable to attend the benefits introductory session. The first session included a discussion of the process, defining terms used in benefits realisation and benefits of developing the map. The project manager completed a first draft map based on the benefits outlined in the funding application. This document was shared at a team meeting, with little feedback due to insufficient time dedicated on the agenda. This was mitigated by sharing the draft template and requesting feedback via email. The project leads and the project manager refined the map iteratively through regular catch-ups. Once a final draft was agreed, the map was shared with the wider project team for final sign-off (principal investigator, patient representative, hospital site implementation clinical leads, four consultants and two research fellows).

PDSA 7: The project team were unable to attend the benefits introductory session, however, a summary was provided by the project manager at the project team meeting. The project team comprised head of service leads, IT procurement manager and the partnership project manager. As before, the project manager prepopulated the benefits map using the funding application. Project leads reflected it was challenging to identify and refine benefits during a meeting and offered to complete in their own time and bring back to a meeting to discuss. Due to COVID-19 clinical commitments, this was deprioritised, and feedback was that the benefits map template was not helpful in this instance. Instead, the project leads created their own document to list benefits and discussed this with the project manager who then translated this to apply to the benefits mapping template. The complexity of the project also meant that it was difficult to identify longer term benefits as these were dependent on the outcome of an options appraisal which had not yet been completed. A draft of the benefits map was agreed with the intention to revisit the template once the project had progressed sufficiently.

Results

The benefits template was approached and completed with seven project teams. Table 1 summarises the key benefit themes drawn from the completed maps and associated metrics. Examples of completed maps can be found in online supplemental file 2.

Lessons and limitations

The aim of this paper was to identify benefits and metrics across seven transformation projects, using a recently developed template. Building on recommendations from BRM frameworks8 and other literature, we wanted to pursue an approach that enabled key stakeholders in the project teams to drive the identification of potential benefits and metrics. A key reflection was the importance of PDSA cycles, which helped to document the approach and aspects that worked well.17

The BRM framework recommends that ownership of benefits realisation should sit with the project team, and this emphasis lay at the essence of the task. Our plan was to engage with project teams at the onset to explore benefits realisation and how it can be captured. It was anticipated that attempting to engage entire project teams would be challenging, given COVID-19 and the impending second wave. Therefore, we revised our approach to ensure that project teams had an opportunity to review and make suggestions to amend the first draft of their respective benefits map, ensuring direct engagement with the completion of the template from at least one key stakeholder from the project. While cocreating the maps generated a great deal of discussion with regards to the potential breadth of benefits that the change initiatives could achieve, ideally we would have aimed for further engagement through a meeting with the wider project teams. However, COVID-19 pressures made this challenging. This interactive approach could be adapted into a longer session in future, or prepped in way that allows teams to create something ahead of the meeting which can then be discussed later. The template itself was received well by project teams, as most had not considered benefits mapping as part of their project documentation, and valued access to the tool. Across all projects, maintaining and progressing with the benefits map was reliant on the PM, thus resulting in variable ownership of the template by project teams. It is important to note that the project managers in this case had overall oversight of the project plans, as oppose to a benefits realisation focus only. A project manager with a specific focus on benefits realisation may have resulted in less variation. The variable engagement with the template also flagged the need to be flexible in approach to benefits management. While a mechanism for identifying and measuring project benefits is imperative, a prescriptive template may not be the best fit for all projects.

In terms of limitations in navigation of the template, we found that teams were reliant on our facilitation and description of the key information to capture under each section. That is, on its own the descriptors in the template were not sufficiently intuitive for teams to complete independently. The template also requires benefits that meet the criteria in table 1, which meant it was difficult for teams to capture anecdotal benefits that may not be measurable but could demonstrate value. The level of project manager input to coordinate utilisation and engagement with the benefits map also leads to considerations around whether this approach is sustainable.

As an early and initial approach to benefits mapping, teams were encouraged to capture the entire scope of benefits that could be attributed to the project initiatives. This, however, meant the final output of the templates resulted in a long list of benefits, some of which not directly linked to the aim of the initiative. Additionally, the identification of metrics at this early stage did not account for ease of obtaining and reporting this data. It also presented the metrics to bear equal weighting, without reference to ownership or frequency of reporting. This highlighted that the primary value of the template was to facilitate identification of project metrics, and therefore, approach the first step of benefits realisation. On completion of the benefits maps, it was clear the next logical step would be to reach consensus to identify the priority metrics to capture, and establish accessibility of baseline data. Separate templates or reporting systems, such as measurement and data collection plans, could help to capture the reporting details relevant to the metrics identified.

Conclusion

In summary, we were able to produce completed versions of the benefits mapping template across transformation projects as per our initial aim. The project teams we piloted this template with comprised wide-ranging transformation initiatives within varying heart, lung and critical care clinical subspecialties. This demonstrated the generalisability of the template. To our knowledge, this is the first paper to report on the methodological approach of teams identifying benefits and metrics to be accrued from their transformation initiatives. Our PDSA approach was valuable to capture learnings and adopt a flexible approach. To sustain the continued review and improvement of BRM reported here, further work is underway to build on the next steps following completion of the benefits templates. The authors are scoping further tools and templates to add to the partnership’s benefits framework to help project teams capture the approach to the reporting and measurement details of agreed metrics to be monitored overtime. This will also include a standardised framework for capturing non-measurable, wider benefits of our work. Further, we aim to continue to audit the eventual measurement of those metrics and review the process with project teams to ensure that BRM support teams to realise their initiatives’ potential.