Article Text
PDF
Abstract
Background Internationally, healthcare systems face challenges from population demographics and rising care costs. Systematic methods of quality improvement (QI) are considered key to delivering needed transformation and change. Large-scale training in QI skills is required.
Massive open online courses (MOOCs) are an inexpensive strategy for QI training across large regions and countries. The Lean Fundamentals MOOC was developed in the English NHS to train health and care staff in Lean QI methods. It supported participants to learn and apply process improvement skills through a free-to-access, practical, learning-in-action approach.
Methods Lean Fundamentals’ cost-effectiveness and its impact on participants’ knowledge and confidence to apply Lean to improve processes were assessed. Using the Kirkpatrick framework, participant reaction, knowledge and confidence change, results and overall return-on-investment (ROI) were evaluated. Quantitative data were collected via pre and postcourse surveys to analyse participants’ knowledge and confidence change using the Wilcoxon signed rank test. Qualitative learning platform and postcourse survey data demonstrated participants’ results from application.
Results Over 18 months, Lean Fundamentals attracted 6617 enrolments and supported 3462 active participants. 97.6% (n=829) of participants completing the postcourse survey indicated Lean Fundamentals met their expectations and 97.2% (n=823) indicated they would recommend it. Self-reported changes in knowledge and confidence to apply Lean showed significant differences (p<0.001). Learning was applied to operational healthcare priorities (such as post-COVID recovery of services) and participants shared 511 project improvement reports.
Conclusion Lean Fundamentals helped large numbers of participants to develop Lean process improvement skills—avoiding costs associated with commercial Lean training in the range £1.7 m to £3.4 m and generating ROI of between £11 and £23 per every pound spent on delivery. This demonstrates that massive online is an effective and efficient method for building improvement knowledge and skills at scale.
- Lean management
- Continuous quality improvement
- Health professions education
- Continuing education, continuing professional development
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. ‘Not applicable.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- Lean management
- Continuous quality improvement
- Health professions education
- Continuing education, continuing professional development
WHAT IS ALREADY KNOWN ON THIS TOPIC
Massive open online courses (MOOCs) have been proposed as a means of training health and care professionals in systematic quality improvement (QI) methods at scale. MOOC evaluations have often been limited to participant reaction and calls for assessment of impact and cost-effectiveness have been made.
WHAT THIS STUDY ADDS
An evaluation of a Lean QI MOOC covering all levels of the Kirkpatrick framework including its reach and reaction from participants; impact on participants’ knowledge and confidence to apply Lean techniques; examples of impact and overall return-on-investment.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
A concrete example that demonstrates that MOOCs can be effective and efficient methods for building impactful improvement knowledge and skills at scale.
Introduction
Internationally, healthcare systems face challenges from population demographics and rising care costs.1 A global ‘quadruple aim’ has been established in healthcare to focus on improving care quality, population health, per-capita cost and staff experience of care provision.2 In England’s National Health Service (NHS), the quadruple aim’s objectives are embedded in its long-term plan for sustainable health and care.1 Quality improvement (QI) has been proposed as key to supporting the transformation and change required.1 3 4 QI involves using systematic methods and tools to redesign healthcare processes and systems to improve care quality and outcomes for patients.3 5 Large-scale capability building in QI skills has been proposed.1 6 7
Massive open online courses (MOOCs) are one strategy for large-scale capability building.8 9 The use of online platforms can scale up QI capability training across large regions and countries.10 MOOCs are designed to share knowledge and deliver learning to unlimited numbers of participants.8 9 11 They have been proposed as flexible and inexpensive methods to train health and care staff in QI.12 13
There are now several MOOCs for building knowledge for healthcare QI. Some lack evaluation and where evaluations have been made, important aspects are missing. Some emphasise participant satisfaction with the training but lack pre and postcourse measurement of participants’ knowledge.14 Others lack evaluation of how knowledge was applied and the impact of application.12 13 15 Importantly, little is known about the impact of QI MOOCs from the application of acquired knowledge.12
MOOCs have also been criticised for low completion rates with fewer than 10% of enrolments being noted as typical.16–19 However, completion rates alone may not reflect a MOOC’s quality or a participant’s perception of value.12 16 If participants learn what they wish to, such as by completing specific course components to bridge an identified learning gap, a MOOC’s quality may be perceived as high and participants’ overall assessment of usefulness may be positive.12 16 Evaluations establishing whether MOOCs met participants expectations have been noted as lacking.12
Lean Fundamentals MOOC
Systematic approaches to improvement comprise various methods and tools—such as the Institute for Healthcare Improvement’s Model-for-Improvement, Plan-Do-Study-Act cycles and Lean.4 Application of these methods has been noted as most impactful when deployed as part of a holistic, organisational approach, rather than via discrete improvement projects.5 20 21
To develop holistic, organisational improvement approaches in healthcare, there is increasing interest in quality management systems (QMS).4 22 23 A QMS coordinates an organisation’s activities to improve effectiveness and efficiency of processes on a continuous basis to achieve quality objectives and meet patient and regulatory requirements.24 25 Some NHS providers have implemented QMS ideas to begin the journey towards organisation-wide redesign and optimisation of processes—examples include East London21 and trusts involved in the NHS partnership with the Virginia Mason Institute (VMI).26 27
The VMI is the consulting arm of Virginia Mason Franciscan Health (VMFH)—a US based healthcare system with an international reputation for their adoption of QI using Lean.27 28 QMS approaches have been closely associated with Lean4 and VMFH has been highly rated for its care quality and safety, which it attributes to its QMS built on Lean principles and methods.27
In 2015, the national regulator of NHS trusts in England, NHS Improvement, now part of NHS England (NHSE), invested in a partnership programme with VMI to embed their Lean QMS in five NHS organisations.27 28 Concluding in 2021, the partnership was evaluated by Warwick Business School (WBS) to identify key lessons for building an organisational approach to quality management and a culture of continuous improvement.27 Key lessons identified the need for engaged leadership, cultural readiness, social connectedness, behavioural accountability and strategically aligned improvement priorities.26 27 The evaluation also found a need to build improvement capability ‘through a comprehensive training programme’ (27, p11) that could embed QI routines into daily practice.26 27
To maximise the impact and return-on-investment (ROI), there is a need to share the knowledge outputs from the VMI partnership widely across the NHS—including building improvement capability.8 Responding to this need, NHSE’s Improvement Capability Building and Delivery (ICBD) team developed a MOOC to build foundational Lean QI capability at scale.8 29
The Lean Fundamentals MOOC was designed to support participants to enhance their individual improvement capability and undertake a Lean process improvement project. The MOOC introduced Lean through a practical, structured learning-in-action approach that could be applied immediately by participants. Comprising an induction module followed by six, 1-hour content modules, Lean Fundamentals was a free course available 24/7 over an 8-week period to support participants to implement health and care process improvements.8 29 The course was certified to allow participants to receive continuing professional development (CPD) points. Table 1 presents an overview of the topics covered by the MOOC’s modules.
Overview of the Lean Fundamentals MOOC topics
Lean Fundamentals was developed by a faculty with experience of Lean. The faculty comprised three ICBD staff and an external associate. Two of the ICBD staff and the external associate had received extensive training in Lean from VMI as part of their earlier work with the NHS in North East England.30 The faculty were supported with input from an advisory board that included chief executive representation from VMI partnership trusts and the WBS evaluation team.8 The Lean Fundamentals MOOC was designed to encourage connection and social interaction (both between participants and between participants and faculty) and was delivered via a learning platform that facilitated this.8 9
First launched in February 2021 as a small private online course to support the NHS national vaccination programme,8 29 Lean Fundamentals has subsequently run as a MOOC in four further cohorts between May 2021 and September 2022. This paper reports on the evaluation of Lean Fundamentals, describing: its reach and reaction from participants; impact on participants’ knowledge and confidence to apply Lean techniques; examples of impact; and overall ROI.
Methods
MOOCs are an increasingly popular educational format16 31 and are now being used in healthcare to develop various skills.17 Due to their growing popularity, ease of access and potential to reach large audiences, there is a need to evaluate MOOCs to appraise their success and effectiveness.17 31 However, due to their open, large-scale nature, MOOC evaluations can be subjected to sources of bias and appropriate methods must, therefore be used.31
MOOC evaluation methods should be selected to match the evaluation aim. Mixed methods have been recommended combining analysis of both qualitative and quantitative data from a diverse range of sources.12 17 31 For quantitative data, the use of pre and postcourse metrics compared using statistical tests has been recommended to reduce risk of bias.31 Qualitative data have been recommended to enhance findings and explain phenomena.31 To ensure evaluations are comprehensive and systematically review a MOOC’s aspects, the use of an appropriate framework has been recommended.31
Approach
The aim of the Lean Fundamentals evaluation was to assess impact on participants’ knowledge and confidence to apply Lean techniques to improve health and care processes and the overall cost-effectiveness of the MOOC. To achieve this, a modified version of the Kirkpatrick framework was adopted.
Kirkpatrick is a popular framework used to evaluate training and development programmes.32 33 It supports evaluation of impact over four levels: reaction, learning, behaviour and results.33 The first level (reaction) is concerned with participants’ attitudes towards the programme. The second level (learning) is concerned with participants’ acquisition of knowledge and skills. The third level (behaviour) is concerned with participants’ confidence to translate knowledge into practice and the extent to which this occurs. The fourth level (results) is concerned with the overall success of the training intervention in achieving organisational goals.12 17 32–34 Kirkpatrick has been proposed as an appropriate framework for evaluating MOOCs, including those designed to build healthcare improvement capability.12 15 17
It has been argued that healthcare QI training programmes should also assess cost-effectiveness and ROI.35 36 A fifth level, focussing on ROI and intangible benefits, can be added to the Kirkpatrick framework as a credible way of achieving this.35
Data collection and analysis
A blend of quantitative and qualitative data was used to evaluate the Lean Fundamentals MOOC against the modified Kirkpatrick framework. Levels 1 and 2 of Kirkpatrick emphasised the quantitative data. At levels 3 and 4 of Kirkpatrick, a greater emphasis was placed on qualitative data.
Quantitative data were collected via pre and postcourse participation surveys for ipsative assessment.37 Ipsative assessment refers to comparing a course participant’s results to their previous results. In this study, the ipsative assessment compared participants’ knowledge and confidence of each topic before and after undertaking the Lean Fundamentals MOOC.37 SPSS statistical software and the Wilcoxon signed rank test were used to analyse pre and postcourse surveys. This tests for significant difference between paired samples of non-parametric data.38 39 Paired samples occur in studies measuring participant attributes before and after an intervention.40 In this study, participants rated their knowledge and confidence before and after undertaking the Lean Fundamentals MOOC. To create paired samples of participant assessments for ipsative analysis, pre and postcourse surveys collected the data using ordinal Likert scales—from 1 (lowest) to 5 (highest).34 The post participation survey also captured participants’ experience of the course using an ‘yes/no’ response to whether it met their expectations and a five-point Likert scale of how likely they would be to recommend the course.
Qualitative data were collected via the post participation survey and from the learning platform in which the Lean Fundamentals MOOC was delivered. Learning platforms have been identified as important data sources for MOOC evaluation.31 In addition to quantitative information (such as learner enrolment numbers and completion rates), learning platforms can also provide qualitative information on various components of the MOOC and learner activity such as social sharing space comments and work produced by participants.31
The post participation survey presented a combination of open and closed questions covering: participants’ reaction to and experience of the course; what they had learnt; how they were using the learning; how the course did or did not meet expectations and its overall impact.
As part of completing Lean Fundamentals MOOC, participants produced improvement project reports and uploaded them to the learning platform. Improvement project reports were short, three-page documents comprising up to two pages of text and space for up to three figures. The format was based on the Standard for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines41 using a heading structure proposed by Moss and Thompson and refined by BMJ Publishing Group.42 43 To capture participants’ improvement project focus, a health and care priorities survey was built into the learning platform based on national NHS annual operating priorities. Uploaded improvement project reports were reviewed to provide examples of application by participants. Two high-impact project improvement reports were fully written up and published as peer-reviewed QI reports in a BMJ Group Journal.44 45
Return on investment (ROI) is a metric calculated as the ratio of benefits generated to costs input for a given initiative, intervention or investment.35 36 ROI is typically calculated by evaluating the costs and benefits in monetary terms, so that the ROI ratio can be expressed as £GBP returned per every £GBP spent. £GBP benefits from Lean Fundamentals have been evaluated in terms of cost avoidance associated with cost of training. The cost of Lean Fundamentals was evaluated in terms of: fixed costs (learning platform and CPD licenses); development costs (a one-off cost comprised of staff and faculty time for course design, project management, content production and course build) and delivery costs (staff and associate faculty time for facilitation of each cohort of 8 weeks’ duration).
The relationship of quantitative and qualitative data to the modified Kirkpatrick framework’s five evaluation levels is shown in table 2.
Data collected and levels of the Kirkpatrick model
Results
Reach and participant reaction
Between February 2021 and September 2022, Lean Fundamentals attracted 6617 enrolments. Of these, 52% participated in the MOOC—where participation is defined as an enrolled individual who views at least one component of the course.12 In the precourse survey (n=2050), 92% of enrolments reported being from England—which was the target audience of Lean Fundamentals. A total of 857 participants completed the postcourse survey giving an overall response rate of 24.8% of course participants. A total of 802 completed all requirements of the MOOC giving a completion rate across all cohorts of 12.1% as a proportion of enrolments (and 23.2% as a proportion of participants).
Four other QI MOOC evaluations were identified which reported both enrolment and completion data.9 13–15 As a proportion of enrolments, these evaluations reported completion rates (and enrolments) of 1.5% (5751),15 6.2% (30,636),13 14.8% (12,222),9 and 20.4% (578).14 The 20.4% completion rate reported by Dwyer et al14 was for relatively low enrolment numbers. Also, when comparing completions as a proportion of participants (see online supplemental data), Dwyer et al’s MOOC achieved a similar completion rate to Lean Fundamentals—25.7% vs 23.2%, respectively.14 The 14.8% completions rate reported by Guest et al9 relates to another MOOC from within NHSE’s ICBD group and is comparable to Lean Fundamentals. Compared with studies of large-scale, international MOOC platforms, Lean Fundamentals achieved a completion rate as a proportion of enrolments of almost two times that identified by Jordan (6.5%)18 and almost threefold that identified by Reich (4.5%)19—see online supplemental data for graphical comparison.
Supplemental material
Table 3 shows enrolment, participation and completion data for each cohort of Lean Fundamentals and the postcourse evaluation response rate. Precourse responses and demographics are available as online supplemental data.
Lean Fundamentals enrolments, participants, completions and overall survey response rates
Participants reacted positively to the course. In the precourse survey, participants indicated their expectations in terms of what success would look like (n=1929) and how they expected to apply learning (n=1807). Participants indicated that success included learning new skills and increasing their knowledge and confidence to apply Lean. Specific successes included applying learning in daily practice, improving processes within their own area, combining knowledge with other QI methods and supporting others to undertake improvement. In the postcourse survey, 97.6% (n=829) indicated that Lean Fundamentals met their expectations. Factors cited by participants that contributed to expectations being met included a high-quality learning design, materials and an easy to navigate platform; convenience and ability to work at own pace; being part of a community with other participants; a high level of interaction and coaching support from faculty and practical applicability of the content to real problems and processes. Overall, 97.2% (n=823) indicated that they were extremely likely (n=517) or likely (n=306) to recommend the course to colleagues.
Change in knowledge and application confidence
Of the 857 postcourse survey responses, 740 were matched to completed precourse surveys—giving a total of 740 paired samples to assess change in knowledge and confidence to apply the content.
The Wilcoxon signed rank test showed a significant difference (p<0.001) in participants self-reported change in knowledge and their confidence to apply it before and after each of the Lean Fundamentals MOOC’s topics. Data are summarised in table 4—aggregate graphical distributions of before-and-after data for each topic are presented as online supplemental data.
Self-reported change in knowledge and confidence to apply by topic
Example feedback from participants on what and how they were learning is shown below:
‘Learning about LEAN in a way in which brings it to life, is easily digestible and is practical. I really enjoyed it. Thank you!’ Head of Governance and Risk.
‘Online platform worked wonders—meant could approach at own pace to obtain new skills’ Quality Improvement Lead.
‘(Through) the exercises and applying the processes to your workplace, I feel I have gained valuable knowledge to support my current project and leadership skills’ Clinical Lead for Covid Vaccinations.
Participants described how they were using their new knowledge and skills and how they planned to continue using them in the future. Example feedback is shown below:
‘I feel more empowered and … more passionate to influence the delivering of the Covid vaccine. I feel I will take this into my future roles’ Vaccine Centre Manager.
‘I will use this regularly and from department to department as I work to support (improvement in) the NHS.’ Improvement Project Manager.
‘LEAN has given me the tools to apply this to the changing situations I work within and allows me to ensure efficiency and productivity’ Head of Clinical Governance & Operations.
Examples of impact
Lean Fundamentals was designed to support participants to undertake a Lean process improvement project. Participants indicated their area of focus as part of course completion. Participants also uploaded examples of practical activities undertaken as part of the MOOC. This included completed templates covering Lean tools such as specific improvement project aims, process time observation, process flow mapping; and project improvement reports.
Project improvement reports were documents produced by participants to describe their process improvement project. Participants used a template, provided as part of the course, to share a concise description of: the process problem they worked on; their approach to measurement; how improvement ideas were selected and tested; their results and lessons learnt—see online supplemental data for an example report.
Participants uploaded 511 project improvement reports to the learning platform on completion of the course to evidence real-time application of learning in improvement projects. Note, however, that considerably more improvement projects were initiated by the 3462 active participants. How participants’ improvement projects aligned with national annual operating priorities was assessed using data from the health and care priorities survey (which was embedded in the Lean Fundamentals learning platform). 51% (n=1005) of participants indicated an improvement project focus area aligning with NHSE’s operational priorities. Of these, 31% (n=308) indicated a focus on transforming community and urgent and emergency care, 28% (n=282) post-COVID elective recovery, and 15% (n=148) COVID vaccination programmes (see online supplemental data). Specific examples of improvement project aims included:
‘Reduce patient waiting times and improve the flow in the emergency department’.
‘Streamline clinic processes to improve patient care and reduce waiting times for elective and day-case surgeries’.
‘Increase vaccination clinic throughput by reducing waiting times and queues’.
Results reported in participants’ project improvement reports varied depending on available data. Some reported on creating improved working environments through application of Lean Fundamentals’ ‘workplace organisation’ techniques. Others reported modest reductions in processing time. Some reported highly impactful process changes. Published examples of high-impact improvement projects undertaken by Lean Fundamentals participants are shown in box 1.
Project improvement report impact case examples
Large-scale vaccination centre throughput
A large-scale national health service (NHS) vaccination centre received 1560 bookings for vaccinations (over 500 more than expected) creating challenges with managing queues and waiting times. Leanlean Fundamentals provided process analysis tools to quantify the bottlenecks, and a structured improvement routine was used to reduce processing times, improve client experience and increase throughput by 50% with no additional resources.44
Paediatric waiting list recovery
An NHS hospital trust used Lean Fundamentals to support an improvement project to recover the waiting list of a paediatric service to pre-COVID levels. Over a period of 8 weeks, the project reduced the waiting list from 1109 to 212. Waiting times were reduced from a maximum of 36 months to a 2-to-3 month average.45
Return on investment
Access to Lean training has not previously been readily available to NHS staff at meaningful scale and not without costs. Training in Lean improvement skills have been largely the preserve of consultancy firms and specialist, fee for service, training providers. The NHS partnership with VMI, for example, used traditional classroom-based training designs to build capability.27 While evaluated as successful,27 such designs can limit the number of delegates able to access the training and invoke high costs relating to consulting expertise, travel, venue costs and delegates’ time to attend—particularly if attempted at national scale. Conversely, through its massive-online design, Lean Fundamentals is highly scalable and can support thousands of participants to learn and apply Lean methods to solve problems within their organisations while avoiding costs associated with specialist provision.
To calculate ROI, training costs avoided by Lean Fundamentals have been estimated based on an assessment of content and delivery similarities and differences with commercially available online offers. For example, in addition to traditional face-to-face training designs, VMI also offer open-access, fee for service online training.46 It is against online training offers such as this, that Lean Fundamentals has been assessed to estimate a reasonable cost avoidance value per participant to the NHS in the range: £500 (low) to £1000 (high)—see online supplemental data.
Using the numbers presented in table 3 and multiplying by the cost avoidance value per participant estimates above, an aggregate cost avoidance can be calculated for Lean Fundamentals’ five cohorts. Between February 2021 and September 2022, 6617 people enrolled on Lean Fundamentals and 3462 actively participated in the course. Using enrolment figures suggests a cost avoidance associated with Lean training in the range £3.3 m–£6.6 m. Using participant figures suggests a cost avoidance in the range of £1.7 m–£3.4 m.
To calculate ROI, fixed costs and development costs were estimated at £10k and £65k, respectively. Delivery costs were estimated at £15k per cohort of 8 weeks’ duration—for facilitation, administration and platform technical support. Therefore, the total cost of delivering five cohorts was estimated as £150k. Calculating ROI for participants at both the low and high cost avoidance estimates suggests a positive ratio of benefits to costs of between £11 and £23 per every pound spent on Lean Fundamentals.
Limitations
The paper has several limitations.
Not all initiated projects were followed up. The priority area focus survey, and review of submitted project improvement reports, demonstrated practical application of knowledge. The publication of two, peer-reviewed QI reports demonstrated impactful results. However, it has not been possible to follow-up on all 511 project improvement reports uploaded to the learning platform or the unsubmitted projects initiated by the almost 3500 active participants. Future studies of QI MOOCs might usefully explore this—particularly to ascertain the results and impact of non-completing participants, which is currently under investigation. Furthermore, an assessment of the sustainability of process improvements has not been made.45 In addition to process changes, sustainability has been related to other factors, including leadership, culture and organisational infrastructure to support improvement.47 This could be a useful focus for future studies of QI MOOCs, particularly where longitudinal follow-up is planned.
Self-selection to enrol and subsequently participate in the course may introduce some bias. Those completing the postcourse survey were largely those who completed all course requirements (n=857). This amounts to fewer than half of those who completed the precourse survey (n=2062). It is possible, therefore, that course completers had a greater intrinsic interest in Lean. However, similar to other QI MOOC evaluations, data on reasons for non-completion were not collected and selection bias cannot be ruled out.14 15 Some have proposed that future studies collate non-completer perspectives. However, this must be balanced with the evaluation burden placed on participants which may contribute to bias via lower completion rates.13 15
The sample sizes for postcourse surveys (n=857) and paired ipsative data (n=740) were relatively low compared with total enrolments (n=6617). While this has been identified as a potential source of bias in QI MOOC evaluations, Lean Fundamentals compared favourably to other studies.13–15
ROI estimates are based on cost avoidance of training. Participants may have joined because the course was free and may not have joined a fee-for-access alternative. However, in such a scenario, participants would have missed the opportunity to develop Lean process improvement knowledge. Also, the costs avoided figures are modest compared with leading commercial offers. Furthermore, the commercial offers used for cost comparison were similar to Lean Fundamentals in terms of content; delivered in online formats and open to any participants willing and able to pay the fee. Therefore, following Buzzachero et al,35 the author believes that the cost avoidance figures represent a reasonable estimate to calculate ROI.
Different training modalities offer different benefits. MOOCs bring participants together as a virtual community and offer interorganisational opportunities to connect and share learning across organisational boundaries.9 10 13 Training individuals from the same organisation in a classroom-based setting offers an intraorganisation opportunity to network and collaborate on solving shared organisational problems.26 27 The aim of this study was to evaluate the effectiveness of Lean Fundamentals to build improvement capability of individuals. However, Lean Fundamentals has not been used to build QI capability within an organisation as part of a systemic approach. Including cohorts of people from the same organisation could be a useful focus for future research to determine the efficacy and cost-effectiveness of MOOCs for building organisational QI capability at scale.
The author played a lead role in the design and delivery of the Lean Fundamentals course. Therefore, to mitigate the potential of positivity bias, drafts were shared with ICBD colleagues not responsible for authoring the paper to gain feedback.
Implications and conclusion
Using a modified version of the Kirkpatrick framework, the Lean Fundamentals MOOC was evaluated and impact demonstrated to level 4 (results) and level 5 (ROI). At level 1, participant reaction was positive. At levels 2 and 3, change in knowledge and confidence were quantified and shown to be significant. At level 4, participants were shown to initiate improvement projects aligned to national priorities and, at level 5, ROI was calculated to show a positive ratio of benefits to cost.
Part of the motivation to develop the Lean Fundamentals MOOC was to share widely across the NHS knowledge outputs from the VMI partnership. However, Lean fundamentals did not replicate it. The VMI partnership evaluation identified several key lessons for building systematic approaches to QI, including engaged leadership, cultural readiness, social connectedness, behavioural accountability, the importance of strategically aligned improvement priorities and the need to build improvement capability ‘through a comprehensive training programme’27 that could embed QI routines into daily practice.26 27 The findings of this evaluation suggest that Lean Fundamentals can contribute at-scale to building the QI capability of individuals to work on strategic improvement priorities.
Lean Fundamentals helped large numbers of participants to develop their knowledge of Lean concepts, principles and tools. Participants indicated that they considered the course successful if it increased their knowledge and confidence of Lean allowing them to apply it in their daily practice. Postcourse respondents indicated that these expectations were met and this was supported by significant change in participants’ ipsative assessment of knowledge and confidence (p<0.001).
To date, few QI MOOC evaluations have demonstrated impact against organisational goals.12 In this evaluation, over half of participants indicated an improvement focus aligning with NHSE’s operational priorities. Also, following Lean Fundamentals’ structured improvement routines, participants could apply what they had learnt and used Lean methods to improve processes. Submitting over 500 project improvement reports to the Lean Fundamentals learning platform, participants applied their learning to various operational priorities. These included notable examples of impact on COVID-19 vaccination processes and elective waiting list recovery.8 44 45
Lean Fundamentals attracted over 6600 enrolments over an 18-month period. While MOOCs have drawn criticism for low completion rates,16–18 Lean Fundamentals’ compared favourably with other QI MOOCs9 13–15 and international benchmarks.18 19 Lean Fundamentals supported almost 3500 active participants to learn Lean process improvement skills—avoiding costs associated with commercial Lean training in the range £1.7 m to £3.4 m and generating ROI of between £11 and £23 per every pound spent on delivery. This demonstrates that massive online is an effective and efficient method for building improvement knowledge and skills at scale.
QI MOOCs, such as Lean Fundamentals, may offer a high-quality alternative to traditional training that is accessible to NHS staff and avoids costs associated with specialist provision. National regulators and policymakers may wish to consider the contribution that QI MOOCs can make to large-scale improvement capability building7 in terms of efficacy and value-for-money.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. ‘Not applicable.
Ethics statements
Patient consent for publication
Ethics approval
The Health Research Authority online decision tool was used to determine that NHS research ethics was not required for the study as it was considered evaluation of service improvement. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The author acknowledges Elaine Bayliss for providing feedback on earlier drafts of the paper.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @DrIainMSmith
Contributors IMS was responsible for: reporting on the work; writing and revising the article; and the overall content as guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests We have read and understood BMJ policy on declaration of interests and declare the following interests: IMS works as an improvement and transformation professional in the NHS and led the design and delivery of the Lean Fundamentals MOOC.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.