Article Text
PDF
Abstract
Introduction Quality improvement (QI) is a growing field of inquiry in healthcare, including neonatology. However, there is limited information on the study setting, and the methodologic approaches used to develop, implement and evaluate QI interventions in neonatology studies. In this study, we describe these intervention characteristics and approaches.
Methods Articles were taken from a previous publication. There, we searched MEDLINE for publications of QI studies from 2016 to 16 April 2020. We retrieved all relevant full-text publications and sampled 100 of these articles for data abstraction, stratified by the year of publication. For each QI study, we described several methodological characteristics that included: the clinical topic of QI, setting, whether the study was multicentre, stakeholder engagement, root cause analysis and related problem identification methods, implementation techniques for QI interventions, types of outcomes and statistical analysis methods used.
Results We assessed 100 studies; most were conducted in the USA (56%). Academic settings and multicentre settings comprised 44% and 24% of studies, respectively. Most studies reported stakeholder engagement (81%), but infrequently reported engagement with leadership (32%) and caregivers (10%). Frequently used techniques for implementing interventions include provider education (82%), formal QI methods (42%) and audit, feedback and benchmarking (40%). Both patient-important clinical outcomes (78%) and process outcomes (89%) were frequently reported. P values were frequently reported (80%), but other statistical techniques were infrequently used.
Conclusion QI studies in neonatology use diverse multicomponent interventions. Reporting of these methodologic details can be useful in designing, implementing and evaluating QI studies in clinical practice.
- Quality improvement
- Evidence-based medicine
- Quality improvement methodologies
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Quality improvement (QI) in healthcare, defined broadly as systematic activities conducted to achieve sustained improvements in patient outcomes and health systems performance, has become an increasingly active field of inquiry.1 There is a wide variation in the definition, objectives and methodologic features of studies that can be termed as a ‘quality improvement’ study. The Agency for Healthcare Research and Quality largely sees QI as activities that bridge the gap between the ‘ideal’ evidence-based standards and local clinical practice.2 The SQUIRE V.2.0 QI reporting guideline considers QI to be any ‘systematic effort to improve the quality, safety and value of healthcare’,3 which can also include cohort studies, randomised controlled trials and comparison studies. QI is also interwoven with implementation science, health services research and many other study types.4 Hence, it may take on research methods and publication practices of these fields of inquiry.
Understanding how QI work is defined and conducted amid this diversity can inform efforts to improve the reporting, effectiveness and adaptation of QI activities and evidence in clinical practice. Previous literature reviews have examined the reporting of methodologic attributes for specific types of QI studies,5 6 summarised the various QI methodologies that are used and examined QI reporting among the general healthcare literature,7 and in the specialties of diabetes,8 antimicrobial selection9 and perioperative care.10 However, to our knowledge, a methodologic review of QI studies has not been performed in neonatology.
Furthermore, numerous reporting guidelines and QI primers have been published to guide clinicians on how to contextualise, develop, implement and evaluate QI studies in hospital and community settings. These methodologic attributes include identifying change ideas through stakeholder engagement and problem-framing methods, implementing interventions using QI methodologies such as plan-do-study-act cycles or total quality management and evaluating the project using outcome, process and balancing measures and statistical analysis using statistical process controls.11–14 Understanding how recently published QI work has incorporated these characteristics can inform future directions in QI reporting and effectiveness efforts.
The objective of our study is to describe the characteristics of the setting, intervention development, implementation and evaluation approaches of QI studies in neonatology. Herein, we will quantify how often various techniques for developing, implementing and evaluating QI activities are used or reported in published articles.
Methods
This study uses articles from a previous systematic survey of the literature.15 The literature search, full-text screening and random selection of articles are described in a previous publication.8 In summary, we searched the Medline database for publications from 2016 to 16 April 2020, as defined by the ‘year of publication’ field using the search strategy shown in online supplemental appendix A1. Subsequently, for the title/abstract and full-text screening, we sought to include all QI articles. Here, we defined QI as any study that described a systematic effort to improve the quality, safety or value of healthcare, in line with the definition stated in SQUIRE V.2.0, the current guideline for reporting quality improvement studies.3(p0)
Supplemental material
After obtaining all relevant full-text articles, we selected a random sample of 100 articles to assess, stratified by the year of publication. The random selection process involved first determining the number of articles that were published each year (2016–2020), and then assigning a probability of selecting each stratum. Second, within each year-of-publication stratum, we sorted the articles randomly using Excel. Finally, we sampled the articles in their sorted order, based on the probability that the article belonged to a specific stratum. A flow diagram of our selection process, adapted from our previous study, is shown in figure 1.
Flow diagram describing the process of selecting articles for this study. QI, quality improvement.
Determining attributes to evaluate
The methodologic attributes to assess, such as stakeholder engagement or statistical methods, and the categories within each attribute, such as the type of statistical method used, were selected by ZJH. These attributes and its categories were adopted from published QI primers by Silver et al14 (Quality improvement in emergency medicine), Chartier et al11–13 (Quality improvement in haemodialysis), and Shojana et al16 (Closing the quality gap reports). After selecting these attributes and its categories, ZJH consulted with LT, SeH and GF to determine whether these attributes were important features of quality improvement reports and if they were feasible to evaluate. The categories for each methodologic attribute were further adjusted as the review progressed.
Some of the attributes that describe the study were chosen to examine specific questions of interest. Most attributes were chosen as they represented the design/framing, implementation and evaluation of QI interventions. These attributes, our rationale for selecting them and our questions of interest, are shown in table 1.
Attributes assessed and their importance for determining the success of QI projects
A posteriori attributes
Following comments from peer reviewers for deeper analysis, we added additional categories for ‘intervention implementation’ based on intervention categories derived from Chartier et al and Shojana et al.12 16 The categories of intervention implementation, and its definition, are shown in table 2.
Components of quality improvement interventions and their definitions12 16
Data abstraction
We provided training to the student reviewers and then pilot tested two articles each to assess their agreement with ZJH and with each other. Each reviewer was provided a codebook that briefly defined each methodologic attribute. Following the pilot testing, we updated the codebook based on feedback to ensure a shared understanding. Finally, the remaining articles were divided between three student reviewers and assessed independently with ZJH. Disagreements were resolved arbitrarily by ZJH or by discussion for more subjective cases.
Analysis/synthesis
We summarised the characteristics of the included studies using descriptive statistics with categorical variables reported as frequencies. ZJH noted the clinical topic of quality improvement of each study using free text, and generated Word clouds to describe them. The Word Cloud was created using Word Cloud Pro in Microsoft Word (Microsoft 365, V.2202). Frequencies were computed in Microsoft Excel (Microsoft 365 V.2202).
Results
Study setting and context
The frequency that each item was reported is shown in table 3. Most studies were conducted in the USA (56%) and considered the individual patient as a unit of intervention (70%). A large number (44%) of publications reported that their activities were conducted in an academic or teaching hospital setting, though we note that this may be higher as studies may not report this detail. Only 24% of studies were multicentre; these included QI interventions implemented at multiple centres, a QI programme implemented in hospitals participating in a statewide quality improvement network or in multiple communities. The clinical topics of QI studies were very diverse, and there were no dominating clinical topic (see figure 2). Nearly half (48%) of studies were waived for ethics approval, and, hence, was viewed by the institutional review board as a quality improvement study. More studies may have been waived but did not declare it.
Word cloud of clinical topics of QI studies in neonatology: The size of words represents the frequency of single words that were used to describe clinical topics throughout the review. QI, quality improvement.
Descriptive characteristics of QI studies in neonatology
Intervention development, implementation and evaluation
The frequency that each intervention implementation technique was used or reported is shown in figure 3. Few studies used root cause analysis and its related techniques to explore and identify problems requiring QI. Most studies reported stakeholder engagement (81%). However, few reported engagement with hospital leadership (32%) and even fewer with parents and caregivers (10%).
Characteristics of QI intervention implementation techniques and their frequencies. QI, quality improvement; PDSA, Plan-Do-Study-Act; TQM, Total Quality Management; CQI, Continuous Quality Improvement; CRM, Crew Resource Management
The majority of studies included healthcare provider education and training (76%) in implementing their intervention. Other implementation strategies that were used often (greater than one-third) include clinical guidelines, standardisation of practices, audit, feedback and benchmarking, communication (including teamwork and simulation) and QI techniques such as Plan-Do-Study-Act \ cycles and Total Quality Management.
Most studies incorporated patient-relevant outcome measures (78%) and measures that examine or monitor the processes of care and implementation (89%). Most studies also assessed the statistical significance of their interventions (80%). However, the use of other statistical methods including CIs, statistical process control and adjustment of confounders through statistical modelling was seldom mentioned. Therefore, most studies did not examine the variability of their intervention effect estimates, assess for special cause variation over time, nor adjust for confounders when estimating intervention effects.
Discussion
This systematic survey investigated various methodologic approaches in the development, implementation and evaluation of quality improvement reports in the neonatology literature. Studies were conducted mostly in the USA, targeted to patients as a unit of intervention, mostly engaged healthcare providers as stakeholders, used education and training as an implementation technique, frequently included patient-important outcome and process of care measures and frequently reported p values in their statistical analysis.
A key strength of this study is that we assessed methodologies of design, implementation and evaluation all together. This approach allows researchers to better understand the methodologies employed in QI studies across its entire lifecycle, which can inform future directions on QI education efforts and publication practices. In comparison, previous QI literature syntheses did not examine these attributes all together, or only focused on specific types of QI methodologies.5 7 8 10
A key limitation is in accurately assessing the frequency of numerous attributes is the way they are reported. On study setting, we may have undercounted the number of studies that occurred in academic or teaching hospitals because the manuscript did not always mention this detail in the study setting. Furthermore, we were unable to assess how many studies took place in community non-academic hospitals because many studies did not report whether their studies occurred in community hospitals or not. Thus, we could not claim whether QI studies were dominated by academic/teaching hospitals or not. The same issue arose for ethics approval, where we cannot ascertain whether all studies have reported their waiver of consent. Likewise, many studies may not have reported their engagement with leadership or parental stakeholders, as such information is not required directly by SQUIRE V.2.0 reporting guidelines.3(p2)
Herein, we have identified several aspects of QI study design, implementation and evaluation that can be targeted for future improvement efforts. First, studies can engage in more stakeholder engagement with leadership and caregiver stakeholders. Successful engagement with hospital leadership may ensure that leadership will use their influence, and devote resources towards conducting a rigorous, successful and sustainable QI project.14 17 Likewise, engagement with caregivers is central to ensure patient-centred healthcare delivery in the neonatal context.1 Reporting these details will allow researchers to share information and appraise engagement strategies.
Second, efforts for improving QI in neonatology can focus on improving the reporting, implementation and effectiveness of intervention strategies we identified above: clinical practice guidelines, standardisation of care, provider education and training, audit, feedback and benchmarking and communication. Each of these strategies have its own methodologic nuances and frameworks. Education efforts can focus on these techniques to better support improving QI.
Finally, the infrequent use of other statistical techniques among publications indicate that improvements are needed in the statistical analysis of QI studies. Studies can enhance their statistical analysis by employing statistical process control, as doing this would imply that studies both monitored continuously and used statistical techniques to infer if the intervention has achieved sustained change in outcome.18 Interventions focused on pre–post change can also benefit from employing CIs, as this indicates the magnitude of change seen,19 which is more in line with goals of improvement.
Conclusion
Overall, QI studies in neonatology are characterised by diverse forms of interventions. Additionally, this study has identified areas for further methodologic work, such as the reporting of stakeholder engagement with leaders and caregivers, targeting specific intervention techniques and approaches to statistical analysis. These improvements can strengthen the effectiveness of quality improvement activities and contribute to advanced healthcare practices for providers and improved health outcomes for patients.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors ZJH drafted the manuscript, designed the study, reviewed articles and made the final decision on data abstraction. LT mentored ZJH and provided expertise on manuscript writing and items to include in data abstraction. SeH and GF were consulted in formulating items for the data abstraction form. CH, JYW and MM acted as second reviewers for ZJH. All authors read, provided feedback and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.