Article Text
PDF
Abstract
Background Quality improvement (QI) is becoming an important focal point for health systems. There is increasing interest among health system stakeholders to learn from and share experiences on the use of QI methods and approaches in their work. Yet there are few easily accessible, online repositories dedicated to documenting QI activity.
Methods We conducted a scoping review of publicly available, web-based QI repositories to (i) identify current approaches to sharing information on QI practices; (ii) categorise these approaches based on hosting, scope and size, content acquisition and eligibility, content format and search, and evaluation and engagement characteristics; and (iii) review evaluations of the design, usefulness and impact of their online QI practice repositories. The search strategy consisted of traditional database and grey literature searches, as well as expert consultation, with the ultimate aim of identifying and describing QI repositories of practices undertaken in a healthcare context.
Results We identified 13 QI repositories and found substantial variation across the five categories. The QI repositories used different terminology (eg, practices vs case studies) and approaches to content acquisition, and varied in terms of primary areas of focus. All provided some means for organising content according to categories or themes and most provided at least rudimentary keyword search functionality. Notably, none of the QI repositories included evaluations of their impact.
Discussion With growing interest in sharing and spreading best practices and increasing reliance on QI as a key contributor to health system performance, the role of QI repositories is likely to expand. Designing future QI repositories based on knowledge of the range and type of features available is an important starting point for improving their usefulness and impact.
- Healthcare quality improvement
- Collaborative, breakthrough groups
- Communication
Statistics from Altmetric.com
Background
Quality improvement (QI) is becoming an important focal point for health systems. QI methodologies are increasingly positioned as a fundamental tool to increase the effectiveness, patient-centredness and efficiency of healthcare services, and ultimately to improve the sustainability of health systems.1–5 Thus, governments, health system leaders, healthcare providers and other health system stakeholders should have an interest to learn about and apply the best QI methods and approaches in their daily work.
While others have documented the need for, and challenges of, increasing publication on QI in peer-reviewed academic journals,3 ,6 ,7 there are few reliable, easy-to-use online resources dedicated to documenting QI activity and experience from around the world for non-academic audiences. This represents collective global inefficiency and a failure to share advances in QI in ways that are commonplace for more traditional forms of health research. For example, the millions of articles on health research reported in peer-reviewed journals annually, from biomedical and clinical research to population health and health systems research, are searchable through sophisticated databases like MEDLINE.8 The U.S. National Library of Medicine provides public access to the MEDLINE database through PubMed, which is searched more than two billion times per year.9 While some formal QI efforts may ultimately be reported in peer-reviewed journals that are indexed in databases like MEDLINE, the vast majority of QI activity (including specific localised initiatives, projects and system-wide programs) is not formally or publicly documented. QI efforts, whether successful or not, represent valuable learnings that can help others build upon past experiences.10 This suggests a need for publicly available, web-based QI repositories to support the advancement of health systems.
A few examples exist of emerging QI repositories that focus specifically on documenting QI methods, practices, projects, interventions, initiatives or case studies, hereafter referred to collectively as QI practices. However, as can be expected with any nascent effort, the approaches to structuring, organising and sharing information on QI practices vary widely among such repositories. Despite thoughtful efforts to develop standards and guidelines for scholarly reporting of QI, such as SQUIRE,7 ,11 and the emergence of dedicated peer-reviewed journals for QI reporting, such as BMJ Quality Improvement Reports,12 there has been less attention directed towards routine capturing of QI efforts by and for non-academic audiences.
The purpose of this study is to answer the following overarching review questions: what web-based QI repositories exist, what features do they present and how have they been evaluated? Specifically, we aim to (i) identify and review current approaches for sharing information on QI practices through publicly available, web-based repositories; (ii) categorise these approaches and document similarities and differences among them; and (iii) identify and review evaluations of the design, usefulness and impact of their online QI practice repositories. We were interested in learning how these web-based repositories identified, obtained, stored, presented and shared QI information, how users searched for and interacted with this information, and whether or not formal evaluations of the QI repositories were conducted, and, if so, what the benefit was for users and for the broader health system.
Methods
For the purposes of this study, we defined QI practices as organised efforts of change that aim to improve healthcare processes, patient health outcomes, broader health system performance and/or professional development13 ,14 as assessed through the standard dimensions of quality: safety, timeliness, effectiveness, efficiency, patient-centredness and equity.15 These QI practices may encompass a range of QI methods, projects, interventions, initiatives or case studies.
As web-based repositories of QI information are a relatively new phenomenon, there is little consensus on what they should be called. Many diverse terms have been proposed and are currently in use, including but not limited to QI repositories, portals, databases, lists and hubs. For the purposes of this study, we have defined QI repositories as searchable web-based tools for documenting and sharing information on QI practices that are intended for practitioner rather than academic audiences.
Given that our overarching review question, with its principal focus on identifying relevant websites, is not standard, most existing systematic review methods do not fully support our needs. Guided by an evolving understanding of the appropriateness of different types of systematic review method for a diverse range of needs in the health sector,16–18 we chose to follow a scoping review approach. While the scoping review methodology continues to evolve, a key strength is its focus on mapping indefinite content through iterative exploration and analysis.19–21 Therefore, we have modified the scoping review methodology to fit our particular needs with the key methodological features described below.
Search strategy
The search strategy involved three components, including a traditional database search, a grey literature search and consultation with experts. In order to be included, QI repositories had to be (i) web-based, (ii) publicly available, (iii) provide content on more than one QI practice, (iv) represent continuous (eg, more than a one-time) documentation of QI practices and (v) be undertaken in a healthcare context. The review was limited to (vi) English sources and (vii) over the period 2009–2014. All searches were conducted between October 2014 and February 2015.
Traditional database search
MEDLINE, EMBASE and CINAHL databases were searched using keywords related to three key elements: (i) quality improvement and (ii) repositories (or portal, database, list, hub) and (iii) the web (or online, website, web-based, internet). It is important to note that for the traditional database search we sought to identify journal articles that would lead us to a web-based QI repository. Therefore, identifying journal articles was not the end point for the search, but rather an interim step that directly informed follow-up search and review of the potential web-based QI repositories.
In order to find current, up-to-date and actively maintained web-based QI repositories, the search was limited to the most recent 5-year period as of the date searches were conducted (2009–2014). This decision aligned with our intention to identify QI repositories that were continuously updated and was strengthened by a test of our search strategy over a longer time period (2004–2014) that yielded no additional articles. All types of citation were included to maximise the possibility of identifying relevant QI repository websites.
Citation information was retrieved and titles and abstracts were screened independently by an investigator (JPB) and a research assistant, with any discrepancies reviewed by another investigator (MJD). Potentially relevant full-text articles were collected and reviewed independently by an investigator (JPB) and a research assistant, with another investigator resolving discrepancies (MJD). If full text was not available, the abstract was relied upon to identify a potentially relevant web-based QI repository. The websites that met our inclusion criteria for a web-based QI repository were systematically reviewed and charted across several characteristics, as described in the section ‘Charting methods’.
Grey literature search
Our scoping review of the grey literature consisted of two subcomponents, including (i) a targeted search of healthcare agencies/organisations involved in QI and (ii) a Google search on this topic.
For the targeted search, an initial list of agencies and organisations was compiled from two recent reports on Canadian and international QI activity.22 ,23 All of the agencies' and organisations' websites were reviewed to identify potential QI repositories. For the Google search, we adapted a set of emerging best practices for using Google in health sciences research.24 Using the same set of terms as in the traditional literature search, we applied expert search principles and practices, including the use of advanced search operators, information retrieval strategies and search hedges. As Google search results can number in the thousands or millions, we limited our review to the first 50 results (five pages) for discovering relevant websites.25 Two search filters within the ‘advanced’ search options page (date range (2009–2014) and language (English)) were applied to mirror the traditional literature search limits. All Google search results (ie, titles and excerpts) were screened by two members of the team (JPB and a research assistant), with any discrepancies reviewed by a second investigator (MJD). Websites deemed potentially relevant were reviewed by an investigator (JPB) and a research assistant, with discrepancies resolved by a second investigator (MJD).
Expert consultation
Finally, with institutional ethics approval from the University of Toronto Health Sciences Research Ethics Board, we consulted with QI experts to augment and validate the other search methods. We received input from colleagues at the Institute of Health Policy, Management and Evaluation at the University of Toronto and at IDEAS (Improving & Driving Excellence Across Sectors, a healthcare QI training programme; (http://www.ideasontario.ca)), who are known leaders in QI methods and practice. Experts were contacted by email with a short request to provide feedback on a working list of QI repositories. In particular, we asked them to help us identify any gaps in the list by providing suggestions for organisations that might have an online QI repository. Organisational websites identified through expert consultation were reviewed independently by an investigator (JPB) and a research assistant, with discrepancies resolved by a second investigator (MJD).
Charting methods
All QI repositories identified through the traditional database search, grey literature search or expert consultation and meeting the inclusion criteria were reviewed comprehensively. Results were charted according to a developed template to systematically extract relevant information from the QI repositories. The charting template was developed iteratively, starting with an initial set of characteristics that were based on features of known QI repositories and then modified as new data emerged from the additional QI repositories reviewed. Thus, the final charting template (table 1) is a reflection of an evolving set of criteria that is consistent with scoping review methodology wherein iteration is a central feature.19 All charting was performed independently by both a research assistant and an investigator (JPB), with discrepancies resolved by a second investigator (MJD).
Charting characteristics and definitions
We were interested in capturing broad features of each QI repository, including
how the site is hosted and/or supported (eg, does it have a parent organisation?);
what was the scope and size of the repository (eg, range of topics covered, geographical locations, the date range of included practices, the number of included practices);
what content was eligible and how it content acquired (eg, eligibility criteria for which practices are included, was content updated, if so how);
how the practice information was structured and searchable online (eg, range of formats for displaying information, what is the search functionality).
We also aimed to document aspects of each repository's outreach and impact, including whether any aspect of the QI repository had been evaluated for impact and value-add, what methods or avenues of user engagement were being used and whether the QI repository was connected to other organisations or leveraged through partnerships for promotion.
In the final analysis, the detailed charting was synthesised to summarise the range of practices among the different repositories, highlight concordant and discordant approaches and identify important gaps.
Results
A total of 13 web-based QI repositories met our inclusion criteria (figure 1). The traditional literature search yielded four web-based QI repositories. The grey literature search yielded a total of six web-based QI repositories, with two identified through a targeted search of organisations and four identified through a Google search. Finally, consultation with experts yielded three web-based QI repositories. The included QI repositories, their host organisation, country of origin and number of QI practices contained within the repositories are listed in table 2, while table 3 provides a summary of QI repository characteristics. Comprehensive charting results for each repository can be found in online supplementary tables.
Supplemental material
Summary of included quality improvement (QI) repositories
Quality improvement (QI) repository features
Flow chart of search results. QI, quality improvement.
Hosting
All QI repositories were sponsored by government, non-profit healthcare organisations, private foundations and/or publicly funded granting agencies. Host organisations were based in the USA (5), Canada (4) and the UK (4). Only 3 of 13 QI repositories provided a clear date for when they were initially established (see online supplementary table A).
Scope and size
Most repositories contained a majority of practices that were conducted within the host organisation's country of origin. Five QI repositories restricted content to specific locations, and while seven included international QI practices, the focus remained predominantly on practices conducted within the host organisation's country of origin. One QI repository included a multitude of international QI practices (the Canadian Foundation for Healthcare Improvement's Patient Engagement Resource Hub).
All but three repositories covered multiple QI topic areas. In the three that did not, they limited their topic focus to one of either health human resources, patient engagement or mental health.
The number of QI practices included in the 13 QI repositories ranged from 52 to >2000, with the majority (9 of 13; 69%) having <300 entries. Although many of the QI repositories do not document the date that new content is added to the repository, of those that did, no content was added before 2008 (see online supplementary table B).
Supplemental material
Content acquisition and eligibility
QI practices were largely user-submitted by the content creators (eg, QI project leader), although the process for submitting QI practices was not found for five QI repositories. For the remaining eight QI repositories, new QI practices could be submitted via online submission forms by email or traditional post.
Inclusion criteria were available for six QI repositories, with varying degrees of specificity. However, only four QI repositories applied a quality of evidence criterion to QI practices, including appraisals made by repository staff and user-submitted ratings. The most detailed evidence quality appraisal for reported QI practices is conducted by the Health Care Innovations Exchange of the US Agency for Healthcare Research and Quality in the United States, which applies an ‘Evidence Rating’ of strong, moderate and suggestive.
Information about how and when existing QI practices were updated, as well as the process for removal of practices, were explicitly available for only two QI repositories (see online supplementary table C).
Supplemental material
Content structure and search
The ways in which QI practice content was structured and presented varied greatly among the QI repositories. One commonality, however, was that most provided a title and brief summary for each QI practice. Additionally, many QI repositories included links to source materials or websites, as well as related information and/or downloadable content (eg, PDF summary documents).
The QI repositories displayed a range of search options for users to access content, including keyword, filtered and categorical searches. Most of the repositories had at least a basic keyword search function, if not an advanced option. Four repositories did not offer keyword search options, but instead offered other means of organising/identifying QI practices, such as listing practices by categories (see online supplementary table D).
Supplemental material
Evaluation and engagement
Based on the publicly available information for each QI repository, none included an evaluation of the impact or value-add of the QI repository. We also did not find information or reference to internal metrics or an evaluation plan for any of the QI repositories.
All but one QI repository engaged in some kind of user engagement, which involved engaging users through various methods including external social networking platforms (eg, Twitter, LinkedIn, Facebook, SlideShare, YouTube, Pinterest), and newsletters, list serves, Rich Site Summary feeds, discussion forums and blogs (see online supplementary table E).
Supplemental material
Discussion
We set out to identify and describe current approaches for sharing information on QI practices through publicly available, web-based QI repositories. While no clear gold standard exists for how to design a web-based QI repository, our review allowed us to identify and document the current range of features employed by various QI repositories and to highlight both prevailing consensus and apparent gaps or discordance in approaches.
Despite a comprehensive search strategy that included both traditional and grey literature and consultation with experts, our scoping review identified only 13 web-based QI repositories. While we cannot guarantee that we found all QI repositories that met our inclusion criteria, our findings suggest they are still relatively rare and have only begun to emerge in recent years. With increasing QI activity and interest,2–4 we anticipate information aggregating resources like QI repositories are likely to continue to grow in number, size and function. Thus, it is important to begin to identify, document and compare approaches, and highlight considerations for future focus.
Although we only had a small set of QI repositories to review and categorise, we found substantial variation across our five main areas of focus (ie, hosting, scope and size, content acquisition and eligibility, content structure and search and evaluation and engagement). Specifically, the QI repositories used different terminology (eg, QI practices vs case studies vs leading practices), focused on different approaches to content acquisition (both in their submission process and jurisdiction focus) and varied in terms of topic areas focused on. While there were few consistencies across repositories, all provided some means for organising content according to categories or themes and most provided access to at least rudimentary keyword search functionality. Notably, none of the QI repositories presented evaluations of their impact or explicitly presented an intention to develop internal metrics or plans for future evaluation.
One of our inclusion criteria was that repositories had to represent ongoing, continuous documentation of QI practices, that is, that they were actively maintained and regularly updated rather than a one-time record of QI activity. While all 13 repositories did meet this inclusion criterion, documentation of practice dates was not always clear or consistent. Only two repositories provided information about how and when existing QI practices already in the repository were updated to ensure accuracy of the content over time. And none of the QI repositories indicated processes for removing content, which may reflect their emergent status, but may become more important as the repositories grow and mature.
With content acquisition processes varying across the QI repositories reviewed, the impact of potential biases on what content is ultimately published is unclear. In particular, there are opportunities for QI repositories to be more explicit regarding any jurisdiction or topic area preferences, and the role, if any, of host organisations and sponsors on content publication decisions.
Another key finding was the varying types of information provided to describe QI practices among the different repositories. While some types of practice information were common across all repositories (eg, title of practice, location), other types varied across repositories (eg, dates of entry, contact information, quality of evidence). In contrast to recommendations for reporting on QI for scholarly publications,11 which emphasises the need to provide basic information on the context/setting for QI practices, most of the QI repositories provided only limited contextual information.
While just over half of the reviewed QI repositories identified a practice lead or contact person, there may be a strong rationale to include this information. QI practices are often led by front-line practitioners, who can provide important insights on the practice setting and contextualised learnings. For example, the ShareIDEAS repository requires QI practices to be accompanied by the name and contact information for an individual whom end users may reach out to and who has sufficient knowledge of the QI practice to provide relevant details and contextual information. ShareIDEAS also has a contact protocol in place where repository staff follow-up with the appointed contact person every six months to ensure information regarding existing entries remains up to date.
Our review suggests that social media is being used to engage users for many of the QI repositories. Further, since most practices reported in the QI repositories reviewed were user submitted, this suggests there is a core group of active/engaged users that could play a useful role in informing content and usability refinements to improve the impact of QI repositories. User engagement beyond the search function may include organising networking opportunities, creating virtual communities of practice, personalising online accounts or offering ties to QI training, coaching or funding and/or resources for QI projects.
Although user engagement through social media was prevalent, our analysis found that no QI repository currently conducts or reports on formal evaluations of its own impact on or benefit for its users, at least not explicitly or transparently. As this is a nascent field, more consideration is needed for how QI repositories should be evaluated and what metrics and other processes for self-evaluation could help QI repositories assess their success in meeting objectives. Building evaluative processes into the design of current and future QI repositories is an important next step to identify best practices and areas in need of improvement, as our review has begun to do.
Several limitations of our review should be considered. To keep the review feasible, we limited our search strategy to articles and websites produced in English. Ultimately, we identified QI repositories in only three countries (Canada, the UK, the USA). Important insights may be gained by broadening the search beyond English-language sources. However, we believe the study still provides preliminary findings that can be relevant and applicable to QI repositories produced in other countries and languages. We also limited our review to information made publicly available through each QI repository's website. It is possible that other non-public information (eg, inclusion criteria, evaluation reports or evaluation plans) exists elsewhere and could add greater depth and comprehensiveness to our findings. However, our approach was completed in a systematic way, relying on public sources that would reflect a user's viewpoint. Finally, this review was designed to identify web-based QI repositories, document the range of features available and the extent of evaluative work rather than recommend best practices. While the lack of evaluation generally is a concern, our review highlights the need to build on this review to critically assess the aims of QI repositories and the best designs and approaches to meet them.
Conclusion
Given that few QI repositories were identified in this study, it is perhaps not surprising that considerable variation was observed among them, with few indications of work to standardise approaches between repositories. This finding supports the need to summarise the current state of QI repositories and suggest considerations for improving current and future QI repositories. With the increasing reliance on QI in healthcare improvement and redesign, and growing interest in sharing and spreading best practices, the number of QI repositories is likely to increase over time. Designing QI repositories based on a broader knowledge base of the range of features already available and the principles and recommendations summarised here is an important starting point for improving their relevance, usefulness and impact.
We hope the findings from this study will assist individuals searching for information on QI practices to find an appropriate database for their needs among those we identified and also provide guidance to those aiming to improve existing QI repositories or considering establishing new ones.
Acknowledgments
The authors acknowledge the QI experts who provided feedback on this project and thank Stephanie Lim for her assistance with the search and review.
Footnotes
Contributors MJD had the idea for the study and is the guarantor of the paper. GG led the initial study design and JPB conducted the literature search and review. JPB and GG led writing of different sections of the paper, and all authors contributed to its critical review and revision. All authors contributed to the planning, conduct and reporting of the work, and gave final approval for the submitted version of the paper.
Funding This study was funded by a Partnerships for Health System Improvement grant from the Canadian Institutes of Health Research (no. PHE114126). The authors retain sole responsibility for this project.
Competing interests MJD was consulted on the design and development of IDEAS, one of the QI repositories reviewed.
Ethics approval University of Toronto Health Sciences Research Ethics Board.
Provenance and peer review Not commissioned; externally peer reviewed.