Article Text
PDF
Abstract
Introduction Patient-reported outcomes (PROs) are important for research, patient care and quality assessment; however, large-scale collection among the US surgical patient population has been limited. A structured implementation and dissemination programme focused on electronic PRO collection could improve the use of PROs data to improve surgical care. This study aims to (1) evaluate the feasibility of PRO collection among a larger volume of surgical patients through the stepwise implementation of PRO collection processes in a sample of American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) hospitals; (2) identify best practices and barriers to PRO collection through qualitative study of participating hospitals and patients; and (3) evaluate the utility of PROs at detecting differences in the quality of care among surgical patients.
Methods and analysis ACS NSQIP-participating hospitals are being recruited, and patients at participating hospitals who undergo elective surgical procedures receive invitations via e-mail or short message service ‘text’message to complete PROs after surgery. Validated PRO measures which evaluate physical and mental health-related quality of life, pain, fatigue, physical function and shared decision-making were selected. The scalability of PRO collection will be assessed by site enrolment, patient accrual and response rates. Qualitative interviews and focus groups will be performed with patients and hospital personnel to identify best practices and barriers to successful enrolment and PRO collection. Multivariable hierarchical regression models will be used to evaluate the distinctness of PROs from clinical outcomes captured in ACS NSQIP and the ability of PROs to detect differences in hospital performance.
Ethics and dissemination This study was reviewed by the Advarra Institutional Review Board (IRB) and deemed to be exempt from IRB oversight. Findings will be disseminated through peer-reviewed manuscripts, reports and presentations.
- Healthcare quality improvement
- Access to Information
- Health services research
- Quality improvement methodologies
- Patient Reported Outcome Measures
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- Healthcare quality improvement
- Access to Information
- Health services research
- Quality improvement methodologies
- Patient Reported Outcome Measures
Introduction
Patient-reported outcomes (PROs) are health-related outcomes obtained directly from patients.1 PROs measure aspects of health, such as quality of life (QOL), physical function, pain or other symptoms, that would otherwise be nearly impossible to directly assess. PROs are assessed using validated questionnaires called PRO measures (PROMs), which permit rigorous measurement of these subjective outcomes. PROs give patients a voice in their healthcare by including outcomes that are important to them and have been shown to be an effective tool for clinical monitoring of patients, resulting in improved survival and decreased hospital encounters.2 3 Furthermore, PROs have been incorporated into multiple international clinical registries as measures of quality.4–6
Although the importance of PROs is recognised, large-scale collection among the US surgical patient population has been limited, largely to patients undergoing orthopaedic, urologic and plastic surgery procedures.7 8 In part, the lack of widespread PRO measurement may be due to the logistical and administrative barriers associated with large-scale PRO collection, which are more challenging than abstracting traditional clinical outcomes from an electronic medical record. Additionally, as the US healthcare system is made up of numerous public and private insurers and hospitals, routine system-wide PRO collection from surgical patients is difficult. However, the absence of PROs from surgical quality improvement (QI) represents a significant shortcoming, and efforts are therefore needed to incorporate PROs into surgical QI programmes.
The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is a prominent programme for surgical QI with more than 700 participating hospitals and collection and dissemination of risk-adjusted data for more than 1 million surgical procedures annually, with an estimated 400 000 (40%) ambulatory procedures.9 ACS NSQIP records many clinical outcomes within the 30-day period following a surgical procedure, including mortality and morbidity. Since its creation, ACS NSQIP has been a main driver of surgical QI programmes and has significantly improved surgical outcomes.10
ACS NSQIP is a robust programme which allows us to incorporate PROs and holds the potential to measure patient-centred outcomes and to help improve shared decision-making by providing relevant information to guide choices. Traditional clinical outcomes currently collected by the ACS NSQIP have limited utility following ambulatory surgical procedures. For these procedures, quality from the patient’s perspective is likely less related to the rates of rare complications, such as death, than to QOL or other individual outcomes. Additionally, the ACS NSQIP Risk Calculator, a tool developed to aid shared decision-making about the risks of surgical procedures, can estimate traditional surgical outcomes, but does not directly measure patient-centred outcomes such as QOL, which may be more salient to patients.11 PRO collection would also create new avenues for surgical QI research opportunities and result in a more patient-centred, comprehensive assessment of surgical quality. Widespread collection through a national surgical QI programme can be used to provide immediate feedback to patients on their responses in relation to their peers as well as increase patient satisfaction acknowledging the patient voice to influence the surgical quality of care for future patients.
As an initial step to explore PRO collection in ACS NSQIP, multiple stakeholders were engaged and an alpha pilot study was performed to assess the feasibility and validity of PRO collection from surgical patients.12 Between October 2017 and March 2018, 1324 patients from 17 ACS NSQIP hospitals responded to an electronically delivered postoperative questionnaire, achieving an estimated 20% response rate. This study demonstrated that PRO collection from surgical patients via the ACS NSQIP is feasible and suggested that PROs may have adequate discriminative ability to detect variations in care.
Scaling such an initiative to the national level, however, faces additional challenges. This study aims to (1) evaluate the feasibility of PRO collection among a larger volume of surgical patients through the stepwise implementation of PRO collection processes in a sample of ACS NSQIP hospitals; (2) identify best practices and barriers to PRO collection through qualitative study of participating hospitals and patients; and (3) evaluate the utility of PROs at detecting differences in the quality of care among surgical patients.
Study description
Study design
This is a feasibility study evaluating the scalability of electronic collection of PROs through a national surgical QI programme. This study has three aims and will be further detailed. First, the spread of PROs across a diverse, national sample of hospitals will be evaluated. Second, best practices and barriers to participation by hospitals and patients will be identified through qualitative feedback, and the programme will be iteratively improved based on these findings using the plan–do–study–act (PDSA) QI framework. Third, the utility of the PRO data obtained through the programme will be assessed for its ability to identify meaningful differences in care quality.
Patient and public involvement
To ensure that the development of this study incorporated input from patients and the public, a group of key stakeholders, including payers, patient advocates and policymakers, participated in in-person focus groups to provide feedback regarding the domains which would be measured using PROs and the methods by which PROMs would be administered. Patient interviews will be performed as part of the study protocol to understand the patient experience and identify patient-centred barriers. Finally, the study findings will be disseminated to the public through scientific presentations and publications. Results will also be distributed to all ACS NSQIP hospitals, regardless of project participation.
Study aim 1: demonstrate feasibility and scalability of routine electronic PRO collection
The first aim of this study is to evaluate the feasibility of routine electronic PRO collection after surgical procedures on a national level. Building on a previous pilot study which assessed the collection of PROs on a limited scale,12 this aim entails expanding routine PRO collection among surgical patients across a national surgical QI programme in a more structured manner. As PRO collection on this scale has not previously been accomplished among surgical patients in the USA, this presents the opportunity to leverage the unique infrastructure of ACS NSQIP to aid in the spread of routine PRO collection.
Study population and recruitment
All US hospitals which participate in ACS NSQIP are eligible. Due to differences in patient privacy and data management legislation, hospitals outside of the USA will not be included in this initial study. Hospital recruitment will occur in two waves (figure 1). During wave 1, hospitals associated with members of the study team and those which participated in the alpha pilot study will be recruited. Additional recruitment efforts will be extended to small and/or rural hospitals and to ACS NSQIP collaboratives, which are multihospital collaboratives based around geographic proximity, membership in a shared healthcare system, or the desire to focus on a specific disease.13 ACS NSQIP hospitals share data and best practices with the common goal of improved patient outcomes and have existing QI infrastructure, making them ideal for recruitment. Finally, hospitals will be recruited through presentations at the annual ACS Quality and Safety Conference, which focuses on various ACS QI programmes and is regularly attended by more than 2000 individuals. Following qualitative assessment and process improvement, a second wave of hospital recruitment will occur by leveraging existing ACS NSQIP infrastructure and lessons learnt.
Strategy for hospital recruitment and process improvement. Schematic depicting strategy for recruitment and study execution. In wave 1, hospitals will be recruited through ACS NSQIP collaboratives and from among the study team members’ hospitals, participants in the alpha pilot study and hospitals which are small or located in rural areas. PRO collection will commence, and iterative PDSA improvement cycles will be used to improve recruitment and patient enrolment processes. Subsequently, formal focus groups and interviews will be held with patients and hospital personnel to identify barriers and best practices, which will be assembled into a toolkit of best practices. After incorporating these findings, additional academic and community hospitals will be recruited. ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; PDSA, plan–do–study–act.
All ACS NSQIP-eligible patients who undergo an elective ambulatory (ie, 23-hour stay) surgical procedure at a participating hospital will be eligible for PRO collection. ACS NSQIP employs a sampling algorithm based on a rotating 8-day schedule which randomly samples cases for inclusion; therefore, only those patients who are selected by the sampling algorithm will be enrolled. Included patients must also have a valid e-mail address or mobile telephone number entered into the ACS NSQIP record. Patients who meet ACS NSQIP exclusion criteria (age<18 years, minor cases, solid organ transplant or organ procurement operations, operations for trauma, hyperthermic intraperitoneal chemotherapy) will be excluded, as will patients who expire within 30 days of surgery.
Patient-Reported Outcome Measure (PROM) selection
The PROMs selected for use in this study were identified through a multiphase discernment process with involvement of multiple stakeholders as detailed above. Focus groups consisting of surgeons who practice general, breast, colon and rectal, endocrine, hepatopancreatobiliary, thoracic and trauma surgery or surgical critical care were held at the 2016 ACS Clinical Congress to further discuss clinical domains which were identified to be important to capture among surgical patients. A total of 19 surgeons participated in these focus groups and three domains were identified: a global health-related QOL, pain and the surgical care experience.
To assess these domains, a literature search was performed which identified multiple PROMs, three of which were selected for inclusion in the alpha pilot study: Patient-Reported Outcomes Measurement Information System (PROMIS) – Global Health, PROMIS – Pain Interference and the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS).14 15 After completion of the pilot study, the list of included PROMs was modified with the addition of physical function and fatigue measures. Furthermore, the S-CAHPS measure was replaced with two measures of shared decision-making, as this more directly related to a surgeon’s performance. In total, 5 instruments consisting of 34 questions were selected (table 1). During testing, PROM completion took approximately 5–7 min. All PROMs are available in English and Spanish translations.
Patient-reported outcome measures selected for inclusion
Electronic PRO collection process
After surgery, patients will be selected for inclusion based on the ACS NSQIP sampling methodology. After a patient is selected, ACS NSQIP surgical clinical reviewers (SCRs) who abstract clinical outcomes enter patient contact information (e-mail address and/or mobile telephone number) and surgical characteristics into the ACS NSQIP platform (figure 2). These variables will then be used to determine the patient’s eligibility for PRO collection. Patients will be invited to complete the selected PROMs 45 days after their procedure and can complete the PROMs until postoperative day 90. This time interval was selected as it occurs after the acute surgical recovery period and permits measurement of durable outcomes instead of short-term outcomes which may fluctuate during the immediate postoperative period.
Patient enrolment and PRO collection process. Schematic depicting the process of patient enrolment, PRO collection and inclusion of data into ACS NSQIP. ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; POD, postoperative day; PROs, patient-reported outcomes; PROM, PRO measure.
PROs will be collected using a customised version of the IQVIA Connection platform (IQVIA, Durham, North Carolina), a third-party electronic survey administration platform. Dependent on the contact information provided, eligible patients will receive either an e-mail or a short message service ‘text’ message containing a unique link to the secure, Health Insurance Portability and Accounabiliy Act (HIPAA)-compliant, browser-based PRO collection portal on postoperative day 45. After following the link, patients will receive a brief explanation of the use of PROs and are given the opportunity to decline to participate. Patients who agree to participate are asked to complete the PROMs (figure 3). Patients will receive reminder messages 60, 75 and 90 days after surgery if they have not completed all PROMs. PROs provided by the patient will then be incorporated into their ACS NSQIP case record along with clinical outcomes.
Electronic PRO collection platform interface. Representative image of the electronic PRO collection platform interface that is displayed to patient. ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; PRO, patient-reported outcome.
Process improvement
During hospital enrolment and initiation of PRO collection, process improvement will be performed iteratively using the PDSA QI framework (figure 1). At each phase of the project, informal feedback will be gathered from representatives at participating sites, including hospital administrators, surgeons and SCRs, through monthly project meetings. Additionally, performance metrics including patient enrolment numbers and response rates will be monitored regularly. Based on the feedback received and performance metrics, process improvements will be implemented for the hospital enrolment, patient enrolment and PRO collection. The nonadoption, abandonment, scale-up, spread, and sustainability framework by Greenhalgh et al will be applied to identify solutions for limited large-scale sustained adoption and programme failures.16
Outcomes and data analysis
The success of scaling electronic PRO collection will be assessed based on the number of participating hospitals, the number of patients enrolled and patient response rates. Response data will be analysed to identify the presence of non-response bias among patients based on sociodemographic or clinical characteristics. Additionally, the characteristics of patients with and without the necessary contact information for enrolment will be compared. As improvements are made to the PRO platform, trends in patient enrolment numbers and response rates will be monitored to assess the effectiveness of each intervention. All statistical analyses will be performed using SAS version 9.4 (SAS Institute, Cary, NC).
Study aim 2: identify best practices to spread PRO collection
The second study aim focuses on a formalised qualitative evaluation to identify program-wide and institutional best practices. Semistructured interviews will be held with patients and focus groups will be held with hospital stakeholders, including surgeons, administrators and SCRs. These sessions will explore patient perspectives, barriers and facilitators for completing PROMs and will attempt to identify best practices which have encouraged institutional participation, patient enrolment or patient responsiveness. The identified best practices will be assembled into an educational ‘toolkit’ which will be disseminated to participating hospitals. Additionally, barriers to institutional participation, patient enrolment or patient responsiveness will be identified. Based on these barriers, the PRO collection programme will be modified to improve institutional and patient engagement.
Recruitment strategy
Recruitment of patients for interview participation will occur through participating hospitals recruited for at least 6 months. Patient recruitment will continue until thematic saturation is reached and is expected to fall between 20 and 30 patients. To guarantee that our sample is representative of the surgical population, purposive sampling will be used to ensure diversity in patient age, race/ethnicity, gender, procedure type, presence of postoperative complications and the type of hospital. Additionally, patients who did and did not complete PROMs will be included to aid in the identification of barriers to completion and avoid selection bias. Interviews will occur via telephone, and patients will receive a gift card to an online retailer as compensation.
Focus group participants for the hospital stakeholder interviews will be recruited through e-mail communication. Each of three focus groups will consist of 5–8 randomly selected participants, including surgeons, hospital administrators, SCRs and one representative from the research team, with a target of 15 participants. Focus groups will be held virtually over the Zoom videoconferencing platform (Zoom Video Communications, San Jose, California).
Data analysis
Audio recordings of interviews and focus groups will be transcribed by a professional service. Transcripts will subsequently be analysed using the immersion–crystallisation method,17 which is characterised by repeated cycles of immersion in the collected data and subsequent crystallisation into primary themes. A qualitative codebook will be developed incorporating the identified themes, and all transcripts will be coded independently by two members of the study team using NVivo software (QSR International LLC, Melbourne, Australia). Coded transcripts will then be reviewed by the two team members, who will adjudicate any disagreements in coding. Themes identified during coding will be synthesised to develop an educational toolkit and to inform process improvements to the PRO collection programme.
Study aim 3: explore the potential of identifying quality gaps using PROs
The third study aim consists of assessing the utility of PRO data at identifying gaps in the quality of care. This will be accomplished in three ways. First, the discrimination of individual PROMs will be assessed to identify differences in outcomes among hospitals and surgeons. Second, the statistical reliability of PRO data will be assessed. Third, the relationship between PROs and ACS NSQIP clinical outcomes will be explored to determine whether PROs permit the assessment of quality distinct from routinely captured clinical outcomes.
Statistical analysis
Each PROM will be assessed for its ability to identify differences in quality among hospitals and among surgeons. This will be accomplished by using hierarchical multivariable regression models.18–20 Models will be constructed for each outcome incorporating surgeons and hospitals as random effects, with clinical characteristics as fixed effects. Subsequently, the statistical reliability of PROs as a measure of quality, defined as the proportion of total variability in a hospital performance metric due to between-hospital variability, will be evaluated using variance partitioning techniques.20 21 Finally, to evaluate the added discriminative ability of PROs beyond clinical outcomes multivariable hierarchical regression models will be constructed, and alignment of hospital-specific and surgeon-specific performance estimates for PROs and clinical outcomes will be compared. These methods will form the basis of potentially formulating PRO-based performance measures.22
Limitations
While this study is the first to harness a robust surgical QI programme to scale national PRO collection after ambulatory procedures across multiple sites, some limitations should be considered. This is a feasibility protocol intended to assess the widespread capture of PROs across a large volume of surgical patients and so collection of PROs is restricted to one timepoint. Results and lessons learnt from this protocol will be used to inform best practices for PRO collection and future studies, which will include sampling across multiple timepoints. Additionally, while this study uses PROMS and PROs to understand and influence improvement of surgical quality of care at the national level, we believe there will be downstream effects to clinical care at the individual patient level.
Ethics and dissemination
This study was reviewed by the Advarra Institutional Review Board, which deemed the study to have minimal risk to patients, and therefore exempt from further oversight (Project ID#: Pro00022840). Patients at enrolled sites who are invited to complete PROs are not obligated to participate and are presented at the time of invitation the option to decline to participate, in which case they will not receive any additional communications. Patients are informed that any information collected will remain confidential and will be accessible only to personnel at their hospital and at the ACS. Only patients at hospitals participating in the ACS NSQIP are eligible for participation.
The findings of this study are anticipated to have meaningful impact on surgical QI in the USA. The successful incorporation of PROs into a national QI programme like ACS NSQIP would result in a major shift in how surgical quality is measured, by complementing clinical outcomes with PROs. Widespread measurement of patient-centred outcomes through PROs would provide generalisable data that could be incorporated into preoperative discussions, improving shared decision-making.
To ensure dissemination of the study findings, the infrastructure of ACS NSQIP will be leveraged. Study findings will be shared with all hospitals which currently participate in the programme and will be used to guide the future of ACS NSQIP. Additionally, findings will be disseminated through publication in peer-reviewed scientific journals and presentations at national meetings. The study findings hold the potential to spread beyond ACS NSQIP, with PRO collection possibly being incorporated into other ACS Quality Programs.
Ethics statements
Patient consent for publication
Footnotes
ALP and LKT contributed equally.
Contributors JBL, CYK, ALP and LKT contributed to the conception of the study. All authors participated in the design of the study; read, revised and approved the final manuscript; and agree to be accountable for all aspects of the work ensuring questions related to its accuracy or integrity.
Funding This work was supported by the Agency for Healthcare Research and Quality grant number 1R18HS026189-01A1. BCB was supported and ADM is supported by the American College of Surgeons as part of the Clinical Scholars in Residence program and by a training grant from the National Cancer Institute (T32CA247801).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.