Article Text
PDF
Abstract
Objective Missed or cancelled imaging tests may be invisible to the ordering clinician and result in diagnostic delay. We developed an outpatient results notification tool (ORNT) to alert physicians of patients’ missed radiology studies.
Design Randomised controlled evaluation of a quality improvement intervention.
Setting 23 primary care and subspecialty ambulatory clinics at an urban academic medical centre.
Participants 276 physicians randomised to intervention or usual care.
Main outcome measure 90-day test completion of missed imaging tests.
Results We included 3675 radiology tests in our analysis: 1769 ordered in the intervention group and 1906 in the usual care group. A higher per cent of studies were completed for intervention compared with usual care groups in CT (20.7% vs 15.3%, p=0.06), general radiology (19.6% vs 12.0%, p=0.02) and, in aggregate, across all modalities (18.1% vs 16.1%, p=0.03). In the multivariable regression model adjusting for sex, age and insurance type and accounting for clustering with random effects at the level of the physician, the intervention group had a 36% greater odds of test completion than the usual care group (OR: 1.36 (1.097–1.682), p=0.005). In the Cox regression model, patients in the intervention group were 1.32 times more likely to complete their test in a timely fashion (HR: 1.32 (1.10–1.58), p=0.003).
Conclusions An electronic alert that notified the responsible clinician of a missed imaging test ordered in an ambulatory clinic reduced the number of incomplete tests at 90 days. Further study of the obstacles to completing recommended diagnostic testing may allow for the development of better tools to support busy clinicians and their patients and reduce the risk of diagnostic delays.
- diagnostic errors
- information technology
- ambulatory care
- quality improvement
- randomised controlled trial
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- diagnostic errors
- information technology
- ambulatory care
- quality improvement
- randomised controlled trial
WHAT IS ALREADY KNOWN ON THIS TOPIC
Studies have shown that 44% of diagnostic errors involve failures to order or follow-up on diagnostic testing. Each time that a patient misses a radiology test there is a missed opportunity that may result in diagnostic delay.
WHAT THIS STUDY ADDS
An electronic outpatient notification tool improved radiology test completion rates by 36% as compared with usual care among a cohort of patients who had a missed scheduled imaging study. Our study highlights a tool to facilitate loop closure for a vulnerable patient population to avoid diagnostic delays.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Future research can further hone outpatient notification tools to incorporate AI to target patients at high risk, address specific barriers to test completion, leverage technology to empower patients to engage in their own care and develop more targeted tools to improve diagnostic safety.
Introduction
Diagnostic errors are common and preventable, affecting as many as 1 in 20 US adults annually.1 Process-of-care failures figure prominently among missed and delayed diagnoses,2 3 including the failure to complete recommended tests and procedures. In an analysis of over 500 physician-reported diagnostic errors, Schiff and colleagues observed that 44% involved a failure to order or follow-up on laboratory or imaging tests.4 Each time a patient misses or cancels a radiology test to follow-up on an incidental finding, to evaluate a new symptom or to complete a standard cancer screening exam, there is a missed opportunity which may result in diagnostic delay.
While many electronic health records (EHRs) alert clinicians of abnormal and completed test results, few EHRs alert physicians of missed or cancelled studies. To address the vulnerability associated with missed diagnostic imaging, we developed an outpatient results notification tool (ORNT) that alerted physicians of patients’ missed radiology studies.2 In a pilot study involving three ambulatory clinics, we found that patients of clinicians who received a missed-test alert were 22% more likely to have completed the test within 90 days and that this effect was most robust among patients requiring modalities such as CT and MRI. To assess the reliability and validity of these results, we conducted a larger Quality Improvement study of 276 physicians randomising them to either ONRT or usual care to assess the impact on 90-day ambulatory radiology test completion rates. We hypothesised that the intervention would improve 90-day test completion rates among patients of intervention-group physicians compared with patients of usual-care clinicians.
Methods
Patients and project site
Our project was conducted at an urban academic tertiary care centre serving a diverse urban and regional referral population. Patients included all adult and paediatric patients seen at any of 23 hospital-based adult and paediatric primary care and subspecialty ambulatory clinics (online supplemental appendix 1) who had at least one ‘future’ (at least 5 days from date of order) imaging study ordered. Radiologic studies from all patients seen in the ambulatory primary care and subspecialty clinics were included. Patients received a combination of text, phone call and email reminders based on their personal preferences at 7 days, 4 days, 1 day and 60 min prior to the scheduled radiology test appointment. Clinics used a variety of EHRs during the study period, including eClinicalWorks (Westborough, MA), Centricity (General Electric, Boston, MA) and Soarian (Cerner, Kansas City, MO) with limited interface capabilities. All clinic and radiology appointment scheduling and results reporting used the hospital’s enterprise Soarian system. The study was performed from 15 February 2021 to 15 August 2021.
Supplemental material
Study design
This project evaluation used a random assignment of physicians to intervention and usual-care control groups to assess the efficacy of our ORNT to increase rates of 90-day test completion. Information about the project was shared broadly with the hospital medical staff and department administrators at meetings and via email. It was described as a quality improvement project intended to improve care follow-up and reduce diagnostic delays. Physicians were afforded the opportunity to opt out of the project; however, none chose to do so. We randomised 276 physicians across 23 clinics to intervention or usual care groups using a random number generator. Tests ordered by resident physicians and advanced practice providers were tracked to the responsible attending physician. Physicians were not blinded to assignment due to the nature of the intervention.
During the implementation phase of the project, we surveyed a convenience sample of physicians in the intervention and usual care groups regarding their awareness of missed radiology tests for their patients. We used this information to inform the intervention using classic plan–do–study–act cycles. Language in the email notification was modified to clarify to the provider that action may be required.
Intervention
An interdisciplinary team, including information technology and clinicians, collaborated to create an ORNT within the Soarian Workflow Engine (‘Workflow Engine’) with the goal of preventing missed diagnostic testing (online supplemental appendix 2). This tool identified ambulatory radiology tests scheduled greater than 5 days from the date of order entry. The notification tool then set a timer for 14 days after the scheduled test date. Tests scheduled within 5 days of order entry were suppressed due to high completion rates for same-day or same-week tests. The 14-day delay after the originally scheduled test date for provider notification was chosen to allow for usual-care practices to detect and address the missed tests. Cancelled or rescheduled tests automatically re-set the notification tool timer. The notification tool was piloted and refined as part of a previous study.2 If the test was not completed within 14 days, an email missed-test notification was sent to the responsible physician. Notifications were transmitted to the clinician’s hospital email address because of inconsistent use of embedded alert notifications in the various EMRs and perceived improved salience of email communication. Clinicians and department administrators were educated about email capabilities that allowed clinicians to auto-forward message recipients and to streamline communication with clinic staff. Clinic staff was responsible for rescheduling the patient for their missed test. Tests ordered by resident physicians and advanced practice providers defaulted to the responsible supervising physician.
Supplemental material
Patient and public involvement
This project was motivated initially by clinicians' observations about missed testing. The project question and design was then informed by including a patient (JM) as a full member of the study team, by discussions with the hospital patient and family advisory council and through patient interviews and focus groups incorporated into the project design. Patients were not involved directly in subject recruitment, as the intervention was randomised by physician, and the unit of analysis was the test result. Results were disseminated to through presentations to the hospital quality committees. The intervention burden was assessed by physician participants and was used to revise the intervention itself during the pilot phase and to permit participants to opt-out at any time.
Measures
The primary outcome was the rate of 90-day test completion from the scheduled test date in intervention group and usual care groups by imaging modality and overall. We also examined time to test completion.
Data analyses
Analyses were performed with the radiology test as unit of analysis. We began with 6188 missed radiology tests of 24 602 total ordered ambulatory studies from 15 February 2021 to 15 August 2021. We abstracted an electronic database that included all imaging tests scheduled to be performed from with a completion date at least 5 days in the future and incomplete by 14 days after the scheduled test date. The database included sociodemographic variables (age, gender, race, ethnicity, primary language and interpreter use and insurance type), clinic, responsible physician, imaging modality and type of radiology test, and dates associated with the date when the test was scheduled, the anticipated completion date and the date of any test that was ultimately completed.
Pearson’s χ2 and Wilcoxon rank sum tests were used to examine differences in patient characteristics and imaging modality by intervention and usual care group. The primary outcome, completion of imaging by 90 days after the missed test, was examined using logistic regression. Time to test completion was examined with Kaplan-Meier curves to compare completion by intervention and with Cox proportional hazard models. Observations were censored at 90 days if there was no follow-up image.
While randomisation occurred at the level of the physician, our preliminary analysis revealed statistically significant differences in patient characteristics between the intervention and usual care group—likely reflecting underlying differences in the panels of high-volume clinicians. To address the imbalance between the groups, we performed adjusted analyses using logistic regression incorporating variables that distinguished the groups (sex, age and insurance type). Multivariable models include a random effect for ordering physician, to account for non-independence of patients clustered by physician. Model results were considered statistically significant if p <0.05 using 2-sided tests; 95% CIs were also reported. Adjusted analyses were not performed stratified by testing modality due to small sample sizes. Analyses used Stata/SE V.16.0 and V.17.0 (StataCorp LLC, College Station, TX, USA). This project was reviewed in advance by the hospital’s investigational review board and determined to be a quality improvement project.
Results
Patient and test characteristics
We included 3675 radiology tests in our analysis: 1769 were ordered by physicians in the intervention group and 1906 by physicians in the usual care group. Patients in the intervention and usual care groups were well matched in terms of race and need for interpreter services, but intervention patients were slightly younger (median of 57 vs 59, p=0.0002), had a lower percentage of women (68.5% vs 71.6%, p=0.04) and included more patients with Medicaid insurance (25.4% vs 22.4%, p<0.001) (table 1). Intervention and usual care groups had a similar distribution of imaging test utilisation by modality (table 2). There were 2695 unique patients included in the analysis. Of those, 1973 (73%) had 1 test, 568 (21%) had 2 tests and remaining 154 (5.7%) had 3–11 tests.
Patientcharacteristics represented in usual care and intervention group
Imaging modality by group
Radiology test completion
As given in table 3, a higher per cent of intervention group patients completed CT, general radiology, nuclear medicine and ultrasound tests than their usual care counterparts. The results trended toward significance for CT (20.7% vs 15.3%, p=0.06) and were statistically significant for general radiology (19.6% vs 12.0%, p=0.02) and overall for all imaging modalities (18.1% vs 16.1%, p=0.03). In the unadjusted logistic regression analysis, the intervention group had a 21% greater odds of 90-day test completion than the usual care group (OR: 1.2 (1.018 to 1.433), p=0.03). In the multivariable regression model adjusting for sex, age and insurance type and accounting for clustering with random effects at the level of the physician, the intervention group had a 36% greater odds of test completion than the usual care group (OR: 1.36 (1.097 to 1.682), p=0.005; table 4).
90-day completion rate by modality
Adjusted logistic regression—with clustering (random effect for physician)
Time to test completion
The intervention group had a shorter time to test completion compared with the usual care group. Unadjusted Cox regression modelling showed patients in the intervention group were 1.19 times more likely to complete their test (HR: 1.19 (1.02 to 1.39), p=0.03). When the Cox model was adjusted for sex, age and insurance type and accounted for clustering with random effects at the level of the physician, patients in the intervention group were 1.32 times more likely to complete their test (HR: 1.32 (1.10 to 1.58), p=0.003, table 5).
Adjusted Cox proportional hazard regression—with clustering (random effect for physician)
Discussion
Implementation of an electronic ORNT was associated with a 36% relative improvement in the rate of test completion in the intervention group compared with the usual care group among a cohort of patients who had missed a scheduled imaging test. The magnitude of this effect varied by modality, with the greatest relative impact noted for general radiology and CT. While the magnitude of improvement varied across the various imaging modalities, we found a positive trend across all modalities except for mammography, which we hypothesise may be due to enhanced patient navigator services for cancer care that were already in place that may have attenuated the effect of the intervention. Overall, electronic closed loop notification of missed diagnostic opportunities can provide physicians and their staff an opportunity to interface with patients to reschedule these studies, complete more tests and reduce the risk of diagnostic delay—especially among a cohort of patients who had demonstrated a propensity to miss test follow-up.
Closing the loop on care is a longstanding challenge in healthcare. We know, for example, and that missed appointment rates vary from 5% to 48%, and that completion of high-value tests and procedures occurs at rates of 21%–42%.5 6 The aetiology of this phenomenon is complex and includes barriers such as the financial cost to patients, logistical challenges related to transportation and mobility, and competing family responsibilities including childcare.5–7 Access is often problematic, wherein patients have difficulty with scheduling of appointments at convenient times or places that are feasible given work or home obligations. We know that language, health literacy, communication and trust all conspire to affect patient–clinician communication and alignment on shared goals of care.
Numerous methods to facilitate care loop closure have been proposed and tested, with mixed results.8 Roseland and colleagues, for example, found that automated text and phone call reminder systems did not result in meaningful improvements in CT and MRI missed case rates.9
We elected to investigate an alert that signals to the responsible physician that a test was incomplete, and to build into physician workflow the opportunity to efficiently transmit that information to office-based staff who could follow-up and rescheduled missed tests. We selected a cohort that was comprised of patients who had missed a scheduled radiology test, a group that may face adherence challenges. In this ambulatory population, the missed rate was 25.1% during the study period. The fact that many of these patients did in fact complete the test suggests that a modest investment in patients at high risk can successfully close the loop on a pending test. Despite the impact of ORNT, the majority of patients in our cohort with a missed imaging test did not ultimately complete the test within the 90-day period, indicating a significant residual risk for missed or delayed diagnosis. This may indicate that additional, customised and targeted interventions are needed to match interventions more precisely to adherence barriers. It is also possible that some ordered tests may become clinically unnecessary since the initial order, thus obviating the risk of a missed diagnosis. Unfortunately, a planned qualitative component of our study with patient focus groups that may have answered some of these questions was unable to be completed and more research is necessary.
While many EHR systems allow clinicians to investigate pending but incomplete orders, few of them push the lack of result out to the responsible clinician or the care team. This approach stands in contrast to widely used results notification systems that transmit completed findings to the care team and, often, to the patient. Making the invisible missed test visible allows the responsible clinician to act in a way to investigate and mitigate the lapse. A similar approach has been tried in the past for unfilled medication prescriptions, but with mixed results. Interventions must not only make the alert available to the responding provider but should be presented in a that readily and seamlessly integrates into their workflow, and allows for a specified mitigation strategy or solution.10 Making this process ‘smart’ with AI or automation that identifies patients at highest risk, allows for a patient interface to elicit the cause of the lapse (forgetfulness, ambivalence, transport, child care, etc) and tailors the intervention to the obstacle, will be an important attribute of future ORNTs and will allow for targeted intervention without necessarily engaging the physician.11 Future work should explore ways to engage the patient directly in the rescheduling process without the physician’s attention. This approach would be less burdensome for physicians and improve the efficiency of this process. However, taking the physician out of the process would eliminate the clinician’s opportunity to reassess the test’s clinical value and the chance to reinforce the importance of the test with the patient. Leveraging population health tools for identifying missed preventive screening testing may offer models that inform the development of missed-test alerts. Rather than general screening methods, approaches that mine electronic health systems and registries for gaps in care related to warning signals or results requiring follow-up may be particularly helpful.12–15
Our project has several limitations. As a single institution study, the results may not be generalisable to other practice settings; replication at other institutions would strengthen our findings. Additionally, patients who moved or transferred care outside of our institution may have ultimately completed their tests at another location and imaging study would not have been captured in our database. As patients seen in resident clinic or by mid-level providers had alerts routed to the supervising attending, this may have diminished the effect of our intervention. Our project was conducted during an international COVID-19 pandemic, which likely influenced patient willingness to interface with the healthcare system for scheduled radiology tests or follow-up care. Additionally, we excluded studies scheduled within 5 days of order placement. Including those tests would have generated a higher completion rate but would have made it more difficult to detect a statistical difference in those we were most concerned about missing, which were those booked in the future. All tests included in our study were scheduled; therefore, we were unable to assess the percentage of ordered tests that were not scheduled appropriately. Finally, although we are able to show greater odds of 90-day test completion among the intervention group, the study was not powered to assess its impact on diagnostic errors. Physicians in the intervention group described compelling stories of ‘good catches’ prompted by the alerts, further study is needed to validate these observations.
In sum, an electronic alert that notified the responsible clinician of a missed imaging test ordered in ambulatory care reduced the number of incomplete tests at 90 days. Further study of the obstacles to completing recommended diagnostic testing may allow for the development of better tools to support busy clinicians and their patients and to reduce to risk of diagnostic delays.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
This study was approved by the Tufts Health Sciences Institutional Review Board.
Acknowledgments
We would like to thank Diep Kim for his contributions.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @NinaDadlezMD
Contributors NMD is the guarantor of this manuscript. NMD contributed to study design, implementation of the intervention, data analysis and wrote the manuscript. AMLC contributed to study design, reviewed the data and critically reviewed and edited the manuscript. SW and NM participated in data collection and refining intervention, reviewed the data and critically reviewed the manuscript. WFH contributed to study design, implementation and refining of intervention, reviewed the data and critically reviewed and edited the manuscript. KR contributed to study design, reviewed the data and critically reviewed and edited the manuscript. JM contributed to study design, implementation and refining of intervention, reviewed the data and critically reviewed and edited the manuscript. EL contributed to designing and refining the intervention, study design, reviewed the data and critically reviewed and edited the manuscript. BCK contributed to data review and analysis and critically reviewed and edited the manuscript. SNW contributed to study design, implementation and refining of intervention, reviewed the data and critically reviewed, edited and wrote portions of the manuscript.
Funding This quality improvement project is funded by the Society to Improve Diagnosis in Medicine under a grant from the Gordon and Betty Moore Foundation.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.