Article Text
PDF
Abstract
Background Careful design of preprinted order sets is needed to prevent medical overuse. Recent work suggests that removing a single checkbox from an order set changes physicians’ clinical decision-making.
Local problem During a 2-month period, our coronary care unit (CCU) ordered almost eight times as many serum thyroid-stimulating hormone (TSH) tests as our neighbouring intensive care unit, many without a reasonable clinical basis. We postulated that we could reduce inappropriate testing and improve clinical laboratory stewardship by removing the TSH checkbox from the CCU admission order set.
Methods After we retrospectively evaluated CCU TSH ordering before intervention, the checkbox was removed from the CCU admission order set. Twelve weeks later, we commenced a prospective 2-month assessment of TSH testing and clinical sequelae of thyroid disease among all CCU admissions. If clinical indications were absent or testing had occurred within 6 weeks, TSH requests were labelled as ‘inappropriate’.
Results Physician ordering and, specifically, inappropriate ordering decreased substantially after the intervention. In 2016 among physician-ordered TSH tests, 60.6% (66/109) were inappropriate; in 2017 this decreased to 20% (2/10, p=0.01). Overall, the net effect of checkbox removal saw the decrease in TSH testing without clinical indication outweigh an increase in missed testing where indications appear to exist.
Conclusions Provision of an optional checkbox for a laboratory test in an admission order set can promote overuse of laboratory resources. Simple removal of a checkbox may dramatically change test ordering patterns and promote clinical laboratory stewardship. Given our reliance on order sets, particularly by trainees, changes to order sets must be cautious to assure guideline-directed care is maintained.
- clinical decision-making
- medical overuse
- TSH
- intensive care units
- coronary care units
Statistics from Altmetric.com
Introduction
Year over year, laboratory testing for inpatients and outpatients has increased exponentially in North America, and multiple studies have documented significant laboratory overutilisation.1 Prior to the widespread clinical use of order sets, efforts to reduce low-value blood testing in hospitals employed educational interventions with moderate success.2 3 A recognised limitation of this approach is lack of long-term sustainability of intervention behaviour in the absence of sustained feedback for ordering physicians.4 Furthermore, multiple reasons for overordering exist, including defensive medicine, local ‘routine’ practices, complex clinical cases and standard protocols that were recently reviewed.5 6 The latter of these include preprinted order sets, used in the inpatient and outpatient setting to improve adherence to guideline-directed patient care4 but with associated, unintentional increases in low-value blood testing.7 8
In the realm of inpatient critical care, where order sets have become standard, the subsequent routine ordering of large volumes of blood work has been a target for change. A recently published study employed a multistep educational and electronic medical record (EMR) altering intervention that improved blood work stewardship in the intensive care unit (ICU) setting with no adverse impact on length of stay and 9-month mortality.9 Similarly, changing computer order entry blood work from preselected to optional has been shown to decrease the quantity of blood work ordered in clinical practice.10 11 But not all efforts to curb excessive ordering in this context have been successful. Rosenbloom et al experienced an unexpected paradoxical increase in ordering after introducing an intervention aimed to reduce serum magnesium testing.12 Such surprises suggest that one should be diligent in assessing unexpected consequences of streamlining order sets, which could conceivably include missed diagnoses. Fortunately, a recent review of thyroid-stimulating hormone (TSH) testing, the focus of our study, indicates that in addition to potential cost-effectiveness, deferring TSH testing in patients who are critically ill may be wise as the results are complex to interpret and rarely impact care.13
The concern fuelling this current study was the observation that over a 2-month period, our coronary care unit (CCU) ordered almost eight times as many serum TSH assays as our neighbouring ICU. In contrast to our ICU, our CCU used an admission order set with a TSH checkbox. Moreover, most CCU patients with TSH ordered had no evidence of a thyroid disorder. We postulated that the number of unnecessary serum TSH orders would significantly decline if the TSH checkbox was removed from the admission order set (online supplementary appendix 1). Thus, the specific aim of this project was to evaluate the anticipated changes in TSH assay ordering, both in regard to frequency and appropriateness, brought about by the removal of the associated checkbox from our CCU order set.
Supplemental material
Methods
Setting
At our teaching hospital, the nine-bed CCU is run by an on-call cardiologist, with 971 admissions in 2016 and 910 admissions in 2017. The average daily census in these years was 2.66 and 2.5 admissions, respectively. There are usually 5 rotating residents and medical students per month, 31 nurses on a yearly shift schedule, 3 rotating unit clerks who work during daytime hours and a cardiology fellow on call after hours and on weekends. Once ordered by a physician, most of the blood work is entered by the charge nurse and unit clerk; however, all nurses have access to all applications and will frequently enter blood work ordered on their own specific patients. All ‘standard’ blood work is ordered on the preprinted order set by checking the appropriate boxes, with an additional space to order further blood work if needed.
Initial evaluation
The problem definition phase of the project began with the suspicion of excessive TSH assay requests in our CCU. Retrospectively reviewing ordering data from a 2-month period (1 May to 30 June 2016), this concern was further validated and quantified. It was noted that a percentage (25%) of TSH assays were ordered during the admission without a documented physician order. Most of the orders were thought to represent verbal orders from the physician or transcription errors. From this it was hypothesised that frequent ordering was driven by the format of the admission order set, namely the inclusion of a checkbox for TSH ordering. We proceeded to test this theory in a simulation, the details of which have been published elsewhere,14 and found that the presence of the checkbox did appear to influence ordering.
Intervention and measurement
With this supporting knowledge, we designed a preintervention/postintervention cohort analysis, comparing the frequency of TSH ordering among the cohort of patients examined during our initial evaluation phase before intervention to a cohort of CCU patients after intervention. Our intervention removed the TSH checkbox from the CCU order set on the unit; to access this test, physicians now had to specifically list TSH as an ‘other test’. Through the appropriate administrative channels, the order set was modified and accessible through an online portal for printing. We used our neighbouring ICU as a control group as their admission order set did not contain a TSH checkbox. After a 12-week integration interval, the same two reviewers who had assessed the preintervention cohort prospectively evaluated the charts of the postintervention cohort, consisting of all subsequent patient admissions to the CCU for 1 May to 30 June 2017. The time period corresponded to the same 2 months during which the preintervention cohort had been reviewed in the previous year to control for possible seasonal variation in cardiovascular disease admission and for any other unknown calendar-dependent factors. If charts did not contain an admission order set, they were excluded as they did not represent true admission but rather temporary residence within the CCU for postinterventional recovery, pericardiocentesis or cardioversion. Any considered to be true admissions but found to have been managed using the previous, unaltered order set were recorded, allowing insight into potential selection bias and adequacy of intervention fidelity, although these patients were not included in the analysis.
During the chart review, we recorded admitting diagnosis, demographic information, atrial fibrillation status, heart failure status, history of thyroid disease, history of thyroid replacement therapy, history of amiodarone therapy and any documented history of thyroid-related symptoms,15 including fatigue, weakness, temperature intolerance, weight change, skin change and hair changes. The criteria used to judge clinical appropriateness of TSH ordering included any typical historical symptom, new atrial fibrillation,16 new heart failure,17 current amiodarone therapy, current thyroid replacement or any history of thyroid disease. These very inclusive criteria were used to decrease our risk of a type I error (ie, misclassifying an appropriate order as inappropriate).
If all of the previously named criteria were absent based on a thorough review of the admission consultation and electrocardiograms, the TSH was labelled as ‘inappropriate’ for the purposes of data analysis. If a TSH assay had been performed in the last 6 weeks and was accessible via our hospital’s EMR at the time of admission, the order was also labelled as inappropriate. This exception is based on the knowledge that changes in TSH occur over weeks, not days.18 Whether or not there was an actual physician-signed order (referred to in the Results section as ‘charted order’) for the TSH assay was also tracked, as the absence of such an order would suggest the TSH was added on by a clerk, a nurse or as part of a physician’s unsigned verbal order. If thyroid studies were abnormal, regardless of a clinical indication, a more in-depth chart review was pursued to evaluate the blood work’s impact on patient care. We also documented when ordering was not undertaken among patients with any of the above reasonable indications for testing, evaluating the frequency of possibly ‘missed’ thyroid disease detection. To promote consistency in the application of these criteria, the reviewers reviewed a small number of charts together at the beginning of preintervention data collection, and any uncertainties throughout the chart review process were resolved by consensus.
Although we hypothesised that physician behaviour was the primary driver of overordering, the role of others involved in the testing process was recognised as an additional important consideration when aiming to effect overall change within the CCU. To further evaluate the perspectives of others involved in test ordering, informal discussions involving all ward clerks, three senior front-line nursing staff and multiple medical students and residents rotating through the service were undertaken either during or after the 2-month data after intervention collection period. These were primarily conversations initiated with staff while at work in the CCU, intentionally informal in tone to avoid introducing bias by unintentionally encouraging interviewees to modify ordering behaviour. Questions were asked regarding each professional’s approach to ordering TSH, entering the order and acting on specific results.
Analysis
Descriptive statistics of the sample, both before and after intervention, were undertaken using means and SDs for continuous variables and frequencies with proportions for categorical variables. Comparisons between the two cohorts, performed to appraise potential confounding, were made using the t-test for means and the χ2 test for proportions. Endpoints evaluated included the proportion of patients with any TSH assay request, as well as those with orders specifically signed by a physician; comparisons between the groups were again made using the χ2 test. Change in physician ordering among patients without an indication (ie, inappropriate ordering) was similarly assessed. Any potential differences in patient characteristics before and after intervention (p<0.20) which may have impacted overall and physician-specific ordering were adjusted for using logistic regression modelling. Proportions with at least one indication for testing who did not have a TSH assay ordered by a physician were also determined for each time period. Changes over time, their sustainability and their comparability with ordering patterns in the ICU were reviewed visually using run charts created with Microsoft Excel. Statistical calculation and comparisons were undertaken using SAS software, V.9.4 (SAS Institute).
Evaluating our conversations with staff members, notes for each discussion were reviewed. Recurring dynamics and concerns were identified and subsequently paraphrased to protect participant’s confidentiality on presentation.
Results
Patient characteristics from both the 2016 and 2017 observation periods are presented in table 1. In the postintervention time period, the subjects sampled tended to be slightly younger, more often male and had marginally less documentation of new atrial fibrillation or potentially thyroid-related symptoms (table 1) compared with the 2016 sample. Although every effort was made to purge the original order set which had been stored in paper form on the unit, six postintervention subjects who were eligible for standard care had the previously unaltered order set used. Given their small number and the fact that their findings did not contribute to evaluation of our primary interest, the effectiveness of the actual altered order set, these patients were excluded from the analysis. The fidelity of the intervention was 95%, as measured by the number of modified order sets used. We grouped TSH results within each cohort based on whether or not the patient had a corresponding physician-signed charted TSH order or not (table 2). Before and after intervention, 135 of 158 (85.4%) and 43 of 118 (36.4%) admissions, respectively, underwent TSH testing (p<0.0001), whether specifically ordered by a physician or not. Adjustment for differences in preintervention and postintervention characteristics did not change the association between the intervention and reduced testing (online supplementary appendix 2). Our informal control group, the neighbouring ICU, did not appear to experience a change in TSH testing frequency (figure 1).
Supplemental material
Patient characteristics
Categorisation of thyroid-stimulating hormone (TSH) testing based on preintervention and postintervention cohort status, physician ordering, actual testing performed and presence or absence of clinical indication
Thyroid-stimulating hormone (TSH) assay lab request frequency before and after intervention in both the coronary care unit (CCU) and intensive care unit (ICU).
Across all patients, the number who had a physician-signed charted order decreased from 109/158 (69.0%) to 10/118 (8.5%) (p<0.0001) between the two time periods. Again, adjustment for group differences via multiple logistic regression did not change the above association (online supplementary appendix 2). When written, physician TSH orders were not clinically indicated 60.6% (66/109) of the time in 2016 and 20% (2/10) of the time in 2017 (p=0.01). Among patients without a clinical indication, the proportion with a charted order for testing decreased from 73.3% (66/90) in 2016 to 2.7% (2/74) in 2017 (p<0.0001), although 84.4% (76/90) and 29.7% (22/74) were actually tested, respectively. In contrast, 36.8% (25/68) of patients in 2016 with a clinical indication for testing by our criteria did not have a TSH assay requested by a physician, compared with 81.8% (36/44) in 2017 (p<0.0001); the corresponding proportions where testing was not actually performed were 13.2% (9/68) and 52.3% (23/44). As indicated in table 3, the most common missed indication for patients who did not receive testing was new heart failure. Observations in table 3 were collected to assess if the intervention increased diagnostic error (potentially missed patients with thyroid disease). The pattern of clinical indications for TSH testing among patients who were not tested was not influenced by the intervention. Despite the rise in potentially ‘missed’ testing in 2017, the decline in TSH testing occurring without an indication exceeded the increased number of missed opportunities in the absence of the checkbox. The count of non-charted orders among all TSH tests was static at 34/135 in 2016 and 33/43 in 2017, which, with a considerable reduction in the overall number of tests in the denominator, led to an increased proportion of non-charted orders for TSH testing (from 25.2% to 76.7%). Please refer to table 2 for a complete breakdown of order origin, clinical indication and results.
Thyroid-stimulating hormone test indications among non-tested patients with documented clinical reasons (ie, potentially ‘missed’ patients with thyroid disease)*
When we excluded patients with TSH values in keeping with sick euthyroid, 8/178 thyroid studies on admission were likely pathological (4.5%). Four of these patients had known thyroid disease and TSH testing did not change hospital management, follow-up arrangements or medications in hospital. Four patients were diagnosed with new thyroid abnormalities: the first patient (TSH 0.23 m(IU)/L, T4 17 pmol/L, T3 3 pmol/L) had end-stage ischaemic cardiomyopathy with refractory monomorphic ventricular tachycardia and died shortly after admission; the second patient (TSH 7.71, T4 14) also had severe ischaemic cardiomyopathy and was palliated in hospital; the third patient (TSH 0.22, T4 24.6, T3 4.7) was admitted with an ST-elevation myocardial infarction, abnormal thyroid studies were not acknowledged in the chart and repeat EMR-accessible thyroid studies several months later had normalised on their own; the final patient (TSH 0.01, T4 29, T3 6.8) had already been identified as being hyperthyroid based on EMR-accessible thyroid studies ordered 2 weeks prior, and antithyroid medication was initiated by the endocrinology service.
From the informal discussions, some common dynamics emerged. Ward clerks reported that admission blood work is quite standard when a patient is admitted. Furthermore, the chart is not always available when the blood work is being entered; consequently, standard blood work is entered to avoid delays in care. In reference to requesting TSH assays, it was felt that ‘we usually send that; it is on the order set.’ Nurses explained that blood work is occasionally added by nurses if there is a feeling that it was missed. Finally, residents and medical students consistently reported around the time of the intervention that nurses were more likely to highlight the need to order TSH assays for patients at admission.
Discussion
Our results are consistent with previous studies9–11 showing that optional blood work on order sets in the clinical context can drive overuse. The simple removal of a checkbox dramatically changed TSH ordering patterns in our unit. Interestingly, instead of testing TSH inappropriately as often occurred previously, the unit is now typically deferring TSH testing in patients who have an indication for testing. It should be recognised that the decrease in the proportion of missed testing from 50.0% to 33.3% among untested patients in table 2 does not contradict this observation; given the even larger shift in patients with no testing indication towards not being tested after intervention, the subsequent substantial increase in the denominator results in a decline in the proportion of untested patients with testing indication, even with an increased absolute number of missed testing opportunities. In the case of new heart failure—which represented more than half of the indications in our sample when TSH testing was missed—guidelines acknowledge that TSH testing is likely low yield.17 Moreover, hyperthyroid cardiomyopathy is rare in clinical practice.19 Among the patients in our sample who did have TSH testing done for appropriate indications, clinical management was not influenced. As a result, there is likely very little negative clinical impact of the change in ordering in our study. However, had we removed checkboxes associated with some form of guideline-directed treatment, such as antiplatelet therapy in acute coronary syndrome (which has mortality benefit), one wonders if there would be a similar but potentially harmful change in ordering patterns. Given our reliance on order sets, particularly by trainees, changes to order sets should be cautious to assure guideline-directed care is maintained; thorough evaluation of missed testing is recommended to avoid clinically important consequences of underutilisation.
Moreover, the current evidence on successful order set generation prioritises convenience as a means to maximise uptake and use by physicians.20 For example, the clinical support decision 10 commandments and usability principles summarised by McGreevey emphasise ease of use, efficiency and simplicity.20 The importance of monitoring for unintended consequences of order sets is appropriately highlighted with specific attention to minimising workflow issues, paper persistence, negative emotions and user unfriendliness. However, these commandments, and other dedicated articles on unintended consequences,21 22 provide little insight into the potential for excessive diagnostic testing driven by order sets which can increase healthcare costs and theoretically affect patient care. The grouping of multiple commonly ordered tests is convenient for the physician, but our study offers insight into how that approach can be costly. The unintended change in ordering patterns associated with order sets should be routinely considered during development and implementation. In our high-volume critical care unit, ward clerks and nurses are relied on to enter blood work for patients being admitted to the unit. This blood work is typically entered after reviewing physician orders but can also originate from verbal orders or as part of the ‘admission work up’. In this case, ward clerks conceded that they requested blood work because ‘[they] usually send that; it is on the order set.’ Residents also noted they were being prompted more by nurses to order TSH during the intervention period. These observations suggest that the TSH checkbox on the order set creates an assumption that the corresponding test will be requested, possibly even providing a ‘license’ to send blood work without a physician order. In effect, clerks and nurses often entered an order for the test, even when it was not ordered by a physician, as a standard practice. Thus, we erroneously attributed the 25% of TSH assay requests sent to phlebotomy without a physician order to verbal physician orders (see the Methods section). It would have been preferable to have conducted a more comprehensive problem evaluation to address this discrepancy, highlighting the importance of this process. Even so, the run chart in figure 1 suggests a large and sustained decrease in ordering which falls definitively below the median 8 months after the intervention. Although we have not collected the data to support this, it may be that these non-physician-ordered assays have become even less common than during the evaluation period, with the change in ordering culture passively shifting non-physician staff away from the routine practice of requesting them.
We believe the decrease in TSH ordering generated by the revised order set is largely due to its prevention of habitual TSH ordering. Physicians now must critically think about the possibility of a thyroid disorder contributing to the clinical presentation and write in the TSH order. This aligns with a respondent’s comment in the survey work of Ash et al who stated that order sets cause physicians to ‘lose critical thinking abilities. Don’t question order sets.’21 A second dynamic may have been reduced importance. Although included for convenience, presentation of the TSH checkbox in a format identical to that of more crucial investigations may have led individuals, especially trainees, to ascribe unintended importance to the test or to perceive a false expectation to order. Investigators who found that the inclusion of an optional elevation of head-of-bed order increased appropriate positioning of ventilated ICU patients viewed their optional order as a ‘discrete and timely physician reminder about the importance of head-of-bed positioning.’23 It is not surprising then that the removal of a similar potential ‘reminder’, as in our study, would decrease ordering.
These are some limitations that should be considered in interpreting the results of this study. As this intervention was situated in a teaching hospital, its effects may have been related to the relatively early professional stage of residency practice. The high acuity setting of the CCU and our ward clerk/nurse order entry system may similarly limit generalisability of findings to other care environments where order sets are used. Some differences in patient characteristics were also noted in the preintervention and postintervention periods, and although we intentionally compared the same calendar months and evaluated adjusted models, we cannot rule out residual confounding. Similarly, recognising that the same staff may look after several admissions, ordering tendency correlation within individuals was not accounted for. Thus, we cannot comment on whether a single physician, or many, was responsible for the overordering. As patient assessment was chart based, there was no standardised collection of information directly from the patient for the purposes of this study, potentially limiting the quality of data regarding thyroid-related symptoms/history. Additionally, reviewers were not blinded to the intervention status when collecting data from the charts; they also did not individually review all charts independently to allow assessment of inter-rater agreement, although clear criteria for the assessment of appropriateness are expected to minimise this potential bias. We also acknowledge that our above conclusions are limited by the fact that TSH is vulnerable to overordering independent of its presence on the order set.13 24 It also appears to be a test of limited value in the CCU, making it less likely to be given considerable thought prior to being ordered. Thus, removal of checkboxes related to other tests/therapies may have a different effect. Although we followed the clinical course of those who had abnormal TSH results and found no impact on CCU care, we did not follow-up patients in whom testing was missed. Finally, a more formal approach to the qualitative assessment of how staff members perceive the ordering of lab work may have yielded additional insights.
However, it is useful to note that even a TSH checkbox can mould the culture of a unit. Our study imposed a small change to the order set, included virtually all patients and evaluated changes in the frequency of TSH ordering and in the appropriateness and origins of the orders. We conclude that the practice-altering potential of order sets should not be underestimated, in the CCU and conceivably elsewhere, reinforcing that their careful design can promote stewardship of investigations. For order set revision, we advise that careful thought be given to the entire ordering process, the perceptions generated by their alteration and the potential consequences of ‘missed’ testing. With the above in mind, we encourage critical review of other low-value tests and the influence of order sets on their use.
Acknowledgments
We thank Dr Gary Groot for feedback on study design and Dr Christopher Pekrul for sharing his professional insights on order sets and their use.
References
Footnotes
Twitter @LeisBenjamin
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The study was approved by the University of Saskatchewan Biomedical Research Ethics Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All data relevant to the study are included in the article or uploaded as supplementary information.