Article Text
Abstract
Introduction Relevant clinical information is vital to inform the analytical and interpretative phases of most investigations. The aim of this study is to evaluate the impact of implementation of computerised provider order entry (CPOE), featuring order-specific electronic order entry forms (eOEFs), on the quality and quantity of clinical information included with investigation requests.
Methods The CPOE module of a commercially available electronic health record (Cerner Millennium) was implemented at a large, tertiary care centre. The laboratory information management system was interrogated to collect data on specimens sent for microbiological culture 1 year before implementation of CPOE (2018), immediately post implementation (2019) and 6 months post implementation (2020). An interrupted time series analysis was performed, using text mining, to evaluate the quality and quantity of free-text clinical information.
Results In total, 39 919 specimens were collected from 16 458 patients. eOEFs were used to place 10 071 out of 13 735 orders in 2019 (73.3%), and 9155 out of 12 229 orders in 2020 (74.9%). No clinical details were included with 653 out of 39 919 specimens (1.6%), of which 22 (3.4%) were ordered using eOEFs. The median character count increased from 14 in 2018, to 41 in 2019, and 38 in 2020. An anti-infective agent was specified in 581 out of 13 955 requests (4.2%) in 2018; 5545 out of 13 735 requests (40.4%) in 2019; and 5215 out of 12 229 requests (42.6%) in 2020. Ciprofloxacin or piperacillin-tazobactam (Tazocin) were mentioned in the clinical details included with 421 out of 15 335 urine culture requests (2.7%), of which 406 (96.3%) were ordered using eOEFs. Subsequent detection of in vitro non-susceptibility led to a change in anti-infective therapy for five patients.
Conclusions Implementation of CPOE, featuring order-specific eOEFs, significantly and sustainably improves the quality and quantity of clinical information included with investigation requests, resulting in changes to patient management that would not otherwise have occurred.
- Data Accuracy
- Diagnostic Services
- Electronic Health Records
- Information technology
- Laboratory medicine
Data availability statement
Data are available upon reasonable request. Data are not publicly available. Certain data may be available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Relevant clinical information is vital to inform the analytical and interpretative phases of most investigations.
Differences in the design of computerised provider order entry (CPOE) systems, which are used to submit electronic investigation requests, may affect end user perceptions around system accessibility, flexibility and usability.
WHAT THIS STUDY ADDS
Our results show that CPOE featuring order-specific electronic order entry forms significantly and sustainable improves both the quality and quantity of clinical information included with investigation requests, resulting in changes to patient management that would not otherwise have occurred.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
More research is needed to understand how CPOE designs can be optimised to: benefit patient outcomes; leverage the potential of clinical decision support rules in guiding investigation utilisation; and safely expand the role of text mining and machine learning in the analysis, interpretation and repurposing of free-text clinical data.
Introduction
Electronic health record (EHR) systems are being implemented in many high-income and middle-income countries with the aim of improving the quality, safety and efficiency of healthcare.1 2 Commercially available EHRs can be cheaper to procure and maintain than locally developed electronic systems, which is likely to drive their ongoing adoption.2
EHRs support multiple healthcare functions, including creation and storage of patient notes, managing electronic prescriptions and patient-facing web portals.3 Widening access to EHRs is also driving greater adoption of computerised provider order entry (CPOE) systems, which are used by healthcare workers (HCWs) to submit diverse electronic requests, ranging from laboratory investigations,4–14 to imaging studies and radiological examinations,15–22 to health education videos,3 to drugs and medications.1 2 23–25 The proportion of hospitals using CPOE is not known.5 By leveraging their sophisticated search modules, CPOE systems can be set up to request practically all imaginable investigations.13
Clinical details included with orders inform the analytical and interpretative phases of most investigations. The provision of accurate and timely patient information is critical to investigation choice, result interpretation and follow-up.20 26 As a result of CPOE, traditional paper-based systems for submitting investigation requests are being superseded by electronic order entry forms (eOEFs), also known as ‘ask at order entry’ questions. When completing eOEFs, HCWs are presented with order-specific questions, requiring a response on an optional or mandatory basis. Individual eOEFs can be bundled into order sets, which colocate similar and related orderables. This change represents a significant opportunity for quality improvement, particularly given the many practical difficulties intrinsic to managing and maintaining paper-based ordering systems at scale.3
Concerns have been raised about the role of CPOE in the proliferation of investigation requests. Differences in CPOE choice architectures, including order set design, can have a dramatic impact on investigation ordering patterns. HCWs with EHR access order more investigations than their non-EHR counterparts27 and may become over-reliant on order sets, accepting bundled orders without making appropriate modifications.5 14
Aim
As the volume of investigation requests grows, there is increased scrutiny over the appropriateness of testing.10 Repetitive testing without explicit justification is not clinically appropriate.8 9 14 28
Knowledge of the impact of CPOE on different care processes is limited,2 and high-quality evidence of benefit is lacking.25 The ability of CPOE to improve communication of important clinical information has yet to be widely demonstrated.26
The aim of this study is to evaluate the impact of implementation of CPOE, featuring order-specific eOEFs, on the quality and quantity of clinical information included with laboratory investigation requests, concentrating on specimens sent for microbiological culture.
Materials and methods
Population and setting
In autumn 2019, the CPOE module of a commercially available EHR system, Cerner Millennium,29 was implemented at a hospital Trust accounting for 1.72 million patient contacts each year and including more than 1500 beds, offering the widest range of tertiary specialist care in the UK.
CPOE design and implementation
To prepare for implementation of CPOE, order-specific eOEFs were codesigned by a multidisciplinary team of professionals, including Cerner representatives, information technology specialists, clinicians and scientists, with targeted input from end users. During electronic order entry, free-text fields highlighted in yellow required a response, whereas free-text fields in white were optional (figure 1A).
In order to avoid ‘over-alerting’ and its product, ‘alert fatigue’, pop-up boxes were avoided altogether in the design of eOEFs.2 5 8 30–33 Mandatory pull-down menus and pick lists were avoided to improve completion rates of free-text fields.
Investigations could be ordered using eOEFs in both inpatient and outpatient settings. Where appropriate, individual eOEFs were bundled together into order sets. Orderables were only preselected by default where this was approved by a multidisciplinary panel of subject matter experts. Clinical decision support (CDS) rules were introduced to facilitate duplicate checking of electronic orders.
Paper-based request forms continued to be accepted post implementation of CPOE, with all included clinical details being routinely transcribed to the laboratory information management system (LIMS) (figure 1B). To reject requests for microbiological culture solely on the basis of requesting method would almost certainly be considered unreasonable by both patients and HCWs. For example, electronic requesting of laboratory investigations requires a certain amount of infrastructure. Practically speaking, stickers need to be printed and applied to specimens. However, the EHR occasionally requires maintenance and sticker printers can malfunction. Completion of electronic request forms also requires easy access to a computer terminal, which may be considered impractical in certain clinical settings, like in theatre. Finally, electronic requesting is less well supported by Cerner Millennium in specific clinical settings, like the dialysis unit, which falls in the grey area between inpatient care and community care. In summary, although electronic requesting of investigations has certainly become very popular and comes with many advantages, precluding the use of paper request forms altogether has its own downsides.
Prior to implementation of CPOE, HCWs were invited to mandatory educational sessions where they were shown how to place electronic orders. HCWs were also instructed to remove paper-based request forms from clinical areas in order to encourage use of CPOE in the immediate post implementation period.
Data collection
The LIMS was interrogated to retrospectively collect data on specimens collected between 13 August 2018 and 15 October 2018 (1 year before implementation of CPOE); between 3 August 2019 and 15 October 2019 (immediately post implementation of CPOE); and between 10 January 2020 and 15 March 2020 (6 months post implementation of CPOE).
Data analysis
An interrupted time series analysis was performed to evaluate the impact of implementation of CPOE featuring eOEFs on the quality and quantity of clinical information included with requests for culture of blood, urine, wound swabs, cerebrospinal fluid (CSF) and drain fluid. Data analysis and tests for statistical significance were conducted in R,34 with figures produced using the ggplot235 or NHSRplotthedots36 packages. The Anhøj rules were used to analyse data over time for non-random variation.37 To analyse free-text clinical information included with investigation requests, text mining (text parsing and vectorisation) was performed using the tidytext package.38 This package was used to split free-text information into tokens, thereby facilitating further evaluation. For example, non-sensical free-text data were detected by means of comparison with common stop words,38 the 10 000 most common English words39 and words included in the Manual of Clinical Microbiology.40 Any remaining free-text data were manually reviewed to finally classify the data as non-sensical.
As there are no established standards available to assess the quality of clinical details included with investigation requests,18 41 text mining concentrated on the following quality metrics.
First, text mining was used to analyse all data for specific mention of anti-infectives for systemic use, which could be mentioned by brand name, generic name or drug class, as defined by WHO Anatomical Therapeutic Chemical/Defined Daily Dose (ATC/DDD) Index: Group J.42 Fuzzy matching of search terms was facilitated using the stringr package.43 Excluding patients under the care of urology, antimicrobial susceptibility testing (AST) for ‘second-line’ anti-infectives is only performed in limited circumstances, including specific mention of these agents in the clinical details included with investigation requests. Furthermore, to support antimicrobial stewardship and combat antimicrobial resistance, AST results for ‘second-line’ anti-infectives are only released from the LIMS to the EHR in limited circumstances, including specific mention of these agents in the clinical details included with investigation requests. Where specific anti-infectives were mentioned, LIMS and EHR data were retrospectively analysed to evaluate the impact on laboratory processing of relevant specimens and subsequent patient management.
Second, data included with all urine culture requests were analysed for specific mention of symptoms or clinical signs suggestive of cystitis, pyelonephritis or urethritis, as defined by Scottish Intercollegiate Guidelines Network (SIGN) guideline 8844 and the National Health Service.45
Exclusion criteria
This study excluded mortuary specimens and investigations requested by community-based general practitioners and other hospitals, as Cerner Millennium29 is not presently used in these clinical settings.
Patient and public involvement
In the course of this research, patient and public involvement was not actively sought as it was not considered feasible or appropriate.
Results
Patient demographics
Over the course of the study, 39 919 specimens were collected from 16 458 distinct patients, of which 8557 (52%) were female. The median age of patients was 55 years (IQR 27–73 years). No information was available on patients’ main comorbidities, ethnic grouping or socioeconomic status.
Specimen ordering method
Of the 39 919 distinct specimens, 13 955 (35%) were collected in 2018, 13 735 (34.4%) in 2019 and 12 229 (30.6%) in 2020. CPOE featuring order-specific eOEFs was implemented in 2019. In total, 20 693 out of 39 919 (51.8%) orders were placed using order-specific eOEFs. As mentioned, paper-based request forms continued to be accepted post implementation of CPOE. Immediately post implementation of CPOE in 2019, there were 3664 out of 13 735 specimens (26.7%) received with a paper-based request form, which remained stable 6 months later in 2020, when 3074 out of 12 229 specimens (25.1%) were received with a paper-based request form.
Specimen collection location
In total, specimens were collected in 107 locations. Of the 39 919 specimens, inpatient areas accounted for 35 899 (89.9%) and the top 10 locations accounted for 14 438 (36.2%) . The emergency department made the highest number of requests, accounting for 3575 out of 39 919 (8.96%) specimens.
Specimen type information
Of the five different specimen types included in this study, blood cultures were the most common, accounting for 15 600 out of 39 919 specimens (39.1%), followed by urine specimens (15 335, 38.4%), wound swabs (7835, 19.6%), CSF (834, 2.1%) and drain fluid specimens (315, 0.8%).
Quantity of information included with specimen request forms
Implementation of CPOE had a significant impact on the total character count per request, and particularly the proportion of specimens received with few clinical details (defined as 1–9 characters). In 2018, 1 year before implementation of CPOE, 4384 out of 13 955 specimens (32.1%) included between 1 and 9 characters, compared with 272 out of 13 530 specimens (2%) in 2019, immediately post implementation, and 183 out of 12 084 specimens (1.5%) in 2020, 6 months post implementation (figure 2)
In total, 653 out of 39 919 specimens (1.6%) were received with no clinical details. Of these, 303 (46.4%) were requests submitted prior to implementation of CPOE, 328 (50.2%) were paper-based request forms submitted post implementation of CPOE and 22 (3.4%) were order-specific eOEFs.
For the remaining 39 266 non-blank request forms, the median character count of clinical details included with specimens was 31 (IQR 31). One year before implementation of CPOE, in 2018, the median character count was 14 (max 148, IQR 18). This increased significantly to 41 (max 171, IQR 31, p<0.05) in 2019, immediately post implementation, and was maintained at 38 (max 196, IQR 26, p<0.05) in 2020, 6 months post implementation. The mean character count of clinical details, stratified by study year, is shown in figure 3.
One year before implementation of CPOE, in 2018, the mean character count of clinical details included with request forms was 19.5 (95% CI 19.3 to 19.8). The mean character count increased significantly post implementation of CPOE, in 2019 and 2020, to 46.9 (95% CI 46.6 to 47.2, p<0.05). When paper-based request forms were used, post implementation of CPOE, in 2019 and 2020, the mean character count increased to 34.4 (95% CI 33.9 to 35.0, p<0.05). This may be explained by HCWs reflecting on the questions included with order-specific eOEFs, and developing a better understanding of the specimen type-specific clinical details that are of interest to medical laboratories (figure 4).
Quality of information included with specimen request forms
Non-sensical clinical details were included with 39 out of 39 266 (0.1%) non-blank investigation requests, of which 32 (82.1%) were pre-CPOE request forms; 4 (10.3%) were post-CPOE paper-based request forms; and 3 (7.6%) were post-CPOE eOEFs.
Free-text clinical details included with specimen requests were further analysed for specific quality metrics. First, at least one antimicrobial agent was mentioned in the clinical details included with 11 341 out of 39 919 specimens (28.4%). When stratified by study year, 581 out of 13 955 specimens (4.2%) collected in 2018 specifically mentioned an anti-infective. Post implementation of CPOE, request forms were significantly more likely to specifically mention an anti-infective, with 5545 out of 13 735 specimens (40.4%) collected in 2019, and 5215 out of 12 229 specimens (42.6%) collected in 2020 mentioning this clinical information (p<0.05).
Second, of the 15 335 specimens collected for urine culture, 7149 (46.6%) were accompanied by a request form specifically mentioning symptoms or clinical signs suggestive of cystitis, pyelonephritis or urethritis. When stratified by study year, 1664 out of 5414 specimens (30.7%) collected in 2018 were accompanied by a request form mentioning symptoms or signs of urinary tract infection. This increased significantly in 2019 and 2020, with requests mentioning symptoms or signs of urinary tract infection in 2975 out of 5436 specimens (54.7%) and 2510 out of 4485 specimens (56.0%), respectively (p<0.05). The impact of CPOE was similarly pronounced when results were stratified by specimen ordering method (figure 5).
Finally, clinical details included with urine culture requests were analysed for specific mention of so-called ‘second-line’ anti-infectives, including ciprofloxacin and piperacillin-tazobactam (Tazocin, Pfizer). Implementation of CPOE significantly increased the frequency with which HCWs mentioned current and/or intended use of specific ‘second-line’ anti-infective agents.
Ciprofloxacin was mentioned in 93 out of 15 335 (0.6%) urine culture requests, of which 5 (5.4%) were paper-based requests and 88 (94.6%) were post-CPOE eOEFs. Culture led to isolation of a urinary tract pathogen from 14 out of 82 (17.1%) non-urology urine specimens. AST was appropriately performed in 13 out of 14 (92.95%) cases. In vitro non-susceptibility to ciprofloxacin led to a change in patient management in 2 out of 13 (15%) cases.
Tazocin was mentioned in the clinical details included with 328 out of 15 335 (2.1%) urine culture requests, of which 10 (3%) were paper-based requests and 318 (97%) were post-CPOE eOEFs. Culture led to isolation of a urinary tract pathogen from 58 out of 305 (81%) non-urology urine specimens. AST was appropriately performed in 57 out of 58 (98.3%) cases. In vitro non-susceptibility to Tazocin led to a change in patient management in 3 out of 57 (5.3%) cases.
Discussion
The results show that implementation of CPOE had a significant and sustained impact on both the quality and quantity of clinical information included with investigation requests. Furthermore, the additional clinical information gathered by means of eOEFs had a positive impact on both the analytical and interpretative phases of specimen processing, directly resulting in changes to patient management that would not otherwise have occurred.
Related research
Concerns have been raised in the literature around the potential for misuse of free-text data entry fields in EHRs.31 For example, Dekarske et al evaluated the appropriateness of 22 HCWs’ over-ride reasons for 1829 electronic alerts. They reported that 13.9% of alert over-ride reasons included non-sensical free-text responses (eg, ‘asdf’ or ‘—’). Of note, 83% of the non-sensical responses were entered by 2 of the 22 subjects.31 However, this study shows that implementation of CPOE significantly improved the richness of clinical information included with investigation requests. Investigations ordered using eOEFs were significantly less likely to be blank or include non-sensical information. This reassuring finding adds to the existing evidence that, compared with pull-down menus and pick lists, free-text data entry fields can facilitate more detailed communication and help mitigate the risk of selection errors.2 7 31 Indeed, CPOE systems designed with mandatory pull-down menus and pick lists have been shown to reduce completion rates of free-text fields.6 Grisson et al maintain that virtually all complex order entry systems must tolerate some level of free-text data entry to be clinically efficient.13 Given the large scale of this study, it is unlikely that the results were disproportionately affected by contributions from only a few individuals.
Delivering high-quality, safe and reliable patient care involves influencing the decisions that HCWs make. Traditional interventions, including new evidence, guidelines and education, can be relatively ineffective at influencing HCW behaviour.8 25 41 Timelines for realising the benefits of new technologies are also often greatly underestimated.2 This study demonstrates that implementation of CPOE, featuring order-specific eOEFs, helps nudge the bedside clinical practice of HCWs, leading to significant improvements in both the quality and quantity of clinical information included with investigation requests.
Few published studies have evaluated the real-world impact of implementation of CPOE.7 Understandably, many of the earlier studies assessed the impact of locally developed systems, rather than modern, commercially available EHRs with CPOE functionality. Electronic ordering can save staff time in trying to decipher handwriting; facilitate faster transfer of information; reduce duplication; and improve accessibility of information.2 There is also evidence that requests submitted by CPOE are significantly more likely than paper-based requests to contain useful information, including clinical questions and information on prior diagnoses.4 15 17 26 41 Furthermore, computerised ordering systems have been shown to reduce healthcare expenditure; decrease the number of tests per admission46; and improve test turnaround time.26
Limitations of this study
As with any observational before-and-after design, the results may have been influenced by other unmeasured factors.25 Due to differences in clinical practice and EHR configuration, this study’s findings may not be generalisable to other hospitals. Furthermore, this study focused on laboratory investigation requests, concentrating on specimens sent for microbiological culture. However, CPOE systems are also used to submit electronic requests for other types of orderables, including imaging studies, drugs and medications. This study’s findings may not be generalisable to these other types of orderables.
Depending on CPOE design, use of eOEFs may require more time than completion of paper-based request forms, which was not evaluated in this study. However, other time savings inherent to EHRs may offset the time spent on CPOE, including less time spent looking for paper-based forms and charts.5
Other studies have reported on the impact of CPOE on the number, cost and appropriateness of investigations requested,2 46 as well as the costs of staffing for EHR and CPOE development, implementation, maintenance and troubleshooting.5 However, an evaluation of these variables was not within the scope of the present study.
This study did not evaluate the effect of different CPOE designs on HCWs’ clinical and laboratory practice. The implementation of new technologies risks the inadvertent introduction of new, often unanticipated clinical hazards. For example, this study did not evaluate the role of CPOE in facilitating the ordering of investigations on the wrong patient, or on electronic medical records propagating copy and paste errors.47
Nor did this study evaluate the effect of different CPOE designs on perceptions around system accessibility, flexibility and usability.1 2 Lack of system flexibility can result from systems attempting to improve safety by making certain tasks compulsory or sequential.2 Mandatory documentation can lead to unintended behaviours, including proliferation of local adaptations and workarounds to usual working practices.2 31 Compulsory tasks and documentation, which are perceived as clinically irrelevant or unnecessarily repetitive, can cause frustration among end users.31 Concerns have also been raised about the impact of CPOE on HCWs’ autonomy in decision-making.5
Further research opportunities
More research is needed to understand how EHR design impacts on patient outcomes,20 and how CPOE choice architectures can be further optimised.2 For example, defaults are one of the most widely employed tools in CPOE design, bearing considerable influence on decision-making.48 When all laboratory tests included in an order set are preselected, HCWs order significantly more investigations, thereby increasing the cost of admission.49 Other CPOE functions that are known to impact on ordering practices include order sets; electronic alerts and warnings; and publicising laboratory test charges.28 31 46
Further research is also required to explore the wider potential of more advanced CDS rules in guiding investigation utilisation.30 For example, alerts and warnings could be triggered on the basis of clinical information included with eOEFs,31 perhaps assisting in the detection of incorrectly placed investigation requests.50 Some CPOE systems already have the ability to integrate orders with pharmacy and laboratory data, to make helpful recommendations and alert requestors of significant issues.8
Beyond CDS, additional research is required to safely expand the role of text mining and machine learning (ML) in the analysis, interpretation and repurposing of free-text clinical data. For example, specific eOEF fields could be automatically prepopulated with clinical information documented elsewhere in the EHR.20 Insights gained through ML algorithms could also be used to optimise healthcare workflow and reduce costs.18 Although not currently in routine use, association-based recommender systems have been described, which automatically analyse inpatient data to inform CDS rules and electronic order set contents.51
Conclusions
This study shows that implementation of CPOE, featuring order-specific eOEFs, significantly and sustainably improves both the quality and quantity of clinical information included with investigation requests. This additional information directly impacts on healthcare workflow, resulting in changes to patient management that would not otherwise have occurred.
Data availability statement
Data are available upon reasonable request. Data are not publicly available. Certain data may be available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
The privacy rights of all subjects have been observed throughout. This report is a summary of a quality and service improvement (QSI) that used anonymous patient data, which are routinely collected at Newcastle upon Tyne Hospitals NHS Foundation Trust, and no further ethical approval or patient consent was required.
Acknowledgments
The authors wish to acknowledge the contributions of the following colleagues employed by the Newcastle upon Tyne Hospitals NHS Foundation Trust: Mr John Clinton, Ms Deborah Osborne and Ms Sarah Evans. The authors have not had any writing assistance.
References
Footnotes
Twitter @send2dan
Contributors DW: conceptualisation, methodology, software, formal analysis, investigation, resources, data curation, writing—original draft, writing—review and editing, visualisation, project administration, guarantor. CC: writing—review and editing. RO: methodology, investigation, resources, writing—review and editing. CJP: resources, writing—review and editing, supervision. All authors read and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Author note DW: clinical informatics lead for the Integrated Laboratory Medicine (ILM) Directorate at Newcastle upon Tyne Hospitals NHS Foundation Trust. CJP: chief clinical information officer (CCIO) at Newcastle upon Tyne Hospitals NHS Foundation Trust.