You are here

  • Home
  • Archive
  • Volume 11, Issue 1
  • Understanding challenges of using routinely collected health data to address clinical care gaps: a case study in Alberta, Canada

Article Text

Article menu
Research & reporting methodology
Understanding challenges of using routinely collected health data to address clinical care gaps: a case study in Alberta, Canada
  1. Taylor McGuckin1,
  2. Katelynn Crick1,
  3. Tyler W Myroniuk2,
  4. Brock Setchell1,
  5. Roseanne O Yeung1,3,
  6. Denise Campbell-Scherer1,4
  1. 1Faculty of Medicine & Dentistry - Lifelong Learning & Physician Learning Program, University of Alberta, Edmonton, Alberta, Canada
  2. 2Public Health, University of Missouri, Columbia, Missouri, USA
  3. 3Division of Endocrinology & Metabolism, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
  4. 4Department of Family Medicine, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
  1. Correspondence to Dr Denise Campbell-Scherer; denise.campbell-scherer{at}ualberta.ca

Abstract

High-quality data are fundamental to healthcare research, future applications of artificial intelligence and advancing healthcare delivery and outcomes through a learning health system. Although routinely collected administrative health and electronic medical record data are rich sources of information, they have significant limitations. Through four example projects from the Physician Learning Program in Edmonton, Alberta, Canada, we illustrate barriers to using routinely collected health data to conduct research and engage in clinical quality improvement. These include challenges with data availability for variables of clinical interest, data completeness within a clinical visit, missing and duplicate visits, and variability of data capture systems. We make four recommendations that highlight the need for increased clinical engagement to improve the collection and coding of routinely collected data. Advancing the quality and usability of health systems data will support the continuous quality improvement needed to achieve the quintuple aim.

  • quality improvement
  • quality improvement methodologies
  • data accuracy
  • health services research
  • healthcare quality improvement
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjoq-2021-001491

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    A learning health system is foundational to achieving the quintuple aim of advancing patient care, population health, equity, cost-effectiveness, healthcare worker experience, and, ultimately, future goals such as precision health.1–3 To be able to rapidly answer important clinical questions, the structure of, and data capture in, electronic medical records and health administrative databases needs to be improved. Alberta, Canada is a globally recognised jurisdiction for its health data infrastructure and capture. However, health service researchers have identified important limitations to its use.4–8 Reasons for these limitations include the historic use of different health information systems across Alberta’s regions,9 and the creation of administrative health databases for non-clinical functions such as payment.10

    The Physician Learning Program (PLP)11 is a provincial programme that works to understand gaps in clinical practice, create clinically actionable information and cocreate sustainable solutions with physicians, allied health teams, patients and community, and health system partners to advance practice. Here, we share four examples of PLP projects on a range of rare to common medical conditions that highlight some of the current challenges of using routinely collected health data to inform real-world clinical problems and support quality improvement. These four projects demonstrate areas where we encountered limitations in data capture, which if rectified, would provide needed information to help advance care of Albertans. We offer guidance in improving routinely collected health data that is broadly relevant to health systems by addressing issues of data completeness, availability, missingness and duplication, and variability in capture. Improvements in these areas are necessary to increase the usability of data for healthcare, health services research, and, eventually, future applications of artificial intelligence and precision health.

    Methods

    The primary objective of this work was to capture, categorise, and label overarching and recurring problematic data patterns in electronic health records and administrative databases observed through work conducted at PLP. The four projects presented were conducted to understand gaps in clinical care and develop baseline data for quality improvement initiatives. Each project is described in table 1, with notes on data sources in table 2. For each project, a series of questions were co-created with clinicians to provide information of importance for clinical quality improvement. We identified whether secondary data from electronic medical records and administrative databases was available or whether primary data collection was necessary. Routinely collected health data from electronic medical records and other administrative databases was feasible and extracted for three projects: (1) Adult Diabetes; (2) Paediatric Diabetic Ketoacidosis, a serious complication of diabetes and (3) Adrenal Insufficiency, a rare, life-threatening hormonal disorder. For the Beta-Lactam Allergy and Surgical Prophylaxis project, the required clinical information was not routinely collected into an administrative database. Thus, primary data collection was required and included manually extracting information from paper charts.

    View this table:
    Table 1

    Description of the Physician Learning Program projects including purpose, representative questions, whether a challenge was encountered, and databases used

    View this table:
    Table 2

    Descriptions of the data sources used to complete the projects

    Figure 1 represents the iterative process used to identify, collect, clean and synthesise routinely collected health information needed for clinical quality improvement. Detailed methods and results of the four projects will be published elsewhere. The data collected and analysed for this paper is not the quantitative data of the four projects, but our observations while conducting them. Briefly, for the projects that used routinely collected health data, we formulated a data query to find and pull the raw data needed to answer each project question. A trained analyst employed by Alberta Health Services extracted the data. Once extracted, the raw data were cleaned and analysed using standard statistical software (Oracle SQL Developer, Python V.3.4, SAS V.9.4 and RStudio V.1.2.5033). The clinicians working on the project reviewed the results to assess the validity and completeness of the data in comparison with their knowledge of clinical workflow and processes. The results were compiled into various formats, including presentations, reports, infographics, and clinical tools, and then disseminated to relevant stakeholder groups. Their purpose ultimately is to inform clinical quality improvement and co-creation of interventions to address clinical gaps in care.

    Figure 1
    Figure 1

    The Physician Learning Program’s non-linear process of quality improvement using routinely collected health data. The key elements are: (1) cocreating clinical questions and identifying whether secondary data are available or if primary data collection is necessary; (2) gathering data from databases or completing primary data collection; (3) deep cleaning of the data; (4) conducting analyses and further data cleaning; and (5) effectively communicating findings that serve as the basis for quality improvement.

    Systematic approach used to capture and categorise main challenges and identify root causes

    Over a 2-year period, recurring difficulties arose when obtaining and analysing the administrative data needed to answer clinical questions for the four projects. We undertook a systematic approach to identify and capture whenever problems arose and then categorise them into main challenges. This systematic approach included: (1) capturing whenever a data problem occurred in a project; (2) discussing the problem within our interdisciplinary team of researchers and clinical experts; (3) discussing recurring issues and patterns through team meetings and with key informant discussions; and (4) synthesising them into main categories that spanned projects, healthcare settings, and health conditions. We identified and verified the root cause whenever possible by: (1) talking to clinical, administrative, and analytical staff within Alberta Health Services and Alberta Health (two regulatory government bodies that oversee the delivery of healthcare within the province of Alberta); (2) reading publicly available database documentation12–14 and (3) talking to front-line healthcare staff with deep knowledge of the healthcare setting and clinical systems. Our systematic approach is summarised in box 1.

    Box 1

    Methods used to identify, collect and analyse the raw data (ie, problems arising in using administrative data to answer the clinical questions)

    Methods to identify the raw data

    • Observe when there was a problem while conducting each of the steps in figure 1.

    • Verify if there was a challenge by checking against known published problems and discussing with data analysts and clinicians to see if it matches reality.

    Methods to collect the raw data

    • Formally document the problem encountered and how it was verified .

    Methods used to analyse the raw data

    • Discuss the problems from each project and collate and summarise them into overarching themes (main challenges).

    Patient and public involvement

    At the PLP, we have the mission to create “actionable clinical information and engage with physicians, teams and partners to cocreate sustainable solutions to advance practice.”11 Inherent in this process is the involvement of broader networks outside of the project team including community physicians, physician networks, policy-makers, patients, researchers, and other healthcare professionals. Involvement of stakeholders starts at project conception with physicians and clinical teams cocreating project ideas with the PLP based on health system gaps. Engagement continues through to the dissemination of project outcomes where we integrate with networks to engage in knowledge translation activities, codesign sustainable solutions, and implement them with health system partners.

    Results

    Through our systematic approach of capturing and categorising recurring problems, as outlined in detail above, we identified four broad challenges of using routinely collected health data to address real-world clinical questions. We present them here framed in four example projects. These four challenges and example project questions are summarised in table 3.

    View this table:
    Table 3

    Data challenges encountered while answering clinical questions

    Description of challenges

    Challenge 1: are the data field(s) needed to answer the clinical question available in administrative databases?

    Not all information collected at a patient encounter has a corresponding data field in an administrative database; some information, although available, is not abstracted from the patient chart into a database. In the beta-lactam allergy and surgical prophylaxis project, 0 out of 3218 audited surgical cases contained allergy information in an available administrative database because there was no routinely populated data field for this information. However, for all cases, we found that allergies were recorded in paper charts. Importantly, inappropriate antibiotic prophylaxis, due to allergy status, is associated with a 50% increase odds in surgical site infections and increased costs to the system.15 Assessing care using paper chart audits is sometimes justified but is not sustainable or scalable on a large basis because of its resource intensiveness. We are now working with health system delivery stakeholders to develop more sustainable solutions for this problem of antibiotic allergy and prophylaxis information not being electronically captured and available, specifically.

    For the paediatric diabetic ketoacidosis project, only 28.6% of children’s admissions across Alberta contained data on medication, electrolyte, and fluid administration. Guideline concordance of care for this life-threatening condition cannot be assessed without this information. This information was only available for patients whose encounter was at a site that used Sunrise Clinical Manager, a specific Clinical Information System. Only five Alberta Hospitals and Health Centres, out of over 100 included in our project, used this system inhibiting the feasibility of assessing guideline concordant care across the whole system.

    When assessing patient comorbidities in the adult diabetes project, we could not determine whether patients had a history of hyperosmolar hyperglycaemic state. Despite the International Classification of Diseases-9 (ICD-9) having a corresponding code for this condition, Alberta Health’s coding taxonomy, which is used to capture visit information to pay providers across the province, does not include all ICD-9 codes.16 Thus, this comorbidity could not be assessed for any of the patients.

    Challenge 2: if the data field needed to answer the clinical question is available, is the information complete and accurate?

    The completeness of extracted data was problematic in two of our projects. When assessing lab results in the paediatric diabetic ketoacidosis project, we found that 46.6%, 94.5% and 12.6% of admissions at one of the children’s hospitals in the province had no results for blood pH, blood bicarbonate, and blood glucose, respectively. These laboratory results are central to guiding diabetes care and confirming a diagnosis of diabetic ketoacidosis. Through our root cause analysis, which included consulting with experts in the hospital laboratory, we uncovered that laboratory tests completed from capillary blood sources may not flow from bedside instruments to administrative databases; a historical legacy of funding restrictions when the system was developed. Additionally, we observed incomplete medication, fluid, and electrolyte administration data, which are all necessary for assessing quality of care in relation to established guidelines.

    In the adult diabetes project, routinely collected health data were often missing for measures such as blood pressure, an important clinical assessment for predicting disease complications. In one clinic, 65.5% of visits did not have a blood pressure measurement recorded in a database. Through consultation, we determined that while front line staff are entering these measures into the electronic medical record, it does not flow into administrative databases.

    Challenge 3: can the number of visits for a particular medical condition be accurately measured using administrative data?

    We were unable to accurately estimate the number of outpatient visits for the treatment of adrenal insufficiency due to visits missing from the databases. Missing visits are a consequence of both imprecise codes used at the time of data submission (eg, visits coded as ‘follow-up’) and variation in data submission requirements in which not all visits are required to be submitted and thus captured. Variation in data submission requirements are a result of various payment structures (eg, alternative payments plans) across and within regions of the province. Thus, it is uncertain of how to compare regional data.

    Furthermore, we encountered difficulty reconciling duplicate entries within and between databases housing different aspects of clinical visits. In this example, both Physician Claims and the National Ambulatory Care Reporting System (NACRS) database are used to capture outpatient visit data. They capture much of the same information but use different taxonomies to capture diagnostic information: one uses ICD-9 where the other uses ICD-10. Some visits are captured only in Physician Claims or NACRS, some in neither, and some in both.12–14 17 There is no official reconciliation for visits captured in both. We found that at least 27% of adrenal insufficiency visits were likely duplicates. Of the 211 207 visits analysed, only 5.7% had a diagnostic code for adrenal insufficiency; clinical colleagues insisted this was implausibly low. This raised concerns that an indeterminate number of visits were missing from both databases, which may be due to visits for more than one medical condition not capturing all of the relevant diagnoses. In 78% of visits, there was only one code provided for the visit. Most codes used for the analysed visits were vague such as ‘general examination’ and ‘follow-up’, making it difficult to identify visits related to the treatment of adrenal insufficiency, which likely contributed to this discrepancy.

    Challenge 4: can laboratory tests across the province be identified, harmonised, and analysed?

    Three laboratory information systems are used across Alberta–a historical legacy of healthcare regionalisation. Laboratory codes are not harmonised across any of the laboratory databases within the province of Alberta. Each laboratory information system uses different laboratory codes, and thus, identifying and matching relevant codes across databases is not a trivial task. For example, haemoglobin A1c, a diabetes test, was found to be coded as HbA1c, ZHBA1C and HBA1X depending on where the lab test was completed. One major consequence was that 919 laboratory codes had to be reviewed to identify and harmonise the codes used in the paediatric diabetic ketoacidosis project. This was also problematic for the adult diabetes project (online supplemental table 1).

    Supplemental material

    Strengths and limitations of the databases used

    Through completing these four projects, we identified both strengths of limitations of the administrative databases for informing clinical quality improvement projects. Strengths and limitations in relation to our example projects and questions specifically, and cautions for their use are summarised in table 4. This is not a comprehensive overview of the strengths and limitations of these databases, but rather a summation of our experiences.

    View this table:
    Table 4

    Strengths and limitations of the databases as elucidated by our example projects

    Discussion

    Rapid access to clinically important information is crucial to building a powerful learning health system3 in pursuit of the quintuple aim. Health data infrastructure that supports rapid access to clinically important information for evidence-informed care and clinical quality improvement is key to supporting practice reflection and innovations to meet patient needs. Our PLP projects illuminate four challenges of using routinely collected health data to achieve these aims. First, we found that not all information collected in a patient encounter has a corresponding data field in an administrative database; costly, time-consuming primary data collection is then needed to assess important clinical questions prohibiting the feasibility of continual monitoring. Second, when data fields are available, they may be absent or not uniformly populated. For instance, we observed this problem when clinical evaluations or readings from bedside instruments are used and the information does not flow to administrative databases. Third, establishing prevalence of medical conditions and number of visits was difficult due to missing records, complexity reconciling various databases that contain the same information, inconsistent diagnostic coding practices, and differing taxonomies used between databases. A key element of this challenge was that imprecise diagnostic codes, such as ‘follow-up’, did not permit clarity as to the topics addressed in the visit. The fourth challenge was the multiplicity of laboratory diagnostic codes used for the same test which made it difficult to develop data queries that capture all relevant tests.

    The mission of the PLP is to create actionable clinical information and engage with physicians, teams, patients, and partners to cocreate sustainable solutions to advance practice. The creation of clinically actionable information from routinely available health data is hindered when there are substantial gaps in the information, as measuring improvement requires relevant baseline data and measurement over time to assess change. The strengths and limitations of administrative and electronic medical record health databases have been described extensively, for instance in the work of Burles et al, Clement et al and Edmondson and Reimer.18–20 The inability to analyse data in real time is not a problem unique to the Canadian context, with challenges being documented in other jurisdictions including the USA.21 The overarching issues relating to data capture, completeness, accuracy, and harmonisation, exist across healthcare systems and settings and challenges with data capture in clinical electronic medical records have been well documented.22–27 Several of the databases outlined are available across Canada, including Discharge Abstract Database and NACRS, and thus these challenges are likely to exist across the country. Ongoing work is being conducted by the PLP and with relevant stakeholder groups to address the issues presented. We acknowledge the importance of collaborating with various stakeholders including data scientists, clinicians, and administrators to fully understand what the meaningful clinical data are and how to mobilise and act on them so that data-driven quality improvement is supported. Increased coordination and leveraging the opportunity of a new provincial acute care electronic medical record should continue to advance this work, particularly as efforts evolve across the care continuum.

    Future directions

    Advancing the quality of health systems data is crucial not only for current quality improvement projects, but also in realising the utility of precision health and artificial intelligence to advance healthcare in the future.28–33 Health system data are necessary to meet the Federation of Medical Regulatory Authorities in Canada’s goal that all Canadian physicians participate in data-driven practice quality improvement.33 The overarching purpose of these efforts is to support the development of a learning health system and to achieve improvements in the quintuple aim of improving population health, patients’ experience of care, equity, cost-effectiveness, and sustainability of healthcare workforce.1–3 We strongly believe that the long-term benefits of improved data capture would significantly offset upfront investments. Importantly, supporting these efforts requires mobilising clinical information in a way that does not overwhelm the clinical workforce and contribute to physician burnout.34

    Addressing these four identified challenges is fundamental to creating a learning health system and to advancing healthcare delivery and health outcomes. We recommend the following:

    1. To have more clinically important data available in readily extractable formats, we suggest expanding and harmonising mandatory data submission requirements with increased clinician engagement to ensure data that is captured is clinically meaningful.

    2. To increase the quality and validity of the data available to assess patient care, we suggest the use of more specific codes and consistent taxonomies across the healthcare system to capture encounter diagnoses; standardisation of data entry processes with clear mechanisms of training and maintenance; and, ensuring the flow of clinically important information from bedside instruments, laboratory settings, and diagnostic imaging results to administrative databases in analysable formats.

    3. To enhance efficiency and speed of data capture so that upgrading data quality, quantity, and structure is not at the cost of the clinical user, we suggest the incorporation of technologies like natural language processing, cross-platform interoperability, and application of human-centred design for workflow process improvement.

    4. To promote real-time usability of data, we propose integrating technologies such as natural language processing and artificial intelligence to automate routinised functions to support appropriate real-time clinical decisions and reduce clinician burden.

    Limitations

    The challenges we identified in our routinely collected health data are specific to Alberta, Canada, however, they are commonly encountered in conducting QI and research work using administrative data and are generalisable internationally.22–27 As information technology advances, integration into different health systems is variable leading to different local challenges in deriving solutions. We submit that the principles stated here may be of interest for consideration but additional factors will exist in different jurisdictions.

    Conclusion

    Through practical, real-world projects, we have identified four challenges in using administrative health and electronic medical record data to address clinical care gaps. Improving data infrastructure and quality will enable more nimble quality improvement efforts and real-world evidence studies. Improving this infrastructure, and the reliability and validity of data, is a necessary precondition for emergent technologies in precision health and artificial intelligence, and to developing a learning health system.

    Ethics statements

    Patient consent for publication

    Ethics approval

    This study was approved by Each project included the appropriate ethics approval from the Research Ethics Board-Health Panel at the University of Alberta, Edmonton, Alberta, Canada. The ethics approval numbers are as follows:Pediatric Diabetic Ketoacidosis-Pro00091652Diabetes Management-Pro00085385Beta-lactam Allergy and Surgical Prophylaxis-Pro00089593Adrenal Insufficienc-Pro00088478. Three of our projects included secondary retrospective analyses of routinely collected health data. The Physician Learning Programme only works with unidentified data. The third project was a paper chart audit and also was deidentified.

    References

    1. ↵
      1. Bodenheimer T,
      2. Sinsky C
      . From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014;12:573–6.doi:10.1370/afm.1713pmid:http://www.ncbi.nlm.nih.gov/pubmed/25384822
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. R. Privitera M
      . Addressing Human Factors in Burnout and the Delivery of Healthcare: Quality & Safety Imperative of the Quadruple Aim. Health 2018;10:629–44.doi:10.4236/health.2018.105049
      OpenUrl
    3. ↵
      1. Friedman C,
      2. Rubin J,
      3. Brown J, et al
      . Toward a science of learning systems: a research agenda for the high-functioning learning health system. J Am Med Inform Assoc 2015;22:43–50.doi:10.1136/amiajnl-2014-002977pmid:http://www.ncbi.nlm.nih.gov/pubmed/25342177
      OpenUrlCrossRefPubMed
    4. ↵
      1. Doyle CM,
      2. Lix LM,
      3. Hemmelgarn BR
      . Data variability across Canadian administrative health databases: differences in content, coding, and completeness. Pharmacoepidemiol Drug Saf 2020;29 Suppl 1.doi:10.1002/pds.4889pmid:http://www.ncbi.nlm.nih.gov/pubmed/31507029
    5. ↵
      1. Drews SJ,
      2. Simmonds K,
      3. Usman HR, et al
      . Characterization of enterovirus activity, including that of enterovirus D68, in pediatric patients in Alberta, Canada, in 2014. J Clin Microbiol 2015;53:1042–5.doi:10.1128/JCM.02982-14pmid:http://www.ncbi.nlm.nih.gov/pubmed/25588657
      OpenUrlFREE Full Text
    6. ↵
      1. Ho C,
      2. Guilcher SJT,
      3. McKenzie N, et al
      . Validation of algorithm to identify persons with non-traumatic spinal cord dysfunction in Canada using administrative health data. Top Spinal Cord Inj Rehabil 2017;23:333–42.doi:10.1310/sci2304-333pmid:http://www.ncbi.nlm.nih.gov/pubmed/29339909
      OpenUrlPubMed
    7. ↵
      1. Jolley RJ,
      2. Quan H,
      3. Jetté N, et al
      . Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open 2015;5:e009487. doi:10.1136/bmjopen-2015-009487pmid:http://www.ncbi.nlm.nih.gov/pubmed/26700284
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Peng M,
      2. Lee S,
      3. D'Souza AG, et al
      . Development and validation of data quality rules in administrative health data using association rule mining. BMC Med Inform Decis Mak 2020;20:75. doi:10.1186/s12911-020-1089-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/32334599
      OpenUrlPubMed
    9. ↵
      1. Donaldson C,
      2. Fire DC
      . Fire, Aim… ready? Alberta's big bang approach to healthcare disintegration. Healthc Policy 2010;6:22–31.pmid:http://www.ncbi.nlm.nih.gov/pubmed/21804836
      OpenUrlPubMed
    10. ↵
      1. Lucyk K,
      2. Lu M,
      3. Sajobi T, et al
      . Administrative health data in Canada: lessons from history. BMC Med Inform Decis Mak 2015;15:69. doi:10.1186/s12911-015-0196-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/26286712
      OpenUrlCrossRefPubMed
    11. ↵
      Physician learning program, 2021. Available: https://www.albertaplp.ca/ [Accessed 13 Jul 2021].
    12. ↵
    13. ↵
      1. Alberta Health Services
      . Analytics (DIMR) Alberta health services data Repository for reporting, 2016. Available: https://www.ualberta.ca/medicine/media-library/research/faculty/clin-res/spor-available-datasets.pdf
    14. ↵
      1. Canadian Institute of Health Information
      . Data quality documentation national ambulatory care reporting system, 2020. Available: https://www.cihi.ca/sites/default/files/document/nacrs-data-quality-current-year-information-2019-2020-en.pdf
    15. ↵
      1. Blumenthal KG,
      2. Ryan EE,
      3. Li Y, et al
      . The impact of a reported penicillin allergy on surgical site infection risk. Clin Infect Dis 2018;66:329–36.doi:10.1093/cid/cix794pmid:http://www.ncbi.nlm.nih.gov/pubmed/29361015
      OpenUrlPubMed
    16. ↵
    17. ↵
      1. Cunningham CT,
      2. Cai P,
      3. Topps D, et al
      . Mining rich health data from Canadian physician claims: features and face validity. BMC Res Notes 2014;7:682. doi:10.1186/1756-0500-7-682pmid:http://www.ncbi.nlm.nih.gov/pubmed/25270407
      OpenUrlCrossRefPubMed
    18. ↵
      1. Burles K,
      2. Innes G,
      3. Senior K, et al
      . Limitations of pulmonary embolism ICD-10 codes in emergency department administrative data: let the buyer beware. BMC Med Res Methodol 2017;17:89. doi:10.1186/s12874-017-0361-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/28595574
      OpenUrlPubMed
    19. ↵
      1. Clement FM,
      2. James MT,
      3. Chin R, et al
      . Validation of a case definition to define chronic dialysis using outpatient administrative data. BMC Med Res Methodol 2011;11:25. doi:10.1186/1471-2288-11-25pmid:http://www.ncbi.nlm.nih.gov/pubmed/21362182
      OpenUrlCrossRefPubMed
    20. ↵
      1. Edmondson ME,
      2. Reimer AP
      . Challenges frequently encountered in the secondary use of electronic medical record data for research. Computers, Informatics, Nursing 2020;38:338–48.doi:10.1097/CIN.0000000000000609
      OpenUrl
    21. ↵
      1. Hersh WR,
      2. Weiner MG,
      3. Embi PJ, et al
      . Caveats for the use of operational electronic health record data in comparative effectiveness research. Med Care 2013;51:S30–7.doi:10.1097/MLR.0b013e31829b1dbdpmid:http://www.ncbi.nlm.nih.gov/pubmed/23774517
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Madden JM,
      2. Lakoma MD,
      3. Rusinak D, et al
      . Missing clinical and behavioral health data in a large electronic health record (EHR) system. J Am Med Inform Assoc 2016;23:1143–9.doi:10.1093/jamia/ocw021pmid:http://www.ncbi.nlm.nih.gov/pubmed/27079506
      OpenUrlCrossRefPubMed
    23. ↵
      1. Wright A,
      2. McCoy AB,
      3. Hickman T-TT, et al
      . Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int J Med Inform 2015;84:784–90.doi:10.1016/j.ijmedinf.2015.06.011pmid:http://www.ncbi.nlm.nih.gov/pubmed/26228650
      OpenUrlCrossRefPubMed
    24. ↵
      1. Kern LM,
      2. Malhotra S,
      3. Barrón Y, et al
      . Accuracy of electronically reported "meaningful use" clinical quality measures: a cross-sectional study. Ann Intern Med 2013;158:77-83. doi:10.7326/0003-4819-158-2-201301150-00001pmid:http://www.ncbi.nlm.nih.gov/pubmed/23318309
      OpenUrlPubMed
    25. ↵
      1. Wang EC-H,
      2. Wright A
      . Characterizing outpatient problem list completeness and duplications in the electronic health record. J Am Med Inform Assoc 2020;27:1190–7.doi:10.1093/jamia/ocaa125pmid:http://www.ncbi.nlm.nih.gov/pubmed/32620950
      OpenUrlPubMed
    26. ↵
      1. Lucyk K,
      2. Tang K,
      3. Quan H
      . Barriers to data quality resulting from the process of coding health information to administrative data: a qualitative study. BMC Health Serv Res 2017;17:766. doi:10.1186/s12913-017-2697-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/29166905
      OpenUrlPubMed
    27. ↵
      1. Feder SL
      . Data quality in electronic health records research: quality domains and assessment methods. West J Nurs Res 2018;40:753–66.doi:10.1177/0193945916689084pmid:http://www.ncbi.nlm.nih.gov/pubmed/28322657
      OpenUrlCrossRefPubMed
    28. ↵
      1. Cohoon TJ,
      2. Bhavnani SP
      . Toward precision health: applying artificial intelligence analytics to digital health biometric datasets. Per Med 2020;17:307–16.doi:10.2217/pme-2019-0113pmid:http://www.ncbi.nlm.nih.gov/pubmed/32588726
      OpenUrlPubMed
    29. ↵
      1. Hudaa S,
      2. Setiyadi DBP,
      3. Lydia SE
      . Natural language processing utilization in healthcare. International Journal of Engineering and Advance Technology 2019;8.
    30. ↵
      1. Shaban-Nejad A,
      2. Michalowksi M
      , eds. Precision health and medicine: A digital revolution in healthcare. Switzerland: Springer Nature, 2020.
    31. ↵
      1. Maddox TM,
      2. Rumsfeld JS,
      3. Payne PRO
      . Questions for artificial intelligence in health care. JAMA 2019;321:31.doi:10.1001/jama.2018.18932pmid:http://www.ncbi.nlm.nih.gov/pubmed/30535130
      OpenUrlCrossRefPubMed
    32. ↵
      1. Wang F,
      2. Casalino LP,
      3. Khullar D
      . Deep learning in Medicine-Promise, progress, and challenges. JAMA Intern Med 2019;179:293. doi:10.1001/jamainternmed.2018.7117pmid:http://www.ncbi.nlm.nih.gov/pubmed/30556825
      OpenUrlPubMed
    33. ↵
      1. Federation of Medicine Regulatory Authorities Canada
      . Physician practice improvement, 2016. Available: http://fmrac.ca/wp-content/uploads/2016/04/PPI-System_ENG.pdf [Accessed 24 Feb 2021].
    34. ↵
      1. Ye J
      . The impact of electronic health record-integrated patient-generated health data on clinician burnout. J Am Med Inform Assoc 2021;28:1051–6.doi:10.1093/jamia/ocab017pmid:http://www.ncbi.nlm.nih.gov/pubmed/33822095
      OpenUrlPubMed
      1. Diabetes Canada clinical practice guidelines expert Committee
      . Diabetes Canada 2018 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes 2018;42:S1–325.
      OpenUrl
      1. Alberta Health Services
      . AHS recommended drug regimens for surgical prophylaxis in adult patients, 2018. Available: https://www.albertahealthservices.ca/assets/info/hp/as/if-hp-as-surgical-prophylaxis.pdf [Accessed 24 Feb 2021].
      1. Alberta Health Services
      . Connect Care - Glossary, 2021. Available: https://www.albertahealthservices.ca/info/Page15455.aspx#cis [Accessed 24 Feb 2021].

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • TM and KC are joint first authors.

    • Contributors TM, TWM ad KC conceived the project idea and drafted the manuscript. BS ensured data accuracy and contributed to the project methods. ROY and DC-S provided local clinical expertise. DC-S, TM, TWM, KC and ROY edited the manuscript.

    • Funding Supported by a financial contribution from the Government of Alberta via the Physician Learning Programme.

    • Disclaimer The views expressed herein do not necessarily represent the official policy of the Government of Alberta (no award/grant number).

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.