Article Text
PDF
Statistics from Altmetric.com
Introduction
The publication of To Err is Human by the Institute of Medicine (IOM, now the National Academy of Medicine) in 2000 proved to be a watershed moment in healthcare.1 For the first time, national attention focused on flaws in the healthcare system that regularly led to patient harm. The report drew widespread attention from the press and began an important national discussion that continues today. A follow-up report from the IOM in 2004, Patient Safety Achieving a New Standard, recommended a series of sweeping changes.2 Principal among them, the IOM argued that ‘all healthcare organisations should establish comprehensive patient safety programmes operated by trained personnel…’ The main aim of the proposed programme was to create systems for collecting and analysing patient safety data and using the knowledge to reduce risk and prevent future incidents.
Systematically identifying and tracking patient safety incidents—defined as events occurring during an episode of patient care that have the potential to, or actually cause injury or harm to a patient—is now regarded as the cornerstone of safety reporting systems.3 The major tool for collecting patient safety reports from frontline clinicians and other patient-facing staff is a voluntary incident reporting system; this tool allows data to be collated and analysed to identify areas of safety risk, and to ameliorate them.3 4 Incident reporting systems are now mandated by The Centers for Medicare and Medicaid Services, the National Health Service and other agencies around the globe. Many hospitals now rely on web-based proprietary reporting systems such as RLDatix, which supports >5000 customers worldwide.
The starting point for any incident reporting system is a common language.5 General classification systems for patient safety incidents have been published by the Joint Commission and by WHO.6 7 Although the development methodologies used were different (structured literature review for the former, Delphi consensus panel for the latter), both stressed the critical importance of consistency of language. Both are intended to have broad applicability across locations of care, healthcare facility type, medical specialty and clinician category. In addition to broadly applicable ones, safety incident classification systems have been developed for use in various specialties including internal medicine, surgery, paediatrics, obstetrics and gynaecology and psychiatry.5 8–13
We have been able to identify only three reports describing the use of an incident reporting system in medical oncology. Two of these describe the implementation and early evaluation of a paediatric oncology incident reporting system implemented at Vanderbilt University Medical Center in 2003.14 15 Ninety-seven incidents were described; ordering errors were the most common. Details of the classification system, including how it was designed, were not reported. Lennes et al described the implementation of an incident reporting system at the Massachusetts General Hospital Cancer Center in 2016.16 The classification system focused exclusively on chemotherapy medication errors, was developed by a group process, but not described further. Our review, therefore, has identified a potentially important gap in the literature—the lack of a high-quality, validated incident taxonomy focused on the practice of medical oncology.
In this report, we describe the development of a comprehensive, validated taxonomy for medical oncology-related incidents. We combined qualitative analysis with multiple, rapid cycles of testing at three different healthcare facilities to validate the classification system. Our goal was to create a taxonomy that will be broadly applicable to the practice of medical oncology across different sites of care.
Methods
Neither patients nor the public were involved in the design, conduct, reporting or dissemination of our research.
Setting and sample
The project was performed across three different healthcare facilities based in the USA. Site 1 is a free-standing National Cancer Institute-designated cancer centre that provides adult and paediatric oncology care. Site 2 is based in an academic medical centre (AMC) that provides adult cancer services. Site 3 is an AMC that provides both paediatric and adult cancer services, but only adult services are included in this project.
The data source included all ambulatory incident reports from site 1, all hematology-related and medical oncology-related incidents from site 2 and all oncology-related incidents from site 3. The measurement period was 1 January 2019 through 31 December 2020. All sites record incident reports using the same version of a proprietary incident reporting system (RLDatix). The details for each site, including incident report inclusion and exclusion criteria, are provided in table 1. Data fields were downloaded into a spreadsheet (Microsoft 365 Excel), de-identified for all patient and staff information and randomly ordered at each site.
Site Characteristics
Selection and modification of coding framework
Development of the taxonomy used an iterative process involving a literature review, content analysis, rapid cycle testing of a draft taxonomy and expert panel review (online supplemental figure 1).
Supplemental material
For the purposes of this project, we define a taxonomy as a scientifically driven framework that hierarchically arranges patient incidents into logically organised domains and categories. It is distinguished from a classification system which arranges incidents by simple grouping only.6
To develop a taxonomy, we systematically reviewed the published literature for safety incident classification systems including those that are intended to be broadly applicable as well as those that focused exclusively on medical oncology. In addition, we reviewed related classification systems such as those designed to catalogue patient and family complaints. We sought a clinically intuitive system that would lend itself to easy applicability by patient safety analysts who are responsible for maintaining incident reporting systems.17 We concluded that the patient and family complaints taxonomy developed by Reader et al was easily applicable to incident reporting, and that it was amenable to modification for medical oncology-related incidents.18 The taxonomy includes three broad domains, each with two to three categories: clinical (categories: patient safety and quality); relationships (categories: communication; humaneness/caring and patient rights) and management (categories: institutional issues and timing/access). An expert advisory panel approved moving forward with the Reader framework.
We employed content analysis on a crosswalk of incident types currently used at the three sites to identify key components for the incident taxonomy and to refine the taxonomy.19 Thirty distinct high-level incident types were identified. We compared these with the categories included in the report by Lennes et al.16 After removing duplicates, an additional five categories were added bringing the total to 35 incident types. These were then fitted into the Reader framework resulting in 26 categories and 22 subcategories divided between the three domains. Next, two coders (JOJ, TM) independently reviewed 50 incident reports from site 1 and coded them using the draft taxonomy. The coders then met to clarify areas of agreement and disagreement and ways in which the coding framework could be improved. Thereafter, one of the coders (JOJ) met with the qualitative researcher (ACR) to review the results and modify the draft taxonomy as needed. Two coders (JOJ and an oncology physician assistant) then separately analysed an additional 62 site 1 incidents during three separate sessions, repeating the process described above, modifying the coding framework between each review cycle. This single site incident coding exercise allowed for draft coding instructions to be created and enabled us to proceed to rapid cycle coding across all sites.
Rapid cycle testing of the taxonomy
The next phase of the project entailed a series of rapid cycle tests based on the principles of the plan-do-study-act methodology employed in quality improvement.20 The process is highlighted in figure 1. The goal of the rapid cycle process was to accelerate the taxonomy development process. Volunteer coders were instructed to read each incident carefully, searching for key words or phrases that best explained why the incident reporter placed the entry, and to search the incident and contributing factor code sheets to identify the best fit for each. Each incident was coded for primary and secondary incident types and up to two contributing factors. Coders were instructed to ignore whether the incident resulted in harm or not, or whether the incident was completed or interrupted (ie, a near-miss). In addition, coders were instructed to concentrate only on the incident as described and not to draw inferences or bring specialised knowledge to bear (see online supplemental coding instructions).
Supplemental material
Rapid cycle testing diagram
Coders catalogued the incidents in a spreadsheet employing dropdown menus to select incident types and contributing factors. Coders were required to select at least one incident type, and to select a second one only if the incident report contained two distinguishable occurrences (eg, the primary incident described a pharmacy delay, but the patient also reported being treated rudely by staff members). Coders were encouraged to select at least one contributing factor if one could be identified. Finally, coders were also asked to provide a one-sentence or two-sentence justification for their choice of an incident type.
Each cycle was designed for a turnaround time of <1 week. One of the coders (JOJ) and eight different secondary coders met sequentially for an hour each to review the coding instructions and the incident code sheet and contributing factors code sheet; this included coding five cases together. The coders then separately analysed approximately 20 incidents and met again within 48 hours to compare results side-by-side and to provide feedback on the coding tools. One of the coders (JOJ) then met with the qualitative researcher (often with SOS, an epidemiologist) to review the results of the rapid cycle with focus on disagreements. Using a structured process in which comments from the coders were catalogued and reasons for disagreements were clarified, the coding instructions and code sheets were updated, and another rapid cycle was performed.
An interdisciplinary expert advisory panel supervised the progress of the project and provided feedback at key inflection points. The panel reviewed and approved the final incident taxonomy. The panel included adult oncologists, a paediatric oncologist/health service researcher, a senior director of patient safety, a systems safety expert and members of the safety teams from all three sites who regularly code medical oncology incidents.
Descriptive statistics and other analyses were calculated using Excel.
Results
Three hundred seventy-three incidents were reviewed including the content analysis and rapid cycle tests. A total of 10 rapid cycle tests were conducted over 12 weeks during which 221 incidents were coded, including 81 from site 1 (JOJ plus four different coders), 89 from site 2 (JOJ, JAZ and two other coders) and 51 from site 3 (JOJ, TM and one other coder). The coding instructions were updated following each rapid cycle to maximise clarity. The incident code sheet was revised a total of 13 times. The final version of the incident coding sheet is displayed in table 2. The final versions of the instruction guide and contributing factors code sheet are included in the online supplemental materials.
Supplemental material
Incident coding guide
The final incident coding taxonomy conforms closely to the Reader framework. The ‘clinical’ domain includes 10 safety-related and 4 quality/knowledge management skills incident categories (along with 19 and 5 subcategories, respectively). There are six ‘relational’ domain categories and five ‘institutional/management’ domain categories (including zero and four subcategories, respectively). Most of the modifications to the incident coding guide were minor changes to the descriptive language (table 2, column 2). A few significant deviations from the Reader framework were necessary. For example, ‘delay in care’ is listed under ‘institution/management’ in the Reader framework. But, because it is a broad, high-level issue, and not easily actionable, the category was eliminated. In its place, delays were reassociated with discreet incident categories (eg, pharmacy-related, provider-related, etc). Other changes to the taxonomy were necessary to encompass safety incident reports related to employee risk rather than patient risk. A category of ‘patient factors’ was added under the relational domain. ‘Waste and inefficiency’ was added under the institutional/management domain. ‘Policies/Procedures’ and ‘protocols and guidelines’ required modification to distinguish between design issues and adherence.
The results of the rapid testing cycles are summarised in table 3.
Rapid cycle test results
Only five incident types (0.03%) could not be assigned by one of the coders; there were no incidents in which both coders were unable to select an incident type. There was domain-level agreement between the two coders in 75% (median, range 67%–95%) of incident reports. The median level of agreement at the category level was 65% (range 50%–85%). During the debriefing sessions, a variety of reasons emerged to explain disagreements between coders. Early in the process, these disagreements were often due to ambiguous language in the event coding guide and in the coding instructions. Over time, as the coding language became better clarified, disagreements were most often attributable to limitations in the reporting narrative. Clarity of the reports varied widely as did report length, with some reports as brief as a single sentence and others many paragraphs in length. A common area of ambiguity related to delays in the receipt of chemotherapy. It was sometimes impossible to distinguish, for example, whether a delay was due to slowness in signing an electronic order or a backup in the pharmacy.
Discussion
Incident reporting systems have been an important tool for evaluating and managing safety in several high-risk industries, most notably commercial air travel, for nearly 50 years.21 Today, they are widely employed in healthcare organisations throughout the world.5 Their usefulness, especially for clarifying risky practices, has been documented in high-complexity, high-risk areas of medicine such as surgery, anaesthesia and critical care.9 22–25 Another high-risk area of medicine is cancer care, which often requires exquisite coordination between different specialties including medical oncology, radiation oncology, surgery, pathology and radiology.26 In addition, there have been major therapeutic advances in cancer care in recent years; new treatments are often complex and carry the potential for significant harm if administered improperly or if poorly coordinated. In medical oncology, whole new therapeutic categories have been introduced including engineered cell therapy, immune checkpoint inhibition and highly targeted agents (often orally administered). In contrast to radiation oncology, where incident reporting has become common place and where a large international registry is active, medical oncology-related incidents have not been well studied.27–33
The aim of the current project was to develop a medical oncology incident taxonomy that will have widespread applicability and that is intuitive, easy to implement and easy to use. Using a qualitative approach, we modified an existing framework for classifying patient and family complaints to create a medical oncology incident taxonomy. Then, employing multiple rapid testing cycles including feedback from coders, we finalised the taxonomy by testing it across three different sites of care. The final taxonomy is hierarchically arranged with 3 domains, 44 categories and 28 subcategories.
To the extent possible, we sought to confirm the validity of the taxonomy, recognising that there is controversy about whether qualitative research is amenable to validity testing.34–36 We chose to determine validity based on the following criteria: assessment of bias, transferability, usability, confirmability and transparency.35 We managed bias in the following way: structured review of the literature at project inception designed to maximise inclusivity; random selection of incidents; double coding of incidents and use of highly structured training of coders using concise language. Bias was further managed during the analysis phase of each testing cycle by including team members with different areas of expertise (qualitative research, epidemiology and a medical oncology). Transferability was addressed by testing at three different facilities (including various combinations of ambulatory and inpatient events, adult and paediatric patients, patients receiving standard chemotherapy and those on research trials and academic and community settings). Usability was determined by the degree to which incidents could be successfully coded (99.7% of incidents). We assessed dependability based on the rate of intercoder agreement and on input from a highly qualified interdisciplinary expert advisory panel. Finally, we addressed transparency by sharing our entire methodology and all the materials that we developed (tables, figure and online supplemental materials).
We believe that the high level of intercoder agreement that we achieved was at least partly due to the emphasis on training. One of the investigators (JOJ) met with each coder for a full hour before each testing cycle to provide detailed instruction (described in the ‘Methods’ section); this was followed by another hour for debriefing, focusing on opportunities to improve the incident coding sheet and training guide. Early in the process, most disagreements between the two coders were due to limitations in the coding tools. Later in the process, most disagreements were due to ambiguity in the incident narratives themselves. We cannot overemphasise the importance of training for anyone who might wish to employ the methodology.
It is important to point out the limitations of our project. Key among them is the narrative reports themselves. By their very nature, the reports are subjective.18 They are often emotive, submitted at a time of anger, distress or frustration. Length and quality of the reports ranged widely. Another limitation is that we focused the project on developing an incident coding taxonomy. Although we collected information on contributing factors, the data are exploratory only. Finally, we were unable to collect information on incident severity.
We recognise that incident taxonomies must be viewed as ‘living documents’. That is, based on accumulating knowledge garnered through usage in different environments, periodic updates will be necessary. At the same time, we understand that there will need to be an ongoing dynamic balance between sensitivity and specificity; adding additional incident categories will increase the ability to pinpoint specific safety areas, but at the risk of making the taxonomy unwieldy. An example in which the current taxonomy lacks specificity is subcategory A3b, ‘administration error’. The current version of the taxonomy does not distinguish between high-risk medications (chemotherapeutic agents) and lower risk ones (eg, antinausea drugs). In addition, the current taxonomy lumps wrong patient incidents with other categories of errors. ‘Wrong patient’ errors are particularly high-risk incidents; the taxonomy will benefit from calling them out specifically.37 These issues will be addressed in a future version.
In this publication, we share how we developed the first validated incident reporting taxonomy for medical oncology. Our approach was novel, using a qualitative research framework in combination with multiple rapid testing cycles based on the plan-do-study-act principles of quality improvement. The combined approach allowed for accelerated completion of the project in less than a year. Our next step is to test the taxonomy at our institutions to determine its usefulness for tracking and analysing incidents, for discerning areas of patient risk and for identifying opportunities for improvement interventions. The taxonomy also has the potential to be deployed as part of a national or international medical oncology incident reporting registry akin to the Radiation Oncology Incident Reporting System which has been in place since 2014.28 Finally, the methodology and framework that we have described may be applicable to other specialties that wish to develop an incident taxonomy.
Ethics statements
Patient consent for publication
Ethics approval
The project was deemed exempt from IRB review since it did not meet the criteria for research involving human subjects as defined by the Department of Health and Human Services and Federal Drug Administration regulations.
Acknowledgments
We acknowledge the contributions of Heresa LaForce who provided early project management. We are grateful to Tandi Dandiwate, MD, MPH, Dana Platt, MSHCA, RN, CPHRM, Kelly Boyle RN, DNP, Gail Marlowe, RN, JD, Jessica McGuire, RN, James Doolin, MD, Aya Sato-Dilorenzo RN and Taryn Lacasse, PA for volunteering their time to test the taxonomy and for providing critical feedback. We appreciate the contributions of Expert Advisory Panel which included Sharon Lane, RN, CPPS, Jennifer Mack, MD, MS, Lawrence Wong, PhD, Jessica McGuire, RN, Lisa Fitzgerald, MHA, Dana Platt, MSHCA, RN, CPHRM and Brendan Sendrowski, RPH.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors All authors contributed to the design, execution and analysis of the project and all contributed to manuscript preparation.
Funding Risk Management Foundation of the Harvard Medical Institutions (no award/grant number).
Competing interests None declared.
Patient and public involvement statement Neither patients nor the public were involved in the design, conduct, reporting or dissemination of our research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.