In response to the report that 44 000–98 000 patients die yearly as a result of medical errors, the Institute of Medicine (IOM) identified mandatory and voluntary reporting systems as important components of patient safety improvement. Mandatory reporting has focused on holding providers and healthcare organizations accountable to the public for serious events.¹ Voluntary reporting systems vary in scope from event specific national reporting systems (for example, medication errors, sentinel events) to hospital based internal reporting systems that capture a variety of events. Although the approaches and information collected differ—for example, pre-coded drop down menus, open ended narratives, anonymous, confidential, etc—the importance of reporting and the unifying goal to learn from experience remains the same.

Voluntary reporting beyond medication incidents and falls has not been widely reported to date. In particular, efforts to promote voluntary reporting of adverse clinical events and near misses have not been a traditional part of hospital reporting programs. Historically, hospital quality assurance programs have focused on medical error and less on harm, injury, near misses, and unsafe conditions.

Characteristics considered to be important for a successful voluntary reporting program include a non-punitive or safe environment, simplicity in reporting, and timely and valuable feedback.² The best methods of reporting are not yet known and technology to facilitate reporting has not been studied.³ The private sector has begun to address the inadequacies of the older reporting systems through a variety of new electronic systems.

Here we report on our early experience using the Risk Prevention and Management (RPM) system, a proprietary medical error tracking system from DoctorQuality Inc, as our hospital’s voluntary electronic reporting system (ERS).^3,4

The study objectives were: (1) to develop and implement educational initiatives to facilitate the use of the ERS; (2) to measure the impact of the ERS and educational initiatives on knowledge of the technology, reporting behavior, and safety attitudes; and (3) to evaluate the accuracy of the information being reported.

METHODS

Study setting

The study took place at Strong Memorial Hospital, a 750 bed teaching hospital of the University of Rochester Medical Center which employs 597 graduate medical trainees across 67 training programs. The hospital treats 38 000 inpatients, 325 000 outpatients, 86 000 emergency patients, and performs 35 000 operations annually. In addition, New York State’s mandatory external, potentially punitive, reporting system (NYPORTS) has been in existence for over 15 years.⁵ The study was reviewed and exempted from the local institutional review board.

Study timeline

The study timeline is shown in fig 1. It includes three periods: planning, transition, and full implementation. During the 18 month planning period from 1 January 2000 to 30 June 2001, conceptual and preparatory activities occurred and in October 2000 the hospital approved a non-punitive reporting policy. During that period safety related events were reported using multiple decentralized paper based systems. A transition period from 1 July 2001 to 31 December 2001 included the implementation of the ERS and provided staff with a choice to use the new ERS or continue with the old paper based reporting systems. Full implementation began on 1 January 2002 with the elimination of the paper based reporting systems and included the study measurement period through to 31 December 2002.

Electronic reporting system (ERS)

The features of the ERS are shown in table 1. Electronic reporting is our hospital’s voluntary internal and confidential (non-anonymous) system for reporting safety events involving patients and visitors.⁴ The reporting system was implemented throughout all clinical care areas including inpatient, outpatient, same day surgical, emergency department, ancillary, clinical, and administrative support areas. All employees with a user identification and Intranet password are able to access the system. Access to the ERS was divided into two user types: “level 1” users who are able to enter an event only, and “level 2” users who are accountable managers/quality assurance liaisons with additional access to the database functions of event notification, event editing, follow up and closure, data queries, analysis, and reporting functions.

The ERS is arranged into four major event classifications defined as falls, medication, adverse clinical events (ACEs), and administrative events. Using drop down menus, the ERS prompts the user to answer 13 questions including site, location, date, time, type (patient/visitor), medical record number, patient name, date of birth, reporter role, event classification, event subtype, level of impact, and accuracy verification check box. The level of impact uses a taxonomy scaled from 0 to 10. Contributing factor options include system, equipment, human, environmental, and patient. There are additional menu driven prompts to capture more details about falls, medication/infusion, transfusion, and medical device/implant related events. An optional open text field is available for a brief description of the event. Fields are available for a “level 2” user to enter additional information about the event and follow up actions taken or planned. Each event is defined as one of three states: new (event has not been reviewed), open (event has been reviewed and additional details may have been added), and closed (no further action required).

Educational intervention

We implemented a multifaceted educational program to promote safety awareness, understanding, and use of the ERS system.

Event reporting awareness and training

Promotion of safety awareness and the plan to implement ERS occurred through periodic broadcast emails informing of the ERS implementation date, features of the system, purpose, security, access, and usage. Beginning in July 2001, informational sessions and more specific user training sessions were rolled out via standing hospital wide, departmental, and nursing meetings. Electronic event reporting was added to the hospital mandatory orientation and yearly in-service training. Ad hoc training was provided for individuals or small groups as needs were identified. Safety related communications were disseminated through existing medical center newsletters (physician, resident, nursing, and employee newsletters) explaining the ERS, soliciting support for the system, reiterating the hospital non-punitive reporting policy, and promoting the practice of near miss reporting.

Senior and clinical leadership involvement

We buttressed our educational efforts by adopting methods used in previous opinion leader research.⁶ A 1 hour meeting was held with 14 nurse managers and the physician leaders of the adult intensive care units and operative service areas. During that meeting we provided an orientation to the ERS, a tutorial for navigating the management reports query function, enlisted their leadership support for the new system, and explained the importance of reporting. To support the education we developed an information packet that consisted of a notebook of reference materials organized by topics for easy use. The safety information packet included study background information, key contacts, the role of an opinion leader, details about our ERS, data reports, and patient safety information (definitions, key talking points about patient safety, safety science, non-punitive reporting, and the safety survey).

Event feedback

Feedback on ACEs was achieved with a newsletter called “E-Reporter” (available from the author upon request). Each “E-Reporter” comprised four sections: ACE facts, an event reporting tip, a question and answer about how to report, and an improvement effort related to the ACE being highlighted. Five E-Reporters were issued between June and December 2002 covering the following topics: respiratory, skin integrity, laboratory issues, deep vein thrombosis/pulmonary embolism (DVT/PE), and operative/invasive procedure events. Selection of ACEs was based on sufficient report frequency to categorize trends of event details and a perceived opportunity for improvement. Partnering occurred with clinical stakeholders to understand and report improvement efforts related to the highlighted event type. The “E-Reporter” provided a mechanism to make harm more visible, provide improvement tips, and highlight improvement efforts related to the newsletter topic. The newsletter was distributed to senior management, physicians, nurses, pharmacists, department heads, and supervisors via global electronic mail lists. Further distribution of the newsletter to staff without email was requested through forums such as staff meetings and/or posting of newsletters for staff review.

Outcome measures

Several methods were used to measure safety behavior and attitudes.

Safety events

The term safety event includes all events reported in the ERS. Electronically submitted safety events were evaluated for the period from 1 January 2002 to 31 December 2002 and compared with total events reported by our existing paper based systems (falls, ACEs, administrative events, and medication related incidents) from 1 January 2000 to 31 December 2000. ERS events were analyzed by event review status, type of event, reporter, event classification, event location, level of impact, and contributing factors. Separate analyses were performed for the subcategory of all ACEs and the events making up the five “E-Reporter” newsletters.

Central event analysis

The study team (DT, RH and TB) performed a detailed review of the 534 events that contributed to the five “E-Reporter” newsletters. Details of the event were reviewed and an accuracy analysis of the event reporters’ coding decisions for their coding of the event classification, level of impact and event location was performed. Gold standard judgments by the study team were made by consensus after reviewing all the details of the event and, if necessary, contacting the appropriate clinical or administrative personnel. In addition, the extent of underreporting was evaluated by comparing events in the ERS with co-existing methods of collection for comparable time periods in which reporting data were available: (1) the number of submitted DVT/PE events in the ERS was compared with the number of DVT/PE events identified using methods and administrative data as previously described,⁷ and (2) the number of decubitus ulcers submitted in the ERS was compared with existing event specific paper based tracking systems.

Safety knowledge and attitudes

We modified an 11 question safety culture survey used by the Institute for Healthcare Improvement (IHI) Quantum Leaps Collaborative⁸ using a 5 point Likert scale ranging from “agree” to “disagree”. Two questions were added to the IHI survey that were specific to event reporting: (1) “I know how to enter information into the DoctorQuality event reporting system” and (2) “The process of reporting medical errors at my hospital is cumbersome”. An employee was eligible to receive the survey once in each 12 month period. The survey consisted of a one time mailing and receipt and return of the survey was completed by hospital intramural mail.

Analysis of data

All statistical tests were two sided and p values of <5% were considered significant. t tests were used to compare continuous variables and χ² tests were used to compare dichotomous variables. Regression analyses were used to assess the significance of trends over time for the number of reported events and the percentage agreeing on the safety survey (response of 4 or 5). Month was treated as a continuous independent variable in the models, and the significance of its regression coefficient indicated a trend over time in the dependent variable of interest. All analyses were conducted using SAS software (SAS Institute, Cary, NC, USA).

RESULTS

Safety events

A total of 2843 events were entered into the ERS between 1 January 2002 and 31 December 2002. This compares with 1542 paper based reports from 1 January 2000 to 31 December 2000. Of the 2843 events reported, 40% (n = 1126) were medication/infusion events, 30% (n = 851) were ACEs, 24% (n = 693) were falls, 5% (n = 137) were administrative, and 1% (n = 36) were other events. Figure 2 shows that the reporting frequency for ACEs increased over time (p = 0.009), medication events trended upwards (p = 0.095), and no trends were seen in falls (p = 0.623) or administrative events (p = 0.361). In August 2003, 72% of all events continue to remain “new” or “open”.

Table 2 shows the characteristics of all electronically submitted events for ACEs only and for the events that comprised the five “E-Reporters”. Nurses reported most of the events (73%) while physicians reported only 2% of all events. Most of the events were reported on inpatient units (70%) whereas ACEs were most commonly reported in the support service areas (43%). Patient harm occurred in 22% (21% temporary harm and 1% permanent harm) of all submitted events but was more prevalent in ACEs (39%) and within the “E-Reporters” (63%). Physicians tended to report events associated with more harm (38/68 events, 56%) than nurses (407/2071 events, 20%). Near misses contributed to 10% of all submitted events compared with 6% for unsafe conditions. The largest level of impact category was “no harm, reached patient” (n = 1531), and most of this category were patient falls and medication events. Reporters considered human factors as contributing to nearly 50% of all reported events and system factors to less than 10%.

Central event analysis

Table 3 summarizes information obtained in preparing the five “E-Reporters”. A total of 219 submitted events underwent central analysis, 55 of which (25%) were submitted in the wrong event class or category, 45 (20%) had a miscoded level of impact, and six (3%) had a miscoded location. Feedback messages, which included both suggestions for improved reporting and clinical safety practices, are also shown in table 3.

For the period from 1 July 2001 to 31 December 2001, three DVT/PE events were reported in the ERS and 203 were identified and reported to NYPORTS. No nosocomial pressure ulcers were entered in the ERS compared with 30 recorded by the nurse specialist’s tracking system for the same time period (1 January 2002 to 30 June 2002).

Safety knowledge and attitudes

Of the 7095 culture surveys mailed, 733 were returned (response rate 10.3%): 254 (35%) were from nurses, 254 (35%) were from other clinical personnel, and 225 (30%) were from physicians. Responders and non-responders were similar with respect to sex but responders had a longer length of hospital service than non-responders (p<0.0001).

Given the low response rate, conclusions from the survey data are limited. Of those who did respond, however, 85% stated that they knew the proper channels to report safety concerns but the majority did not know how to enter information into the ERS and believed that the process of reporting was cumbersome. Two survey questions showed significant changes in the percentage agreeing over time: (1) “I know how to enter information into the DoctorQuality event reporting system” increased over time (p = 0.001, linear trend) and (2) “My manager/supervisors in my hospital listen to me and care about my concerns” decreased over time (p = 0.04, linear trend). These trends were largely driven by nurses.

DISCUSSION

Our study has shown that implementation of a hospital wide electronic web based reporting system at a large academic medical center is possible. Improvement in the frequency of reporting and evaluation of event information to understand organization performance can be realized within the first year of implementation. Using complementary educational efforts around the purpose, importance, and usage of our new reporting system, we were able to increase reporting significantly and to improve our employees’ knowledge and use of the ERS. The ERS has also become an effective concurrent risk management tool by providing timely communication about real time events as they unfold.

The first goal of an ERS should be to increase the numbers of reported events or errors, as this will provide an increasingly rich source of information for analysis and feedback for improvement purposes. A component of increased reporting should include wider representation by all healthcare personnel to broaden the content of events reported, as well as the perspectives to understanding contributing factors. In our study nurses reported 73% of all events while physicians reported only 2%. The infrequency of physician reporting is probably due to a number of reasons based on cultural factors, time factors, fear factors, or lack of awareness. The latter is supported by a study which found that practising physicians do not view medical error as an important health problem even though they reported personal experiences with medical errors that had serious consequences.⁹ Efforts to support improvements in physician participation in safety practices are clearly needed, and a potential upstream solution is the practice based and systems based core competency within the Accreditation Council of Graduate Medical Education (ACGME) Outcome Project.¹⁰

Even though early emphasis on reporting is necessary, it is not sufficient for the ultimate goal of improving patient safety. In fact, too much emphasis on reporting without timely and useful feedback based on the information submitted will be counterproductive. At least two types of feedback are necessary to improve patient safety.

Firstly, the ERS has facilitated a mechanism to make harm more visible and will help to promote the first precondition for improvement—defined as will.¹¹ Without local visibility it is felt that the national will to improve patient safety will be hard to translate into local intent.¹² In fact, 623 patients or 22% of all events submitted incurred harm during the course of their care: 29 patients suffered permanent harm, near death, or death. The ERS has provided the nomenclature to define and quantify harm, and to facilitate awareness and understanding that harm is much more than sentinel events.

Secondly, the ERS provides the data warehouse to analyze trends of events, potential contributing factors, and the systems based, human based, or patient based solutions ultimately to improve safety. We attempted to operationalize this feedback loop via the “E-Reporter” and, although the newsletter may have had an impact on increasing the number of events submitted, its ultimate impact on safety practices and patient outcomes is not yet known. Providing this type of timely and useful feedback remains one of the biggest challenges to the patient safety movement.

The sheer number of events that could be reported appears daunting—as evidenced not only by the more than 300 event categories within the ERS, but also by the massive undercoding that we uncovered for only two of the event categories. In addition to the adverse events themselves and to fully exploit the potential of the ERS, near misses and other forms of causal information need to be entered in sufficient quantity and accuracy to ensure the detection of non-spurious contributing factors to patient harm. In this study only 10% of all events were near misses and 6% were unsafe conditions, and little is known to date on how best to capture, code, and analyze this information for improvement purposes.

Miscoding of data elements on data entry is particularly relevant to event reporting systems with pre-coded variables. We found that 25% of all events submitted as part of the five “E-Reporter” newsletters were classified in the wrong event category and 20% had a miscoded level of impact. Miscoding is probably the result of many factors including tool design, lack of reporter knowledge, lack of standard nomenclature or a data dictionary, and limited accountable oversight. Although a certain amount of inaccuracy should be tolerated, the threshold at which misclassifications will begin to compromise the central event analysis function is not yet known.

At one extreme, the central event analysis function can be viewed as a Data Safety Monitoring Board (DSMB) used for most multicenter clinical trials. DSMBs periodically review safety data and recommend modifications—and possibly termination—of clinical trials to ensure subject safety.¹³ At the other extreme, the central analysis function can be relegated to a database manager whose function is solely to maintain and clean data. Even though the resources to maintain a web based technology and marshal the expertise required to analyze the events appear to be immense, they can probably be found in waste and inefficiencies in the current methods of quality review and error analysis. In time, the improvement learned and implemented should more than pay for itself.

We are in the early stages of evaluating voluntary event reporting technologies and measuring safety related processes and outcomes. Focusing too much too early on changing patient outcomes may detract us from the ultimate goal, given the complexities inherent in studying policies aimed at changing institutional and provider behavior. Based on the survey responses (although low) and trends in reporting, employees increasingly “know” and “know how” to use the new technology. Continued education and guidance is critical to improving the reliability of coding adverse events. Equally important, however, is continued education around the science of patient safety, given that in nearly 50% of reported events “human factors” were considered contributory compared with less than 10% for “system factors”—probably an underestimate but potentially an accurate perception for those at the sharp end of care.¹⁴

Successful reporting systems have the common characteristics of being safe, simple, and worthwhile.² Use of an ERS supports the call for transparency for both the safety of patients and the satisfaction of caregivers in a non-punitive environment where information is freely shared and responsibility broadly accepted.¹⁵ Our early experience with an ERS has moved us towards transparency in our quest for more knowledge to improve patient safety at our institution.