Article Text
PDF
Abstract
Background Patient complaints and compensation cases are analysed individually and do not allow for organisational learning. Systematic information on complaint patterns needs evidence-based measures. The Healthcare Complaints Analysis Tool (HCAT) can systematically code and analyse complaints and compensation claims, but whether this information is useful for quality improvement is underexplored. We aim to explore if and how HCAT information is perceived useful to inform healthcare quality gaps.
Methods To explore the HCAT’s usefulness for quality improvement purposes, we used an iterative process. We accessed all complaints relating to a large university hospital. Trained HCAT raters systematically coded all cases, using the Danish version of HCAT.
Intervention The intervention had four phases: (1) coding of cases, (2) education, (3) selection of HCAT analyses for dissemination, (4) ‘dashboard’ development and delivery of targeted HCAT reports. To study the interventions and phases, we used quantitative and qualitative approaches. The coding patterns were descriptively displayed on department and hospital level. The educational programme was monitored using passing rates, coding reliability checks and rater feedback. Online interviews recorded dissemination feedback. We used a phenomenological approach with thematised quotations from the interviews to analyse the usefulness of the information from cases coded.
Results We coded 5217 complaint cases (11 056 complaint points). The average case coding time was 8.5 min (95% CI 8.2 to 8.7). All four raters passed the online test with >80% correct answers. Using rater feedback, we handled 25 cases of doubt. None affected the HCAT structure or categories. Interviews verified the usefulness of analyses after expert group dissemination. Three themes were important: ‘overview of complaints’, ‘learning from complaints’ and ‘listening to the patients’. Stakeholders perceived the ‘dashboard’ development as highly relevant.
Conclusion Through the development process with several adjustments, stakeholders found the systematic approach useful for quality improvement. The hospital management evaluated the approach as promising and decided to test the approach in clinical practice.
- Healthcare quality improvement
- Near miss
- Patient-centred care
- Patient safety
- Quality improvement
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Case by case assessment of patient complaints and compensation cases are widely used in healthcare organisations but little evidence exists on its usefulness for learning across healthcare settings.
WHAT THIS STUDY ADDS
This study tests the usefulness of extracted information of complaint patterns based on systematic analysis using the validated Healthcare Complaints Analysis Tool.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Information from systematical analysis of patient complaints seems useful to guide healthcare improvement.
Introduction
Problem description
Most modern health services provide the opportunity for patients to complain about care. Patients that seek injury compensation may have a wish for placement of responsibility or want to help others not to experience the same problems. In Denmark, patient complaints and compensation cases are currently analysed individually to secure the best possible case management and meet legal requirements. The individual approach is appropriate for learning from individual cases but does not allow for comparing cases or detecting trends across healthcare areas, departments and organisations.
If systematic information on complaint patterns across settings and organisations were to be available for quality improvement, existing case handling must be supplemented by evidence-based measures. One approach is by systematically extracting case contents from complaints and compensation claims, using a standardised coding algorithm.
Available knowledge
Every year, the financial costs of complaints and compensation cases amount to over 100 million euro at the Danish national level (and more than 9.5 million euro in 2019 at Odense University Hospital (OUH).1 Apart from causing an escalating cost to health service spending, complaints and compensation claims provide a unique data source on healthcare quality.2 Complaints and compensation claims can be viewed as indicators of patient dissatisfaction and shortcomings in healthcare provision and can inform initiatives to improve healthcare quality and patient safety.3–5 Complaints and compensation claims are often highly motivated by patients’ wishes for improved healthcare quality and safety for future patients.6 7 Patient involvement has been increasingly proven important to quality improvement in healthcare.8 Armstrong et al state that early involvement, real influence and acceptance of the learning potential from patient information are important factors in patient involvement.9 Analysing complaints and compensation claims for healthcare improvement allow for directly involving patients' views and experiences in initiatives aiming at healthcare improvement and can be viewed as fulfilling the perhaps most constructive purpose of these otherwise often highly debated systems. Incentives are many to involve the complaint and compensation cases as essential sources of information in the systematic work to improve the quality of treatment.
Rationale
The Healthcare Complaints Analysis Tool (HCAT) was developed in the UK to systematically code and analyse complaints and compensation claims.10 Gillespie and Reader developed a method to code and classify healthcare complaints,10 and its use is gaining ground.2 11–13 The HCAT coding has four steps: (1) identification of problem category and problem subcategory, (2) identification of the stage where the complaint took place, (3) identification of harm and (4) identification of personnel involved in the complaint.
By coding and analysing complaint and compensation claim letters, patterns can be identified and communicated to management or clinical improvement departments and unlock patient complaints’ potential to inspire quality improvement initiatives. The coding has been shown to be reliable in a Danish sample from emergency care.14 In contrast, it is presently unexplored whether the information in its current form is in fact useful for improvement purposes.
We used an iterative process inspired by the PDSA (Plan, Do, Study, Act) framework to explore whether and how the information may be useful. This model of improvement is used to examine and accelerate improvement processes and has been proved simple and useful internationally.15 Information useful for clinical improvement, evaluation of valid improvement input and bridging this into the daily clinic were inspired by the model of improvement (figure 1).
Iterative development process of the usefulness of systematic analysis of patient complaints and compensation claims. HCAT, Healthcare Complaints Analysis Tool.
Aim
This study aimed to explore if and how patients, clinicians, quality improvement staff and hospital management find the extracting and applying of information from complaints and compensation cases by the HCAT useful to enrich and inform quality gaps in healthcare.
Materials and methods
Context
Disciplinary complaints and compensation claims are assessed centrally by health specialists appointed by the Danish Health Complaints Authority (DHCA) and the Patient Assurance Organization (PAO), respectively. After central assessment, the disciplinary complaint and compensation claim material is sent back to the relevant hospital and is available from local hospital management as well as through DHCA and PAO registries. We assessed the material locally, and throughout this article, complaint and compensation cases are collectively referred to as ‘complaints’.
Material
We accessed all complaints at a large university hospital (965 in-patient beds, 94 500 discharged patients and 974 000 out-patient visits per year) in the Region of Southern Denmark (covering 1.2 million citizens) during the period 1 January 2017 to 31 December 2020. Annually, approximately 2200 patient inquiries are filed to the hospital, and most are complaint cases. These cases are recorded in the hospital archiving system (Acadre). Cases were systematically reviewed and coded by trained HCAT raters according to the Danish version of the HCAT (HCATdk.),5 8 using the Danish HCAT coding manual.16 Before coding, raters had to pass a mandatory coding course developed during the preceding translation and validation process. The HCATdk is structured as follows: Every single complaint identified in the complaint case letter is recorded into the matrix consisting of three overall problem domains (clinical, management and relationship) and seven underlying problem categories (quality, safety, environment, institutional processes, listening, communication and respect and patient rights). The seven problem categories each have four levels of severity (0–3). The stage of care to which the complaint refers is coded into six categories. Harm is recorded from no harm (harm=0) to catastrophic harm (harm=5). Finally, the personnel involved in the complaint are noted (see online supplemental appendix 1). A web-based REDCap (Research Electronic Data Capture),17 coding form was used for entering the data into a database.
Supplemental material
Interventions
The study intervention’s four phases embedded in an iterative development intervention process (figure 2):
Coding of cases: Prior to the study, we translated and adapted the English HCAT coding taxonomy,10 to the Danish context.16 Adjustments to the Danish coding manual were needed, therefore, we translated the entire manual.
Education: The information collected through the systematic coding needs to be highly accurate and reliable. To be able to code the complaint cases, we initially needed to educate a team of HCAT raters. We developed a formal HCAT education system to ensure the quality of coding procedures.
Selection of HCAT material for dissemination: To determine what information was relevant, HCAT findings were repeatedly conveyed to expert groups (health professionals, quality improvement staff, hospital management and patient representatives) and adjusted according to feedback.
Dashboard: To assist further quality improvement and provide an overview of case contents, we developed an easy-to-use ‘dashboard’.
The four phases of the study interventions embedded in an iterative development process.
Study of the interventions and measures
To study the different phases and interventions of the study, we used a mixture of quantitative and qualitative approaches:
Coding of cases: We used the HCAT approach for extraction of case contents (please see above). We translated and adapted the HCAT to the Danish setting (‘HCATdk’).16 Further, the framework and considerations behind the HCAT approach were included in a Danish manual for coding purposes (HCATdk Coding Manual). The HCAT coding patterns allowed us to analyse content types as well as so-called hot spots (defined as ‘hot spot harm’, which are areas with a particularly high incidence of complaints causing major or catastrophic harm or as ‘hot spot near-misses’, which are areas with a particularly high incidence of complaints causing high severity but low actual harm) and blind spots (defined as areas related to admission and discharge problems, system errors related to multiple smaller-scale issues and errors of omission).
Education: We employed four junior research assistants to code the cases for this task. Prior to the coding, they read the Danish coding manual (developed in the first intervention) and attended the developed HCAT coding. The course included a formal online HCAT coding session and a final test (requiring>80% correct answers to pass) to secure the reliability and quality of the HCAT rating. The content of the online course mirrored the English online programme (HCAT Online Training (qualtrics.com)) but included Danish model cases. The online course uses a regional layout and platform and is following regional standards (HCATdk Coding (plan2learn.dk)). Hence, it is directly implementable if a decision is made to put the HCAT approach into routine service in the future.
Further, the raters participated in an interactive coding workshop. At the workshop, model complaint cases were discussed with experienced raters who have obtained certification through the English online education, have extensive coding experience and attended an earlier reliability study.14 After attending the educational HCAT course, certified raters were given access to the administrative system. The first coding of cases was conducted under supervision, and all raters were assigned a certified coding mentor available for questions. To ensure that the coding process was kept on track, quarterly workshops were offered for raters and researchers participating in the study.
Reliability is usually conducted to test to what extent raters can perform tests in a uniform and reliable way.18 19 Though a formal reliability study had been conducted earlier,16 we still needed to monitor the coding. To evaluate the education of raters, we monitored if raters passed the test. We performed informal reliability checks in the initial phase of the coding presenting rater to model cases to see if they were able to code in alignment with a predefined list of coding items and we obtained feedback from the raters. To further evaluate the educational programme, raters were observed and periodically asked for feedback.
Selection of HCAT material for dissemination: We disseminated the HCAT findings to patient representatives and various groups of experts at meetings during the coding process (11 patient representatives, 13 health professionals (9 nurses, 4 physicians), 6 quality improvement staff (all academics) and 4 hospital management staff). Each expert group reflected different compositions of the relevant parties, typically with 4–6 participants at each meeting. At the meetings, all stakeholders participated, except at the meeting for the hospital management, which was a management-only meeting. The dissemination was conducted in a detailed report and through presentations at three meetings. We presented extracts from complaint case letters to exemplify patterns at various levels, from overall trends to specific hot spot and blind spot areas. Results of the dissemination process were evaluated by written feedback included in the report and from observations at the meetings. To elaborate on the feedback and observational data, we conducted four semistructured interviews with participation of two patient representatives and two staff members (the makeup changed between nurses, physicians and academics). We gained information on the usefulness of complaint patterns as a data source to select areas of quality improvement and used the feedback to adjust analyses and presentation of data.
Development of ‘dashboard’: We developed a prototype to illustrate the perspective of an easy-to-use ‘dashboard’, and how it could help quality improvement staff and others obtain an overview of case contents, thereby assisting further quality improvement. By selecting category, stage, severity and harm on the dashboard, departments could tailor specific complaint reports to their department (figure 3). Subsequently, results were discussed by these groups and feedback on the usefulness of the ‘dashboard’ was given. The feedback was used for adjustments of the ‘dashboard’.
Dashboard prototype. An easy-to-use overview of case contents identified by the HCAT instrument. HCAT, Healthcare Complaints Analysis Tool.
Analysis
The patterns arising from HCAT coding of complaint contents were descriptively displayed. Patterns were analysed across healthcare settings and shown at department and hospital levels.
We monitored the passing rates of the educational programme, performed informal reliability checks in the initial phase of the coding and obtained feedback from the raters.
To qualitatively analyse the feedback emerging from presentations of complaint content patterns to quality improvement staff, we used a phenomenological approach conducting semistructured interview online with open-ended questions.20 We thematised quotations from the interviews to elaborate on the dissemination process and usefulness of the coded material previewed by the stakeholders. Results were presented as a condensation of statements, combined with verbatim citations.
Ethical considerations
The highest security data management is required (health and disciplinary sanction data); REDCap and OPEN (Open Patient data Exploratory Network) facilities are reasonable.
Results
Evolution of the interventions and details of process measures
We coded 5217 complaint cases containing 11 056 complaint points into our secure database. We were able to code the complaint cases by educating four junior researchers as raters. All four raters passed the online HCAT coding test with >80% correct answers.
Coding
To ensure reliability, we had previously translated and culturally adapted the coding part of the Danish manual. The results of this work are reported elsewhere.16 In phase one, the remaining parts of the manual were translated into Danish by an academic (the background, conceptual framework and description of the English development work). Junior researchers’ feedback on the manual for coding indicated that the Danish manual was easy to use and feasible for coding. As an indicator of applicability, all entries into the database were monitored. The average time spent completing a complaint case coding was 8.5 min (95% CI 8.2 to 8.7), including reading the patient complaint letter thoroughly.
Education
The usefulness of the coding manual was confirmed at the coding workshop. During the workshop, the manual was used as a reference for consensus discussions and coding of test cases. In cases of doubt about the exact categorisation of complaints, the coding mentor mediated discussions and gave input on how to code. The results of the educational process were reflected in all four raters passing the final online test. The passing scores ranged from 82% to 96% correct answers indicating coding to be reliable. Parallel to the coding, the raters reported cases of doubt in an online logbook that was available to the mentor and project leaders. The logbook comments were accessed and discussed at the quarterly coding workshop. Some logbook comments led to changes in the database (eg, changing multiple response options to single response options or ‘radio buttons’); other comments raised questions on issues such as ‘how to code this complaint category’ or ‘there seems to be overlap between these two subcategories’ (eg, complaints about COVID-19 vaccines at the COVID-19 centre not registered in the database and the overlap between ‘lack of reaction’ and ‘neglect’). Overall, 25 cases of doubt were noted by the raters in the logbook. We handled the recorded cases by discussing until consensus or by applying small insignificant coding changes that did not affect the HCAT structure or categories.
Dissemination
Specific HCAT results were communicated to expert groups according to the group composition. The first expert group consisting of a mixture of patient representatives, clinicians and management was mostly presented with the overall results (figure 4A,B). For the group of quality improvement staff, we chose more specific results as the focal point. We presented both hot spot and blind spot areas and linked the results to specific time points of treatment (figure 4C,D). Finally, a combination of overall and specific results was chosen for the hospital management presentation.
Examples of results presented to the expert groups during the intervention process.
Feedback from participants at meetings was generally positive and constructive, although the non-blame approach of the HCAT encouraging learning from complaints and the HCAT’s distinctive complaint letter focus needed to be clearly emphasised in every presentation. Interviews with participants verified the usefulness of the displayed analyses after the presentations. The interviews revealed three overall themes important to the participants: ‘overview of complaints’, ‘learning from complaints’ and ‘listening to the patients’.
The broad opinion was that the HCAT approach gave a long-awaited overview of complaint data. An interviewee from the improvement staff said: ‘[the systematic approach] makes real good sense because it gives us a foundation for adjusting our improvement efforts’. A patient representative agreed: ‘It is good that it enables … to identify certain problem areas. Otherwise you can’t know … then it is more like … we think and we believe that we have registered’.
An important issue for all patient representatives was ‘learning from complaint cases’. One participant emphasised: ‘I hope [complaints] are used to learn from at the hospital … that they give food for thought and perhaps changes some procedures at the departments’. Quality improvement staff expressed a desire to learn from complaint cases to improve healthcare quality systematically. The HCAT approach of using patient complaint letters was generally perceived as a straightforward way of patient (or ‘user’) involvement.
Questions on ‘real mistakes versus patient-experienced mistakes’ were raised in all interviews. Similarly, clinicians repeatedly focused on whether complaints were found ‘justified’ by the authorities, leading to injury compensation or staff members receiving a reprimand. In contrast, the interviews also highlighted the importance of the patient perspective even though an objective mistake might not have occurred. Management and quality improvement staff mentioned: ‘Staff needs to realize that everything might seem all right, but that the patient might have another experience … you need to take that seriously’.
The participants considered that moving from the aggregated data back to the single case level had significant implications. Overall, the HCAT coding approach was perceived as an add-on to the traditional handling of complaint cases.
Dashboard
The dashboard was perceived as highly relevant at the presentations and by the interviewees. The interest in the dashboard increased with administrative responsibility, with the hospital management being most interested. Feedback from the meetings included, for example, displaying patterns down to department level and different access to complaint patterns depending on the organisational level. The combination of systematic coding and the ability to easily and quickly obtain an overview of the entire complaint pattern led to the hospital management suggesting a 2-year coding pilot period and further dashboard development. Until the dashboard is ready for use by the organisation, complaint patterns reports are delivered to provide an overview of the complaint patterns on department and hospital level. These reports contain customised analyses targeted to specific departments, counselling committees and hospital management.
Contextual elements and unexpected consequences
We had several experiences of clinicians reacting negatively to the fact that the coding is based on the patient complaint letter. Some clinicians revealed a contextual perception of this being ‘subjective’ and raised questions on the validity of approaching complaints in this way, stating: ‘But are they right?’ (Please see interview themes for the noteworthy focus of the clinicians on formal complaint outcomes.)
Otherwise, we did not experience unexpected consequences.
Missing data
Educated raters assessed all relevant patient-reported complaints and compensation claims from 2017 to 2021. Some cases journalised as complaint cases turned out not to contain complaints. Those cases were not coded and entered into the database. Currently, we do not have an exact overview of the number of cases not entered, but in the updated version of our database, we will have added this information to the coding and we will be able to address this in the future. All journalised cases containing a patient compliant letter were coded. No missing data were found in the coded cases.
Discussion
Summary
The study aimed to test if coding and extracting information from complaints and compensation cases was useful to enrich and inform quality improvement work. The study showed that adjustments had to be made before the approach was useful and meaningful. In the end, patients and their relatives, clinicians, quality improvement staff and hospital management all found it contributory and useful in relation to existing practice. Our study indicates that patient safety and healthcare quality concerns raised in complaint letters can be systematically assessed in relation to quality improvement work at the hospital level. The intervention process resulted in a 2-year prospective commitment from the hospital management to pilot the coding approach.
Interpretation and comparison with existing literature
Staff-reported adverse events are the most commonly used source of information to detect unsafe care, but other sources such as trigger tools, observation and charts are suggested to complement adverse events.21 Our study suggests that patient-reported information from complaints contributes to useful knowledge. This is in line with a recent study by van Deal indicating that combining patient complaints and staff reports provides greater insight into unsafe care.22 In connection, it has been previously found that patients deliver valid and supplementary data to unsafe hospital care.23
Patient involvement is a main focus of the OUH (https://en.ouh.dk/about-ouh/patient-first/), and for healthcare in general,24 and various patient involvement methods have been proposed,25 26 to improve the quality of healthcare. However, there has been limited focus on systematic use of patient complaints as a method.10 Systematic coding of patient complaints can be considered direct patient involvement in quality improvement work, which is perceived useful in this study and prior studies.
Implications
OUH has over the last half a decade implemented a lean-inspired quality improvement programme (South Danish Improvement Model (SDIM)) with help from the Virginia Mason Institute.27 28 To improve quality and reduce waste, OUH has initiated workshops focusing on patient value. Often, the value stream workshops have been empirically anchored or based on clinician or management input. In the future, systematic HCAT coding can potentially serve as a supplementary source of information for selecting workshops. OUH quality team members can review cases relating to the over-represented sub-categories to allow for familiarisation with the subject matter behind a specific case. The value of data obtained through HCAT analyses can then be optimised by combining discussions with clinical experts in the field (from involved departments), quality improvement staff, SDIM managers and patient representatives.
Further, if systematic use of the HCAT becomes an integrated part of the quality improvement work, continuous coding of complaint cases may offer long-term benefits. If this systematic approach should be implemented into QI of clinical practice consideration should be given not to overburden an already time-poor system. If implemented, the approach will require sufficient employee time to analyse complaint patterns and develop QI initiatives at different levels of the organisation. The additional coding procedures will only marginally increase the workload of filing complaint cases. Relevant complaint analysis could be added to existing handling of complaint cases in the QI department. Thus, application of artificial intelligence technology in automated monitoring of quality development could be considered.
Another implication of the HCAT approach as a supplementary data source is the linkage to other review systems centred on healthcare events. Clinician-reported events in the adverse event system causing death or being highly severe undergo further investigation. In the future HCAT information on ‘catastrophic harm’ might qualify the learning from these adverse events by combining the patient and the clinician perspective.
Limitations
This study has some limitations. We retrospectively coded the complaints from 2017 to 2020, which could mean that some patterns were altered. However, the overall number of complaints remained relatively constant, and no marked organisational changes occurred, indicating no dramatic changes in complaint patterns.
Delays in coded cases are caused by retrospective coding—the very nature of complaints has an embedded delay. Sometimes, the complaints were filed a while after the actual incident, which caused a delay in itself. However, we previously found the delay from incident to filing of a complaint to be no more than three and a half months on average.29 We used the complaint letter filed by the patient and did not consider whether the complaint was justified or not. Therefore, we did not have to wait for the outcome of the complaint, which shortened the time gap. To introduce more accurate time patterns, it is recommended to code complaint cases prospectively when filed by patients.
A further limitation for HCAT analyses to guide QI in the future is that a large sample is needed to obtain sufficient signal to monitor if QI initiatives actually have changed complaint patterns. Insufficient signal has to be considered for smaller departments and smaller samples. It strengthens the HCAT approach that it is possible to move from the aggregated to single case level. Thus, it is possible to select single cases to learn from in cases with insufficient signal.
During the Danish translation and cross-cultural adaption of the HCAT approach,16 we tested and found substantial reliability for the seven main categories of the Danish edition. We translated and used the subcategories from the original HCAT manual in the process. We have planned an upcoming study to test and qualify the 38 sub-categories from the HCATdk.
Conclusion
We explored the usefulness of a standardised instrument for extracting information from complaints and compensation cases to enrich and inform quality improvement. Through development process, including several adjustments, all relevant stakeholders found the systematic approach useful for quality improvement. The management level eventually found the approach to be promising and decided to pilot the prospective coding of complaint cases for a 2-year period to test the applicability and translation into clinical practice.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
According to Danish law, register-based research requires no approval from research ethics committees. However, it must satisfy general data protection regulation (Directive 95/46/EC; 2016/679 and DK Act 502), and permission for case review was obtained from OUH
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @Lars.Morsoe_dk, @BieBogh
Contributors LM, SFB, SBB and SW have designed and planned the project with advice from HR and PSG. LM, SFB and SBB wrote the protocol and participated in all phases of the project. SW, HR and PSG participated in the data collection. All authors have contributed to interpreting the results. LM has drafted the first edition of the manuscript. All authors have revised and approved the final draft of the manuscript. As the guarantor LM accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish
Funding The present study was funded by the Odense University Hospital development fund (€13.000). Grant number: 2019/84-A3917.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.