Quality indicators development and prioritisation for emergency medical call centres: a stakeholder consensus

Indicator development phase

An observational step was conducted from June 2018 to September 2018 in a French emergency department at the Poitiers University Hospital by the public health team. Two public health residents (LA, JG) came to the CRRA as observers for 2 days to better understand the functions of the SAMU and to identify the risks related to the practice of regulation. Observations consisted of a tour of the premises, presentation of the operation of the SAMU and of the CRRA team, and monitoring of regulation activity through the practice of double listening calls with ARMs and MRs.

In parallel, a rapid international review of literature22 was carried out to identify all existing quality and activity indicators. A rapid review is a form of knowledge synthesis in which components of the systematic review process known as ‘evidence summaries’ are streamlined to produce information in a short period of time. Although numerous rapid reviews have been produced, their varied methodologies can diversely impact research results.23 A rapid review protocol was compiled based on the ‘framework of rapid review methods’.24

The databases queried were PubMed, ScienceDirect, Cochrane Library and Cairn. A first exploratory phase of these databases helped refine the keywords. The search equations are presented in table 1.

The following were the inclusion criteria:

• Studies published in English or French.

• Studies published since 1 January 1999 (in France, the development of quality procedures follows the ordinance of 24 April 1996 establishing an accreditation procedure implemented since 199925).

• All types of studies.

The following were the exclusion criteria:

• Title and abstract published in language other than English and French.

• Title and abstract off subject.

• Irrelevant population or irrelevant outcomes in full text, that is, not meeting the objective (identify quality indicators).

Articles were selected first after reading the title and the abstract by two reviewers (LA, JG), then the full text by one reviewer (LA). In case of disagreement on a selection, an agreement was achieved after discussion between the two reviewers.

Other sources consisted of recommendations and repositories published by various French emergency medicine societies and associations, as well as by the French National Authority for Health (Haute Autorité de Santé, HAS), and the General Inspection of the Administration and General Inspection of Social Affairs report.26

At the end of this preliminary step, all the recommendations and indicators concerning the SAMU regulation were listed in a table using the framework with structure, process and outcome indicators.27 Some indicators were developed by the public health team following the observational phase and after discussion with CRRA stakeholders and the French Society of Emergency Medicine (Société Française de Médecine d’Urgence, SFMU). Some indicators from the recommendations were adapted (reformulated or clarified) for better understanding and greater consistency.

Indicator prioritisation phase

An interventional step was conducted from April 2019 to September 2019 in the largest French region (the Nouvelle-Aquitaine region with 6 million inhabitants, or 10% of the French population) by the public health team of Poitiers University Hospital.

The listed quality indicators were prioritised using the HAS validated consensus method ‘Recommendations by formalized consensus’ (December 2010, updated March 2015).28 This approach is an adapted method from the Delphi one.29 Briefly, the HAS consensus method consists of an individual rating in two rounds, separated by an intermediate meeting. The method involves three groups: one steering group, one rating group and one reading group. A questionnaire is distributed with a discrete numerical scale graduated from 1 to 9 next to each item: a value of 1 means that the respondent considers the proposal totally inappropriate (or not indicated or not acceptable); a value of 9 means that the respondent considers the proposal totally appropriate (or indicated or acceptable); values of 2–8 reflect possible intermediate situations; and a value of 5 corresponds to respondent indecision.

Patient and public involvement

In this study, the steering group was made up of the medical manager of SAMU 86 (HD-F), a doctor from SAMU 86 involved in the quality process (JB), the head of the public health department of the University Hospital of Poitiers and coordinator of care risk management (VM), and a public health resident from Poitiers University Hospital (LA). The rating group consisted of volunteer ARMs and MRs from the 13 SAMUs of the Nouvelle-Aquitaine region. The sample consisted of one MR and one referral ARM, with at least 15 years of experience, for each CRRA. The reading group was extended to all CRRA agents in the region.

First, the approach was presented to the 13 SAMUs of the region in order to obtain their consent and their commitment to participate in the entire prioritisation process.

The questionnaire was drawn up from the exhaustive table of indicators, which was built from the development phase (observational step and rapid review of literature). The design was neat in order to facilitate participant acceptance and increase response rate. To preserve anonymity, the questionnaire was sent to the medical manager of each SAMU, who then forwarded it to the rating MRs and ARMs.

The rating was done individually via an online survey site, SurveyMonkey, and participants were given 2 weeks to answer the survey. To be sent, the questionnaire had to be completed in full, but the filling could be done in several times. Responses were anonymous.

The rating criterion used was the priority given to the indicator. Each item had to be rated on a scale of 1 to 9: 1 if the item was considered non-priority and 9 if it was considered absolute priority. A final item allowed participants to comment or suggest a new indicator. The results obtained were analysed according to the HAS method. The proposal was deemed priority when the median value was ≥7 and all ratings ≥5; non-priority when the median value was ≤3.5 and all ratings ≤5; and uncertain when the median value was between 4 and 6.5 (indecision) or when there was no consensus between the rating group members (all other situations). Proposals with strong (ratings in the range 7–9) or relative (ratings in the range 5–9) agreement for prioritisation were accepted as such, without discussion in the meeting or submission to the second round of rating.

At the end of the first round, a results restitution was planned, as well as a discussion with the rating group about items that did not reach consensus. These were submitted for a second round of rating, but only to members who had attended the discussion, according to the HAS validated method.28 The response and analysis methods remained unchanged.

At the end, all results as well as a process summary were returned to all SAMUs in the region via a newsletter sent by email. A structured descriptive sheet30 was drawn up for each prioritised indicator, which contains the title, objective, complete definition of its numerator and denominator if applicable, data collection modalities, details on the interpretation of the results and characteristics. Finally an operational dashboard was established based on the indicators deemed to be priorities to be able to give visibility of quality within the CRRAs through the results of each indicator: "yes", "no", "partially". This dashboard is in the form of an Excel document containing one tab per theme to be covered, then one line per indicator and its result opposite, with the possibility to directly access the descriptive sheet for each item. Every year it will be completed to follow the evolution of the quality approach within the CRRAs. This entire step was carried out by the steering group.

Indicator development phase

The rapid review of literature included 33 studies in our work, from the initial 414 that resulted from various searches (figure 1).

Request permission

The preliminary phase of observation, discussion with stakeholders and literature review identified 360 quality indicators concerning SAMU regulation.

All were classified in a table structured into five themes according to the field concerned: material resources, human resources, quality approach, call handling and postcall support (table 2). These five subjects were chosen according to the themes found in all the indicators listed, as well as by relying on the repositories of the association ‘SAMU-emergencies of France’3 5; we can thus find what allows a CRRA to function (human and material resources), the indicators which optimise this operation and the safety of the CRRA (quality approach), as well as the indicators directly linked to the activity of the CRRA (performance about calls and support for victims).

Indicator prioritisation phase

After presenting the approach and the prioritisation process by sending an electronic document to the Nouvelle-Aquitaine SAMUs, 13 MRs and 13 ARMs agreed to participate (table 3).

The first round of rating took place between 30 April and 23 May 2019, lasting approximately 3 weeks. The response rate was 77%; only one SAMU was not represented among the respondents (table 3).

At the end of the first round, 18.9% of the items were deemed priority with strong agreement, 33.6% priority with relative agreement, 15.6% uncertain due to indecision and 31.9% uncertain due to lack of consensus (table 4; detailed data not shown).

Before the second round, some items were reformulated by the steering group due to ambiguous or unclear formulation, according to the HAS method.28 Moreover three new items were integrated following the proposals of the participants.

The 171 items which did not obtain consensus in the first round as well as the 3 new items were submitted for a new rating to the 20 respondents from the first round.

The second round of rating took place between 25 June and 23 September 2019, for around 13 weeks. The response rate was 75%; nine SAMUs were represented (table 3). At the end of the second round, 1.1% of the items were deemed priority with strong agreement, 20.1% priority with relative agreement, 31.6% uncertain due to indecision and 47.1% uncertain due to lack of consensus (table 4; detailed data not shown).

Finally, out of the 26 participants included in the consensus approach, 15 participated in the whole process, representing a response rate of 58%. All in all, 70 items (19.3%) were considered priority with strong agreement, 156 (42.9%) priority with relative agreement, 55 (15.2%) uncertain due to indecision and 82 (22.6%) uncertain due to absent consensus (table 4).

At the end of the prioritisation process, a newsletter was returned to the 13 SAMUs presenting the results and prospects. A dashboard of indicators deemed high priority has been established (online supplemental table 1). Seventy descriptive sheets were written. The reading group contributed by simplifying some of them (structural indicators), reformulating some objectives and redrawing some items to improve the feasibility of data collection.

Main results

To our knowledge, this is the first study on the development and prioritisation of quality indicators for emergency medical call centres. There were 360 indicators initially listed, thereby constituting the first structured and exhaustive list in France for prehospital emergency regulation.

The large number of indicators shows the complexity of medical regulation activity, its many risks at all levels (material and human resources, call management and quality approach) and the need to prevent them in order to improve quality of care. Such a large quantity of indicators cannot be realistically accepted, implemented and evaluated, hence the need to prioritise them for progressive implementation in CRRAs. Finally, 70 indicators were deemed to be high priority with strong agreement. This significant reduction in the number of indicators allowed creation of an operational dashboard and the writing of an indicator descriptive sheet for each.

The 70 prioritised indicators concern conventions, procedures and protocols, methods for effector triggering, exceptional health situations, equipment, call recording, training, the roles and number of concerned professionals, securing their work, relations with firefighters, regulation file keeping and call transfers. These emerging themes are coherent with the current context and stakeholder concerns (ARMs and MRs) and illustrate the need to secure regulation activity. It is interesting to note that no quantified performance indicator was found among the 70 highest priority indicators.

Place in the literature

In contrast, many works have been done about performance, in particular with regard to cardiopulmonary arrests,14–17 with the aim of showing how to improve the efficiency and accuracy of the response provided by emergency call centres.31 Agent occupation rate seemed to be the most important factor contributing to service quality as well as all quantitative indicators.

However, quality does not stop with performance. On the contrary, quality and safety of the structure, resources and processes can improve performance, as explained by Giroud in his publication.32 However, few indicators exist in the literature to assess these dimensions. In our study, the diversity of listed indicators and the choice of prioritised indicators confirm this need to consider regulation activity in its entirety, as well as human factors, in view of achieving more secure activity. Moreover, the notion of human factors has appeared in recent literature: for example, Rawshani et al33 concluded that there was a higher risk of not prioritising life-threatening patients during lunchtime. Work by Hamelin and Arzalier34 goes in the same direction; analysis of complaint files showed that considering human factors and fatigue is essential and underlines the importance of reporting on internal dysfunctions. The impact of feedback on performance was explored by Clawson et al,35 which showed that regular and objective information to emergency medical dispatchers about their performance significantly improved their rigour. Professional practice evaluation, feedback and adverse event analysis must be encouraged in order to achieve a positive culture of error.36 37 The role of human factors was also studied in telephone exchanges by Riou et al,38 who analysed the words used to initiate cardiopulmonary resuscitation (CPR) assisted by the dispatcher and their association with caller agreement. The quality of telephone communication should be formalised and evaluated.39 40

Securing the work of agents seems to be a major concern at present, as confirmed in our study by the strong consensus obtained for this proposal. In this complete item, well-being at work is taken into consideration. Indeed, a link has been demonstrated between job satisfaction and performance41; here again, human factors are important.

Thus our study seems to fill a gap in the exhaustiveness of the risk factors associated with CRRA activity, taking into consideration the human factor.

Strengths and weaknesses

This work was carried out in connection with the French Society of Emergency Medicine and was based on a validated method considering peer opinion, thereby strengthening its validity. The search for consensus has confirmed the need to formalise a quality approach as well as the relevance of the indicators. We initially obtained strong adhesion following presentation of the approach, and second the results gave no non-priority indicators and an average of >5 out of 9 (priority) for 96.7% of them. All in all 62.3% of the indicators obtained a strong or relative prioritisation consensus.

The first round of rating required two general reminders by email and a 1-week extension to obtain the maximum number of responses. The participation rate was suitable (77%). For a better questionnaire acceptability and an optimal response rate, the choice was made to adapt the results analysis of the first round, by accepting as such the proposals that have obtained a strong and relative agreement. By resubmitting to the rating only uncertain proposals (for which there was indecision or absence of consensus), the second round would be less heavy.

Furthermore, the possibility for the participants to express themselves freely at the end of the questionnaire was important for proposal integration or modification before the second round.

The second phase took place during the summer period, explaining the two general reminders and the two individual reminders, as well as the time limit for closing the 13-week survey. It nevertheless obtained a participation rate of 75%. Out of the 174 items resubmitted for rating, 37 finally obtained agreement. We can think that when each participant knows the median obtained by the group for each item, it becomes possible for some participants to review their judgement and reposition themselves accordingly.

Finally, only 58% of the participants went through the prioritisation process. However, Nouvelle-Aquitaine is the largest region in France, with 12 departments and 13 SAMUs, which allowed for 15 representative peers to participate in the whole process: doctors and ARMs, who are the main participants in regulation. The length of questionnaire is probably largely responsible, but it was necessary to be as exhaustive as possible from the start in order to be relevant in reduction by prioritisation. Because of this length constraint, it was possible to respond to the questionnaire in several phases and an effort was made on its presentation. Otherwise, the duration of the process (more than 4 months in total) as well as the vacation period during which it took place certainly favoured the number of ‘lost to follow-up’.

It was not possible to hold a meeting with the rating group between the two rounds, which can constitute a potential bias. The aim of the meeting was to return the results of the first round and to discuss the proposals deemed uncertain. Given constraints in each participant’s schedule, it seemed preferable to return the results by email so that everyone could benefit from the same feedback. To do this, each participant of the first round received an overall summary of the results of the analysis as well as a file with their own answers. The median obtained by the group was shown below each item on the second questionnaire so that each participant could situate himself in relation to the rest of the group during the second round, according to the HAS method. In the absence of discussion, comments and proposals brought up during the first round were considered and partly integrated into the questionnaire, after consultation with the steering group.

Implications

This work will facilitate the gradual integration of a quality approach within CRRAs. From now on, each SAMU in the region will be able to assess the quality and safety of its regulation activity. Using the dashboard, or a computerised mapping tool, will enable organisation of a periodic internal audit. The detailed descriptive sheets constitute an operational tool to also benchmark with other CRRAs in order to help each other out and improve everyone’s practice. Ultimately, once the installation is successful, the indicators that were prioritised with relative agreement could gradually be integrated until exhaustive implementation of the 363 indicators is achieved in a few years.

Moreover, this dashboard could be integrated into the quality account of regional establishments as well as in certification procedures. These indicators could also be integrated into the indicators for improving quality and safety of care (IPAQSS).42

Finally, this work could facilitate CRRA inclusion in the IFAQ procedure (Financial Incentive to Improve Quality).43

Areas for improvement

The search for consensus was carried out on a regional scale, but this approach could be extrapolated to national and international scale to increase the power of the study, as well as to standardise practices in terms of quality and safety and to do benchmarking. Moreover, we could integrate other actors of medical emergency, such as paramedics, nurses and general practitioners.

Initiatives are already emerging to strengthen medical regulation quality within CRRAs. Training with diploma was set up in 2019 and is henceforth compulsory to exercise the profession of medical regulation assistant.44 Telemedicine is also growing, with CPR assisted by phone or smartphone, remote video consultations to assess severity of patient illness or remove doubts, etc; these are valuable aids for regulation.

Furthermore, the Information System-SAMU (SI-SAMU) programme is being deployed in French territories and aims to make the information system and telecommunications means used by emergency medical services more reliable and secure. This centralised and national portal is intended to be used by all SAMUs.45