You are here

  • Home
  • Archive
  • Volume 23, Issue 1
  • Effects of a team-based assessment and intervention on patient safety culture in general practice: an open randomised controlled trial

Article Text

Article menu
Original research
Effects of a team-based assessment and intervention on patient safety culture in general practice: an open randomised controlled trial
  1. B Hoffmann1,
  2. V Müller1,
  3. J Rochon2,
  4. M Gondan2,
  5. B Müller1,
  6. Z Albay1,
  7. K Weppler1,
  8. M Leifermann1,
  9. C Mießner1,
  10. C Güthlin1,
  11. D Parker3,
  12. G Hofinger4,
  13. F M Gerlach1
  1. 1Institute of General Practice, Goethe University Frankfurt am Main, Frankfurt, Germany
  2. 2Institute of Medical Biometry and Informatics, University of Heidelberg, Heidelberg, Germany
  3. 3Safety Culture Associates Limited, Leeds, UK
  4. 4Team HF Hofinger Forschung Beratung Training, Remseck, Germany
  1. Correspondence to Dr Barbara Hoffmann, Institute of General Practice, Goethe University, Theodor-Stern-Kai 7, Frankfurt am Main D-60590, Germany; Hoffmann{at}allgemeinmedizin.uni-frankfurt.de

Abstract

Background The measurement of safety culture in healthcare is generally regarded as a first step towards improvement. Based on a self-assessment of safety culture, the Frankfurt Patient Safety Matrix (FraTrix) aims to enable healthcare teams to improve safety culture in their organisations. In this study we assessed the effects of FraTrix on safety culture in general practice.

Methods We conducted an open randomised controlled trial in 60 general practices. FraTrix was applied over a period of 9 months during three facilitated team sessions in intervention practices. At baseline and after 12 months, scores were allocated for safety culture as expressed in practice structure and processes (indicators), in safety climate and in patient safety incident reporting. The primary outcome was the indicator error management.

Results During the team sessions, practice teams reflected on their safety culture and decided on about 10 actions per practice to improve it. After 12 months, no significant differences were found between intervention and control groups in terms of error management (competing probability=0.48, 95% CI 0.34 to 0.63, p=0.823), 11 further patient safety culture indicators and safety climate scales. Intervention practices showed better reporting of patient safety incidents, reflected in a higher number of incident reports (mean (SD) 4.85 (4.94) vs 3.10 (5.42), p=0.045) and incident reports of higher quality (scoring 2.27 (1.93) vs 1.49 (1.67), p=0.038) than control practices.

Conclusions Applied as a team-based instrument to assess safety culture, FraTrix did not lead to measurable improvements in error management. Comparable studies with more positive results had less robust study designs. In future research, validated combined methods to measure safety culture will be required. In addition, more attention should be paid to evaluation of process parameters. Implemented actions and incident reporting may be more appropriate target endpoints.

Trial registration German Clinical Trials Register (Deutsches Register Klinischer Studien, DRKS) No. DRKS00000145

https://doi.org/10.1136/bmjqs-2013-001899

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Safety culture has an impact on how organisations deal with medical errors and how patient safety measures are implemented in daily practice.1 Safety culture is commonly defined as ‘the product of individual and group values, attitudes, competencies, and patterns of behaviour that determine the commitment to and the style and proficiency of an organisation's health and safety programme’.2

    There is an ongoing debate on how to improve an organisation's safety culture and safety performance.3 Assessments are carried out via self-administered questionnaires that measure healthcare workers’ perceptions.1 Self-assessment and interventions to improve safety culture are combined in an approach based on a typology of organisational culture.4 This method is embodied in the Manchester Patient Safety Framework (MaPSaF).5 MaPSaF is a matrix of nine dimensions which can be assigned to five different levels of culture. It focuses mainly on the identification and investigation of patient safety incidents and on learning from them, while other dimensions address patient safety as reflected in, for example, communication and teamwork. Primarily, MaPSaF encourages reflection on the safety culture of one's practice team6 and the development of interventions to change its culture.5 MaPSaF was originally developed to be used in primary care in the UK and was later adapted for use in different sectors7 and countries.8 ,9 In a recent survey, MaPSaF has been identified as the most frequently used safety culture instrument in the English National Health Service in acute care trusts as well as in primary care.10

    It is argued that safety culture is affected by organisational changes such as leadership or newly introduced systems and processes.11 In order to improve safety culture, other groups have assessed safety culture in healthcare organisations and followed up the evaluation with an intervention comparable to MaPSaF. This has had promising effects on safety culture in secondary care.12 ,13 However, these studies applied pre-post designs without randomly selected control groups. No research yet exists on the effects of MaPSaF. In a pilot study we translated and adapted the primary care version of MaPSaF into German (the Frankfurt Patient Safety Matrix, FraTrix), and results from the pilot study showed high acceptability and applicability of the instrument. In addition, after applying the matrix in one team session, 15 of the 19 practice teams in the pilot study decided on actions to improve processes affecting patient safety,14 and hence encouraged us to conduct a proper intervention study that looks at outcomes relevant to safety processes and culture.

    We hypothesised that, by encouraging teams to think about their safety culture, FraTrix would lead to the implementation of specific measures to improve patient safety and safety culture. We thus carried out a randomised controlled trial of FraTrix to investigate whether it can improve safety culture —that is, whether the teams of general practices improved (1) their error management and other structures and processes supporting patient safety; (2) their general safety climate; and (3) their patient safety incident reporting.

    Methods

    Design and participants

    We conducted an open randomised controlled parallel-group rater-blinded trial in general practices in the Hesse region of Germany. As FraTrix targets team processes, a minimum team size of three healthcare professionals was required.

    Recruitment took place between May and September 2009. All registered general practices (statutory health insurance physicians specialised either in family medicine or general internal medicine) in southern Hesse were invited to participate. Practices that demonstrated interest were contacted by telephone to check eligibility and were given further details on the trial. After obtaining consent from the entire practice team (physicians and healthcare assistants (HCA)), practices were randomly assigned to intervention and control groups. Randomisation was stratified by practice model (single-handed or group practice). The course of the recruitment procedure and the study is shown in figure 1.

    Figure 1
    Figure 1

    CONSORT flow diagram.

    Practice teams participated voluntarily. As patients did not take part in the trial and patient data (when used) were anonymised, the institutional review board of the Medical Faculty of the Goethe University Frankfurt waived further ethical review.

    Intervention and control treatment

    The intervention began with an introductory team session during which the course of the intervention was described and instructions on the use of FraTrix provided. This first session was followed by three facilitated team sessions that used FraTrix. In the control group, we offered every practice team member the opportunity to attend a short seminar on patient safety and error management at the Institute of General Practice in Frankfurt. Both types of intervention are described in more detail in box 1.

    Box 1

    Detailed description of the interventions

    Intervention group The Frankfurt Patient Safety Matrix consists of two elements:

    1. Team sessions under the guidance of an external facilitator using

    2. A matrix of five safety culture grades (from ‘generative’ as the most mature, through ‘proactive’, ‘bureaucratic’ and ‘reactive’ to ‘dismissive’ as the least mature grade) and safety dimensions.

    Team members learned about the matrix during the introductory team session and by reading the FraTrix brochure which includes instructions for the use of FraTrix and a detailed description of the matrix. The nine dimensions were

    • Overall commitment to quality

    • Priority given to patient safety

    • Perception of critical incidents and their causes

    • Analysis of critical incidents

    • Learning from critical incidents

    • Communication as it relates to patient safety

    • Personnel management as it relates to patient safety

    • Staff education and training as they relate to patient safety

    • Team work as it relates to patient safety.

    No later than 4 weeks after the introductory meeting, the first FraTrix team session was scheduled and followed by second and third sessions after 3 and 6 months, respectively. Follow-up sessions were always attended by the same facilitator. Team sessions generally took place during lunch breaks or after work.

    Each team session lasted between 60 and 90 min. Before each team session, every team member worked through the brochure independently of his or her colleagues and without discussing any of the issues. At the beginning of the team session, all individual assessments of each dimension were reported and the results were marked on a chart (see online supplementary appendix). Thus, every team member's assessment was seen by the whole team. The practice team members then decided on the three dimensions they wanted to deal with first. In the following discussion, the team members reflected on the strengths and weaknesses of their safety culture with regard to the chosen three dimensions. The facilitator supported the team to work with the brochure, guided the discussion along the matrix and helped to achieve an action plan by the end of the team session.

    The facilitator subsequently recorded the results of the team assessment and the agreed action plan. Furthermore, the team received a form on which to keep records of the planned actions (ie, improvement protocol) for further use. At the beginning of the second and the third sessions, the facilitator asked the teams whether they had achieved the aims of the previous action plans. The remaining dimensions were topics for discussion during team sessions 2 and 3. After 12 months, improvement protocols were copied for data collection during a final practice visit.

    To ensure the facilitation was standardised and of high quality, both facilitators agreed on a facilitation guide. Every third team session was audiotaped and the recordings were used by the other facilitator to ensure that the facilitation guide had been adhered to. Both facilitators were well trained and experienced in facilitating FraTrix sessions as a result of their participation in the pilot study14 (BH, VM). BH is both a physician and a nurse, as well as a health scientist with substantial expertise in patient safety. VM is a healthcare assistant, a study nurse and a family counsellor.

    Control group

    Since we considered the attention given to the FraTrix group as an intervention, we offered every practice team member in the control group the opportunity to attend a short seminar on patient safety and error management at the Institute in Frankfurt. During a lecture of 90 min, the participants were instructed in the basics of patient safety and the main elements of error management. In addition, control practices were contacted twice by phone about 5 and between 8 and 9 months after the baseline assessment. The calls were concerned with issues of error management and gave the practice teams the chance to ask questions about patient safety.

    Measurements

    Patient safety culture

    In safety culture research, a combination of different measures is recommended.15 We therefore assessed safety culture on different levels and by different mean, using the three-layer model developed by Schein16 (artefacts, espoused values, core assumptions; see figure 2).

    Figure 2
    Figure 2

    Model of safety culture measurement based on the three-layer model of organisational culture.

    Patient safety culture indicators

    On the one hand we adapted 12 established quality indicators for general practice17 ,18 that reflect structures and processes relating to patient safety (for more details see online supplementary appendix 2, part 3) in order to investigate ‘artefacts’ or behaviours of the practice team as a whole—what the team does and how with regard to patient safety.

    Patient safety incident reporting

    On the other hand, the measurement of ‘artefacts’ was complemented by the evaluation of patient safety incident reporting in the practice. For this purpose we assessed the number and quality of incident reports. We considered incident reports valid if they reported on patient safety incidents on the basis of a definition adapted from the National Patient Safety Agency.19 We assessed their quality according to a method adapted from an instrument developed for the assessment of Significant Event Analysis in general practice20 (for details see online supplementary appendix 2, part 4).

    Patient safety climate

    Safety climate reflects staff perceptions of the priority of safety.2 To evaluate these ‘espoused values’ and attitudes of the team, we measured safety climate with the Frankfurt Patient Safety Climate Questionnaire for General Practices (FraSiK), as developed and validated by our research group.21 The instrument consists of 44 items with moderate to good internal consistency that circumscribe nine scales. One scale is addressed only to doctors (‘quality and safety of medical processes’) and one only to employees (‘staff perception of management’). Responses to items are given on a 5-point Likert scale ranging from 1 (‘strongly disagree’ or ‘never’) to 5 (‘strongly agree’ or ‘always’).

    Primary outcome

    The primary outcome was assessed 12 months after randomisation. We chose the patient safety culture indicators (PSCI) error management as the primary outcome because the management of patient safety incidents is a core aspect in patient safety and of FraTrix: it is covered by three of the nine dimensions—namely, ‘perception of critical incidents and their causes’, ‘analysis of critical incidents’ and ‘learning from critical incidents’. The indicator is derived from the European Practice Assessment (EPA) indicator system, has been found to be valid and reliable18 and consists of six criteria that were assessed with a total of 13 self-reported items (table 2).

    Secondary outcomes

    Secondary outcomes were 11 additional indicators of patient safety culture, data on patient safety climate (FraSik scale scores) and the volume and quality of incident reporting (see online supplementary appendix).

    Course of the intervention

    We assessed whether the intervention took place as planned. Facilitators made records of the team sessions (duration and participants in the session, individual assessments displayed on the chart, dimensions of FraTrix that were discussed including important aspects of the discussion, and content of the action plan). Furthermore, we assessed the quality of the improvement protocols (see box 1) by evaluating whether the action and its goal were described, which factors contributing towards errors22 were addressed, other dimensions of patient safety practices23 and whether the action was implemented and the goal achieved.

    Sample size

    The sample size was calculated using the primary endpoint —that is, error management at 12 months. The calculation was based on data from 302 practices from the EPA.18 A total of 45 practices would be needed to detect the proposed effect size for competing probability (CP)22 ,23 of 0.73 with a statistical power of 80% and at a two-sided 5% significance level (for details see online supplementary appendix 2, part 2). We adjusted the total sample size to 60 practices to account for possible withdrawals.

    Data management and analysis

    Since the indicators have different numbers of criteria, we standardised the scores by dividing the sum of fulfilled criteria by the total number of criteria per indicator. For the FraSiK questionnaire, scale scores on an individual level were calculated as the mean of the scale's items. Safety climate scores on a practice level were calculated as the mean of the respective team members’ scores. The quality of incident reports was independently evaluated by two blinded raters and consensus sought in case of disagreement. The duration of the trial from baseline to final data collection was 12 months. No outcome measure was changed during the trial.

    Primary analysis followed the intention-to-treat principle, with baseline observation carried forward for missing values. Continuous data were summarised using means and SD, and categorical data with frequency counts and percentages. The Wilcoxon–Mann–Whitney (WMW) test was used to test for differences in error management between groups at 12 months. The effect of the intervention on the primary outcome is expressed as CP from the WMW test statistic with a corresponding 95% CI.24 ,25 Sensitivity analyses, including a non-parametric repeated measures analysis of variance (ANOVA), were conducted (see online supplementary appendix 2, part 2). All tests were two-sided with a significance level of 0.05 and were not adjusted for multiple comparisons.

    Results

    One hundred and sixty-eight general practices were assessed for eligibility; 14 practices (8.3%) did not meet the inclusion criteria, 89 teams (53.0%) declined to participate and 65 practice teams (38.7%) agreed to take part in the trial. These practices were randomly allocated to either the intervention or control group. Since a sample size of 60 practices was required, the five practices that last consented to participation were transferred to a waiting list following randomisation. In the end, 60 practices took part in the trial, comprising 28 in the intervention group and 32 in the control group (figure 1).

    Participating practices

    Almost one-third of practices were group practices, the remaining 41 practices were single-handed (see table 1). Practices were located in small towns (41.7%), mid-sized towns (30.0%) and cities (26.7%). Only one practice was located in a rural community. Compared with non-participating practices, we found no differences in location, practice model, team size and implemented quality management system. In addition, practice characteristics (practice model, location) did not differ from the entire set of general practices in the area. However, participating physicians were, on average, younger (50 years) than general practitioners in the state of Hesse (53 years).26 The average team size was comparable to practices participating in a recent survey on safety culture in general practices in Germany.27

    View this table:
    Table 1

    Summary of characteristics of practices and healthcare professionals

    Three practices were lost to follow-up after baseline data collection that took place between August and November 2009. One intervention practice team withdrew from the trial immediately after baseline data collection because the HCAs viewed the interview on PSCI as intrusive. One control group practice was closed and the final data collection (from August to December 2010) could not be conducted in another due to a lack of cooperation.

    Course of the intervention

    The initial meeting and three FraTrix team sessions took place in each of the final 27 intervention practices. On average, FraTrix team sessions lasted a mean (SD) of 86.2 (20.7) min and the entire team was present at most sessions (in only 7.2% of sessions was one or more team member absent).

    With regard to the safety culture of their practices, healthcare professionals mostly assessed them as ‘proactive’ (the second most mature grade), followed by ‘bureaucratic’ and ‘generative’. The number of ‘dismissive’ or ‘reactive’ assessments was low (see online supplementary appendix). With regard to the aim of the team sessions, only 10 sessions ended without any action plan (12.3%). At the end of the other sessions, 265 actions had been planned (9.8 actions per practice), of which 124 actions (46.8%) were documented and followed up by the team. Of these improvement protocols, 97 actions could be rated with regard to their quality; 90 were rated as appropriate or fairly appropriate for improving patient safety. Most of the actions addressed organisational or environmental factors that may contribute to errors in patient care (93.8%) and/or individual characteristics of the healthcare professionals (48.5%) and only a minority addressed patient and medication or equipment factors. Nearly one-quarter of the actions consisted of one-time structural changes in the practice whereas three-quarters depended on repeated activities.

    About 50% of these actions were fully implemented, and the practice teams judged the goal of the action to have been achieved in 32 cases —that is, about one action per intervention practice. Most actions targeted team communication, quality management in general, organisation of practice, education and training, error management and communication with patients.

    In the control group, representatives from 16 practices (50%) participated in the seminar on patient safety and error management. The participants consisted of 17 physicians (72.2% of the physicians in the attending practices) and 38 HCAs (66.4%). As planned, every control practice was contacted twice by telephone during the intervention phase.

    Primary outcome

    At baseline, control group practices scored higher in terms of PSCI error management than intervention group practices (table 2). After 12 months the mean (SD) score had increased in both groups and was 0.68 (0.14) in the control group and 0.66 (0.18) in the intervention group. Thus, no significant difference between the intervention and control groups was observed in the primary outcome (CP=0.48, 95% CI 0.34 to 0.63, p=0.823). A non-parametric repeated measures ANOVA showed that the group-by-time interaction was not statistically significant (p=0.262).

    View this table:
    Table 2

    Primary outcome patient safety culture indicators (PSCI) error management

    Secondary outcomes

    Intervention and control practices did not show any statistically significant difference in scores of PSCI at 12 months (see online supplementary appendix 2, part 5). Sensitivity analyses did not reveal significantly different intervention effects over time.

    Safety climate

    Safety climate at baseline was perceived by all teams to be positive for eight of the nine scales. The majority of scales did not change from baseline to the end of the trial and, again, no significant differences were found between intervention and control groups at 12 months (table 3).

    View this table:
    Table 3

    Patient safety climate scale scores at baseline and 12 months after randomisation for intervention and control groups

    Reporting of patient safety incidents

    At baseline, 27 events were reported in 11 practices (from one to eight incidents in the preceding 12 months), and 24 of these were assessed as proper patient safety incidents. The mean number and quality of incident reports did not differ between the intervention and control groups. By the end of the trial, 362 events had been reported in 39 practices (1–26 incidents per practice team), and 225 of these were considered to be patient safety incidents. At the end of the trial, more patient safety incidents (mean (SD) 4.68 (4.94) vs 2.91 (5.29), CP=0.65, p=0.045) had been reported in the intervention group, and the reports made were of significantly better quality (2.19 (1.94) vs 1.40 (1.66), CP=0.65, p=0.038; table 4). Reported patient safety incidents mainly related to office administration, communication, investigations and medication (table 5).

    View this table:
    Table 4

    Quality scores for incident report data at baseline and 12 months after randomisation for both intervention and control groups

    View this table:
    Table 5

    Types of patient safety incidents reported by intervention and control practices at baseline (in total n=24) and 12 months after randomisation (in total n=225)

    Discussion

    This is the first study to investigate the effects of a MaPSaF-like instrument on safety culture in primary healthcare within the framework of a randomised controlled trial. No differences between the intervention and control groups could be shown after 12 months in patient safety culture indicator error management and 11 further indicators and safety climate scales. Nevertheless, better reporting of patient safety incidents was observed in intervention practices, as shown by more reports and reports of higher quality than in the control practices. In the interventions practices, about one action plan per practice was successfully implemented.

    Comparison with other research investigating safety culture interventions

    Few studies have used a MaPSaF-like approach to improve safety culture in healthcare. One trial used an adaptation of MaPSaF over the course of 5 years. Fifty departments at three hospitals in Sweden repeated three cycles of preparation, meeting, assessment, reporting and improvement. Contrary to our intervention, this trial demonstrated improvements in patient safety outcomes (eg, hospital-acquired infections) and in safety culture assessment measures.12 In another trial conducted in 23 intensive care units in the USA, the results of a survey on safety climate were discussed in the units to identify areas of improvement in central line-associated bloodstream infection rates and ventilator-associated pneumonias.13 Both safety climate scales and clinical outcomes improved. However, both studies had pre-post designs without any randomly selected control groups in place.

    Generally, studies evaluating interventions that aim to improve safety culture have shown increases in safety climate scores,29–35 while one study also reported improved observed safety behaviour and performance.36 Unfortunately, these studies either had uncontrolled pre-post designs, small sample sizes of selected participants or were case reports from single healthcare organisations.37 In addition, all studies were conducted in an inpatient setting. In a recent systematic review on strategies for improving patient safety culture in hospitals, only leadership walk rounds and multifaceted unit-based interventions were found to show ‘some stronger evidence’ of a positive impact but, again, these studies generally had a number of methodological limitations.3

    Strengths and weaknesses of the intervention and trial

    A strength of the intervention is that it takes into account proposed barriers to culture change such as lack of ownership,38 as well as facilitating factors such as workforce empowerment, engagement and support by leadership.39 FraTrix is based on team processes and supports mutual understanding and agreement on perceptions of safety culture and actions for improvement. It enables the teams to develop their own action plans according to their needs, thus empowering all practice staff. As shown, intervention teams decided on plenty of sensible actions as a result of the team sessions. These actions can be perceived as appropriate since they are mostly directed towards systemic change and consist of structural changes to a considerable degree —that is, changes that are generally regarded as being more sustainable than changes directed at the individual healthcare professional or activities that require constant repetition. However, less than half of the measures were followed up by the team and an even smaller portion was fully implemented, resulting in around one successfully implemented action per practice. On the other hand, the implementation of all planned actions could not be expected in view of the teams’ heavy workload.

    The intervention, however, may be rather weak as it encourages staff to reflect on safety culture and possible actions plans rather than the implementation of specific practical changes. In spite of their very positive self-assessment, the participating teams were probably in an early phase of the change process. According to the Transtheoretical Model of change (TTM), individuals40 or organisations41 go through stages of change when altering their behaviour (pre-contemplation, contemplation, preparation, action, maintenance). The teams in our study may have been in the phase of pre-contemplation that is characterised by the lack of any real intention of taking action in the near future. Therefore, too much (in terms of real actions) may have been expected of what was basically mere reflection. In order to achieve real progress, accompanying measures to support the teams41 may have been needed. Furthermore, the reflection process within the teams may not have been as open as would have been desirable, as HCAs may have found it difficult to conduct an open discussion. Consequently, the ‘real’ problem may not have been identified. However, as most of the participating practices were single-handed, we decided against a solution involving team sessions solely for HCAs for practical reasons.

    Recently, facilitators of quality and safety improvements have demonstrated the importance of safety culture for successful change processes. Reductions in hospital-acquired infections in the Michigan Project were explained, among other factors, as being due to a ‘networked community’, ‘reframing the subject of the intervention as a social problem with a solution’ and the systematic feedback of data on clinical outcomes.42 Furthermore, knowledge about best practices43 and education and training in error reduction techniques were identified as other change-promoting factors.44 The impact of a FraTrix or MaPSaF-like intervention may be strengthened by such educational or organisational amendments.

    We conducted a randomised controlled trial with a predefined sample size of 60 practices and a combined method to measure safety culture. To date, no widely accepted and validated method for the measurement of safety culture has existed. Even though there are a significant number of questionnaires assessing safety climate, the supplementary use of richer data is recommended.2 In other high-risk industries, safety culture is measured by means of staff questionnaires that are usually complemented by audits evaluating risk management documents45 and interviews with staff on how they actually conduct their work.46 We therefore followed recent recommendations2 ,15 and combined both methods (incident reporting documents and PSCI) with a recently validated safety climate questionnaire (FraSiK)21 that was developed for German general practice.

    A further strength is the low dropout rate. In addition, by choosing the PSCI error management as a primary outcome, we selected a parameter that varied considerably between practices in terms of the data collected at the beginning and the end of the trial (table 2). We would therefore expect this indicator to be sensitive to change. We had expected the small team size to result in a shorter time frame being required for the implementation of changes than in hospital care.

    Unfortunately, established instruments for studying interventions concerned with patient safety or safety culture in general practice were not available so we applied quality indicators as outcomes. These are regularly used for auditing but have not been validated for use in controlled trials. It was therefore somewhat unclear whether the indicators were sufficiently sensitive to changes or whether they would register changes induced by the FraTrix intervention. The suitability of error management as the primary outcome also requires some discussion. It is possible that this outcome is too demanding since it requires more patient safety proficiency on behalf of the team than simply planning and implementing actions such as changes in team communication or in the appointment system. In future evaluation studies of MaPSaF/FraTrix-like interventions, outcomes should be chosen that reflect behaviours that are more proximal to everyday care. In our study, the implementation of action plans would be an example of such an outcome.

    The method we used to assess incident reports20 is not a recognised measurement tool for research. We could demonstrate an improvement in terms of reporting rate and quality of reports, but this improvement could only be shown on an exploratory level of analysis. However, incident reporting is one of the dimensions of the matrix and the starting point for error management, and can therefore be regarded as another potential outcome for interventions like FraTrix.

    One drawback when studying safety culture interventions is that every method for measuring safety culture constitutes an intervention itself.3 ,47 At the end of the trial, participants reported that the interviews during data collection visits helped to identify problematic areas, and that changes had then been initiated without any further intervention. Moreover, discussions during the team sessions may also have had an unexpected impact on the data. The criteria for almost all indicators relied in part on self-reporting (see table 2 and online supplementary appendix 2, part 3), and participants in the intervention group reported that they were much more critical of their practices when again confronted with the topic of data collection 12 months later. Thus, it could be argued that we underestimated a possible positive effect of the intervention. Furthermore, we applied a minor intervention to the control group—although with a much smaller degree of involvement of the team members—that could have had a positive impact on the results, making it even more difficult to detect a difference in outcomes.

    In addition, the time frame of the intervention may account for the lack of an intervention effect. As cultural changes require a considerable amount of time to emerge, 1 year may have been too short for a significant change to occur.48 ,49 A short time frame was also thought to partly explain the lack of any greater improvement in another recent controlled trial; measurable effects may have been delayed and are possibly only detectable at a later date than that of the final measurement.50

    Although we randomly allocated practices to the intervention and control groups in our trial, participation alone can be interpreted as demonstrating a commitment to quality51 and patient safety. The participating teams in our trial perceived an overall positive safety climate at baseline and left little room for improvement. We cannot rule out that participating teams were already advanced in terms of safety culture. This selection bias, as well as the low participation rate, should be considered when generalising the results of the trial.

    Implications for practice and future research

    When it comes to cultural change, self-assessment with FraTrix appears to be a good starting point. Self-assessment and team-based reflection alone led to a considerable number of actions aimed at improving safety culture but did not result in a measurable change in structures, organisational processes and climate.

    In conclusion, appropriate and realistic study outcomes and measures to support the potential effects of FraTrix may be needed. The latter could include team support by a supervising coach, and education and training in patient safety in order to strengthen the abilities of teams to successfully design and implement safe structures and procedures in primary care. To truly learn about strengths and weaknesses in safety culture, validated quantitative and qualitative methods45 and adequate time frames for interventions are further required for the evaluation of safety culture in primary care.

    Acknowledgments

    We would like to thank each practice team that participated in our trial and its pilot phase and are particularly grateful for their confidence in our project team. In addition, we thank the AQUA Institute, Goettingen, the National Association of Statutory Health Insurance Physicians (KBV), their AQUIK project, and Dr Susanne Kleudgen, Berlin, for their support in the design of the trial. Furthermore, we thank Phillip Elliott who helped enormously with the writing of the manuscript.

    References

    1. ↵
      1. Kirk SA,
      2. Marshall MN,
      3. Claridge T,
      4. et al
      . Evaluating safety culture. In: Walshe K, Boaden R.eds. Patient safety: research into practice. Maidenhead: Open University Press, 2005. 173–184
    2. ↵
      1. Halligan M,
      2. Zecevic A
      . Safety culture in healthcare: a review of concepts, dimensions, measures and progress. BMJ Qual Saf 2011;20:338–43.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Morello RT,
      2. Lowthian JA,
      3. Barker AL,
      4. et al
      . Strategies for improving patient safety culture in hospitals: a systematic review. BMJ Qual Saf 2013;22:11–8.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Westrum R
      . A typology of organisational cultures. Qual Saf Health Care 2004;13:ii22.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Kirk S,
      2. Parker D,
      3. Claridge T,
      4. et al
      . Patient safety culture in primary care: developing a theoretical framework for practical use. Qual Saf Health Care 2007;16:313–20.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      Parker D. Manchester Patient Safety Framework (MaPSaF), Primary Care. Manchester: The University of Manchester, 2006.
    7. ↵
      National Patient Safety Agency. Manchester Patient Safety Framework (MaPSaF). http://www.nrls.npsa.nhs.uk/resources/?EntryId45=59796 (accessed 16 Dec 2012).
    8. ↵
      1. Struben V,
      2. Wagner C
      . Ontwikkeling van een Instrument voor Zelf Evaluatie van de Pateintvewiligheidscultuur (IZEP). Utrecht, 2006. http://www.nivel.nl/pdf/Ontwikkeling-van-een-Instrument-voor-Zelf-Evaluatie-van-de-Pati%C3%ABntveiligheidscultuur-%28IZEP%29.pdf (accessed 16 Dec 2012).
    9. ↵
      1. Fleming M,
      2. Wentzell N
      . Patient safety culture improvement tool: development and guidelines for use. Healthcare Q 2008;11:10–15.
      OpenUrl
    10. ↵
      1. Mannion R,
      2. Konteh FH,
      3. Davies HTO
      . Assessing organisational culture for quality and safety improvement: a national survey of tools and tool use. Qual Saf Health Care 2009;18:153–6.
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. Parker D,
      2. Lawriea M,
      3. Hudson P
      . A framework for understanding the development of organisational safety culture. Saf Sci 2006;44:551–62.
      OpenUrlCrossRefWeb of Science
    12. ↵
      1. Öhrn A,
      2. Rutberg H,
      3. Nilsen P
      . Patient safety dialogue: evaluation of an intervention aimed at achieving an improved patient safety culture. J Patient Saf 2011;7:185–92.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Vigorito MC,
      2. McNicoll L,
      3. Adams L,
      4. et al
      . Improving safety culture results in Rhode Island ICUs: lessons learned from the development of action-oriented plans. Jt Comm J Qual Patient Saf 2011;37:509–14.
      OpenUrlPubMed
    14. ↵
      1. Müller V,
      2. Hoffmann B,
      3. Albay Z,
      4. et al
      . The Frankfurt Patient Safety Matrix—a safety culture instrument [Die Frankfurter Patientensicherheitsmatrix—ein Instrument zur Selbsteinschätzung der Sicherheitskultur in Hausarztpraxen]. Z Allgemeinmed 2012;87:499–508.
      OpenUrl
    15. ↵
      Health and Safety Executive. A review of safety culture and safety climate literature for the development of the safety culture inspection toolkit. 2005. http://www.hse.gov.uk/research/rrpdf/rr367.pdf (accessed 16 Dec 2012).
    16. ↵
      1. Schein EH
      . Organizational culture and leadership. 3rd edn. San Francisco: Jossey-Bass, 2004.
    17. ↵
      1. Kleudgen S,
      2. Diel F,
      3. Burgdorf F,
      4. et al
      . AQUIK®: starter set of ambulatory quality indicators developed by the German National Association of Statutory Health Insurance Physicians [KBV entwickelt Starter-Set ambulanter Qualitätsindikatoren—AQUIK®-Set]. Z Evid Fortbild Qual Gesundhwes 2011;105:54–63.
      OpenUrlCrossRefPubMed
    18. ↵
      1. Engels Y
      . Testing a European set of indicators for the evaluation of the management of primary care practices. Fam Pract 2005;23:137–47.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      National Patient Safety Agency. What is a patient safety incident? http://www.npsa.nhs.uk/nrls/reporting/what-is-a-patient-safety-incident/ (accessed 16 Dec 2012).
    20. ↵
      1. McKay J,
      2. Murphy D,
      3. Bowie P,
      4. et al
      . Development and testing of an assessment instrument for the formative peer review of significant event analyses. Qual Saf Health Care 2007;16:150–3.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Hoffmann B,
      2. Domanska OM,
      3. Albay Z,
      4. et al
      . The Frankfurt Patient Safety Climate Questionnaire for General Practices (FraSiK): analysis of psychometric properties. BMJ Qual Saf 2011;20:797–805.
      OpenUrlAbstract/FREE Full Text
    22. ↵
      World Health Organization. Conceptual framework for the international classification for patient safety: Version 1.1. Technical report January 2009. http://www.who.int/patientsafety/implementation/taxonomy/icps_technical_report_en.pdf (accessed 20 Dec 2013).
    23. ↵
      1. Dy SM,
      2. Taylor SL,
      3. Carr LH,
      4. et al
      . A framework for classifying patient safety practices: results from an expert consensus process. BMJ Qual Saf 2011;20:618–24.
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Zhao YD,
      2. Rahardja D,
      3. Qu Y
      . Sample size calculation for the Wilcoxon-Mann-Whitney test adjusting for ties. Stat Med 2008;27:462–8.
      OpenUrlCrossRefPubMed
    25. ↵
      1. Rahardja D,
      2. Zhao YD
      . Unified sample calculations using the competing probability. Stat Biopharm Res 2009;1:323–7.
      OpenUrlCrossRef
    26. ↵
      Kassenärztliche Vereinigung Hessen. Health care today [Versorgung heute]. http://www.kvhessen.de/Wir+%C3%BCber+uns/Publikationen/Versorgung%20heute.html (accessed 16 Dec 2012).
    27. ↵
      1. Hoffmann B,
      2. Mießner C,
      3. Albay Z,
      4. et al
      . Impact of individual and team features on patient safety climate: a survey in family practices. Ann Fam Med 2013;11:355–62.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      The Linnaeus-PC Collaboration. International taxonomy of medical errors on primary care—version 2. Washington, DC: The Linnaeus-PC Collaboration, 2004.
    29. ↵
      1. Pronovost P,
      2. Weast B,
      3. Rosenstein B,
      4. et al
      . Implementing and validating a comprehensive unit-based safety program. J Patient Saf 2005;1:33–40.
      OpenUrlCrossRef
    30. ↵
      1. Ginsburg L,
      2. Norton PG,
      3. Casebeer A,
      4. et al
      . An educational intervention to enhance nurse leaders’ perceptions of patient safety culture. Health Serv Res 2005;40:997–1020.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Thomas EJ,
      2. Sexton JB,
      3. Neilands TB,
      4. et al
      . The effect of executive work rounds on nurse safety climate attitutes: a radnomized trial of clinical units. BMC Health Serv Res 2005;5:28.
      OpenUrlCrossRefPubMed
    32. ↵
      1. Timmel J,
      2. Kent PS,
      3. Holzmueller CG,
      4. et al
      . Impact of the Comprehensive Unit-Based Safety (CUSP) Program on safety culture in a surgical inpatient unit. Jt Comm J Qual Patient Saf 2010;36:252–60.
      OpenUrlPubMed
    33. ↵
      1. Pettker CM,
      2. Thung SF,
      3. Raab CA,
      4. et al
      . A comprehensive obstetrics patient safety program improves safety climate and culture. Am J Obstet Gynecol 2011;204:216.e1–.e6.
      OpenUrlCrossRefPubMed
    34. ↵
      1. Haynes AB,
      2. Weiser TG,
      3. Berry WR,
      4. et al
      . Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf 2011;20:102–7.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Sexton BJ,
      2. Berenholtz SM,
      3. Goeschel CA,
      4. et al
      . Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med 2011;39:934–9.
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Stead K,
      2. Kumar S,
      3. Schultz TJ,
      4. et al
      . Teams communicating through STEPPS. Med J Aust 2009;190:S128–32.
      OpenUrlPubMedWeb of Science
    37. ↵
      1. Odwazny R,
      2. Hasler S,
      3. Abrams R,
      4. et al
      . Organizational and cultural changes for providing safe patient care. Qual Manag Health Care 2005;14:132–43.
      OpenUrlCrossRefPubMed
    38. ↵
      1. Scott T,
      2. Mannion R,
      3. Davies HTO,
      4. et al
      . Implementing culture change in health care: theory and practice. Int J Qual Health Care 2003;15:111–18.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. Hale AR,
      2. Guldenmund FW,
      3. van Loenhut PLCH,
      4. et al
      . Evaluating safety managementand interventions to improve safety culture: Effective intervention strategies. Saf Sci 2010;48:1026–35.
      OpenUrlCrossRef
    40. ↵
      1. Prochaska JO,
      2. Velicer WF
      . The transtheoretical model of health behavior change. Am J Health Promot 1997;12:38–48.
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Levesque DA,
      2. Prochaska JM,
      3. Prochaska JO,
      4. et al
      . Organizational stages and processes of change for continuous quality improvement in health care. Consult Psychol J Pract Res 2001;53:139–53.
      OpenUrlCrossRef
    42. ↵
      1. Dixon-Woods M,
      2. Bosk CL,
      3. Aveling EL,
      4. et al
      . Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89:167–205.
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Brandrud AS,
      2. Schreiner A,
      3. Hjortdahl P,
      4. et al
      . Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Qual Saf 2011;20:251–9.
      OpenUrlAbstract/FREE Full Text
    44. ↵
      1. Stock GN,
      2. McFadden KL,
      3. Gowen CR
      . Organizational culture, critical success factors, and the reduction of hospital errors. Int J Production Econ 2007;106:368–92.
      OpenUrlCrossRef
    45. ↵
      1. Badke-Schaub P,
      2. Hofinger G,
      3. Lauche K
      .eds. Human factors. Psychology of safety in high risk domains [Human factors—Psychologie sicheren Handelns in Risikobranchen]. 1st edn. Heidelberg: Springer, 2008.
    46. ↵
      1. Kongsvik T,
      2. Almklov P,
      3. Fenstad J
      . Organisational safety indicators: some conceptual considerations and a supplementary qualitative approach. Saf Sci 2010;48:1402–11.
      OpenUrlCrossRef
    47. ↵
      1. Hoffmann B,
      2. Hofinger G,
      3. Gerlach F
      . Is patient safety culture measurable and if so, how is it done? [(Wie) ist Patientensicherheitskultur messbar?] Z Evid Fortbild Qual Gesundhwes 2009;103:515–20.
      OpenUrlCrossRefPubMed
    48. ↵
      1. Guldenmund FW
      . (Mis)understanding safety culture and its relationship to safety management. Risk Anal 2010;30:1466–80.
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Sax HC,
      2. Brown P,
      3. Mayewske RJ,
      4. et al
      . Can aviation-based team training elicit sustainable behavioral change. Arch Surg 2009;144:1133–7.
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Benning A,
      2. Dixon-Woods M,
      3. Nwulu U,
      4. et al
      . Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ 2011;342:d199.
      OpenUrlAbstract/FREE Full Text
    51. ↵
      1. McLean K,
      2. Taylor W
      . Identifying the barriers to implementing a quality initiative in primary care: the views of practices in Scotland on the Quality Practice Award. Qual Prim Care 2007;15:353–9.
      OpenUrl
    View Abstract

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors BH led the study design with contributions from VM, GH, FMG, CG, JR and DP. ZA contributed to the development of data collection tools. ZA and BH led the data collection with contributions from BM, CM, ML and KW. BH led the analysis with contributions from JR, MG, BM, CG, KW, GH and DP. All authors contributed to the data interpretation and the revisions to the manuscript. The first draft was prepared by BH. All authors had full access to the data derived from the trial and take full responsibility for the integrity of the data and the accuracy of the data analyses. BH is guarantor for the trial.

    • Funding A grant provided by the Federal Ministry of Education and Research (grant no. 01GK0702) enabled this project. The funder had no role in the study design, collection, analysis and interpretation of data, writing of the report or the decision to submit the article for publication.

    • Competing interests http://www.icmje.org/coi_disclosure.pdf None.

    • Ethics approval Patients did not take part in the trial and, when used, patient data were anonymised. The institutional review board of the Medical Faculty of the Goethe University Frankfurt stated that no review of ethical issues for this quality improvement trial was required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Previous presentations Preliminary results of this trial were presented at the 2011 annual conference of the German and Austrian Colleges of General Practitioners and Family Physicians in Salzburg, Austria, 22–24 September 2011, as well as at the 2011 International Congress of the Patient Safety Foundation of Switzerland, 29–30 November 2011.

    • Trial protocol The full trial protocol can be provided by the corresponding author.