Discussion
We systematically developed a survey to assess institutional SLSs. The tool builds on a previous systematic review of SLS outcome factors, and was refined with input from experts in patient safety and quality, ensuring comprehension, face validity and internal consistency. The study had three steps. Initially, we developed the survey’s questions and their corresponding response options, culminating in a draft of 47 questions. During the second step, experienced researchers who were also involved in the systematic review conducted a pre-test. This led to the elimination of two questions, the revision of 15 and the introduction of two new ones. Additionally, to reduce bias, it was determined that 17 questions focused on SLS management should be exclusively answered by Q&S experts and leaders, while the rest could be addressed by both Q&S professionals and other staff members involved in the study. The final step involved clinical sensibility testing—evaluating the clarity and necessity of the questions—by Q&S experts at The Ottawa Hospital. Despite receiving the lowest scores in these areas, four questions (numbers 15, 30, 44 and 45) were retained by the research team because their approval ratings exceeded the 80% benchmark set for question inclusion.
Four demographic questions were also added. The first three questions ‘Institution’, ‘Position title’ and ‘What is your professional designation?’ can help to direct improvement efforts to the right hospital department, while the fourth question ‘Years of experience’ can help to either include or exclude participants based on their experience, which might affect their answers. Finally, the questionnaire was tested for internal consistency using the participants’ evaluation of their SLS; high reliability was indicated with a Cronbach’s alpha score of 0.94.
Although this study did not intend to evaluate the hospital’s SLS, we grouped the responses of the participants’ opinions into either ‘agree’ or ‘disagree’ and only present the percentage agreement in table 1 for simpler reporting. It is also important to mention that improving the SLS is a continuous process; therefore, decision-makers can work on any of the system areas even those with high agreed percentages. We also wish to highlight that the domains ‘prepare for analysis’ and ‘closing the loop’ received the highest totals of ‘disagree’ or ‘strongly disagree’ scores.
This study’s unique strength lies in the rigorous steps taken to develop the survey. To ensure the accuracy of the collected data, we followed the methods of Karen et al16 for face validity checks, based our questions on evidence from a systematic review and piloted the survey at a large, reputable hospital. This pilot allowed us to assess the applicability, clarity and necessity of all items. Finally, we incorporated feedback from both SLS experts and experienced Q&S professionals working within the SLS at The Ottawa Hospital.
Our newly developed questionnaire has many points in common with a 44-item self-assessment tool developed by the WHO in 2020.20 That tool was developed based on feedback from international experts in SLS. Both questionnaires use similar rating scales and cover many of the same major topics, including safety culture, leadership support, training, analysis and investigation, blame-free culture, availability of resources, feedback and the internal and external sharing of learning. On the other hand, our questionnaire was rigorously developed to enhance accuracy and clarity. We carefully formulated the questions to ensure understanding across diverse participant backgrounds. The questions were organised into domains to focus improvement efforts. We divided participants into two groups for obtaining tailored, precise results. To ensure practicality and relevance, we incorporated feedback from Q&S leaders working directly within a large accredited hospital’s SLS system. Finally, demographic data will inform recommendations, allowing us to target improvements for specific departments or employee groups.
Another study developed and validated an online questionnaire survey to test recently graduated doctors’ knowledge and experience of patient safety and incident reporting, and assess-related attitudes and behaviours.21 It was developed based on previously published questionnaires for medical students and nurses. It included 21 questions that partly overlap with our questionnaire including the principles of patient safety in their hospitals, their views on local reporting, training, the reporting stage, blame-free environment, involvement in incident discussions and 1 question on feedback. Our questionnaire has more items to assess feedback in addition to a detailed evaluation of the management role and the analysis process.21 Our questionnaire was tested for validity and reliability to ensure highly precise data collection. Being based on a systematic review, our collected data and subsequent recommendations are more generalisable and applicable.
The strengths of this study include the testing of the face validity of the questionnaire. A group of experienced health services researchers who are experts in patient safety reviewed and gave feedback on each item to ensure it is fit for the purpose. The questionnaire was evaluated again in the clinical sensibility test by experts in the field. Finally, internal consistency was measured and was found to be high. Therefore, we are highly confident that no irrelevant questions were included and that all critical topics were covered. In addition, our questionnaire was based on a systematic review study that included 22 primary research studies conducted worldwide. It collected the views of a range of healthcare professionals on the barriers and facilitators of SLSs. This would be reflected on the generalisability of our results. Being standardised, it might help to compare SLS performances among different hospitals.
All the safety experts, leaders and managers at The Ottawa Hospital were invited to participate in the internal consistency assessment; however, the response rate was low at 36%. Domain C had the lowest Cronbach’s alpha score (0.61), which can be explained by the low number of questions (six) known to affect the accuracy of the test; however, the result is still close to the lower reliability limit. Everyone rating the hospital’s SLS were experts in Q&S and could be concerned with the reputation of their hospital, so the possibility of bias in their responses exists. However, generalisability of the questionnaire is not expected to be affected as questions were based on the result of a systematic review that included studies from all over the world. Furthermore, testing the questionnaire for clarity, necessity and reliability depended mainly on participants’ general knowledge on Q&S. However, despite this theoretical consideration, further testing in a large sample of participants from multiple settings is required