Article Text
PDF
Abstract
Background This evaluation assesses the impact of an electronic safety netting software (E-SN) package, C the Signs, in primary care services across five boroughs in North East London (NEL).
Aim This study evaluates the use of E-SN software in primary care, examining its benefits and barriers, safety implications, and overall impact on individual and practice usage.
Design and setting The study is based on semi-structured interviews with 21 clinical and non-clinical members of staff from all primary care services using the software in NEL.
Method Semi-structured interviews were conducted to gather data on individual use of the software, safety implications and practice use of features such as the monitoring dashboard. Data were analysed using a rapid qualitative methodology.
Results Two approaches to E-SN software adoption were reported: whole practice adoption and self-directed use. Practices benefitted from shared responsibility for safety netting and using software to track patients’ progress in secondary care. Adoption was affected by information technology and administrative resources. Decision-support tools were used infrequently due to a lack of appreciation for their benefits. Selective adoption of different E-SN functions restricted its potential impact on early diagnosis.
Conclusion The use of E-SN software in primary care services in NEL varied among participants. While some found it to be beneficial, others were sceptical of its impact on clinical decision-making. Nonetheless, the software was found to be effective in managing referral processes and tracking patients’ progress in other points of care.
- CANCER
- Patient safety
- Qualitative research
- Electronic Health Records
- Decision support, computerised
Data availability statement
Data are available upon reasonable request. The interviews in this study are confidential and will remain with the authors, however, upon reasonable request we may be open to secondary analysis in collaboration with external research teams.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Primary care patients often leave without a clear diagnosis, leading to the development of safety netting as a means of supporting and monitoring patients until resolution or diagnosis, with the emergence of electronic safety netting (E-SN) tools aimed at assisting healthcare professionals in managing diagnostic uncertainty.
WHAT THIS STUDY ADDS
E-SN software was adopted in two distinct ways: whole practice adoption and self-directed use, with whole practice adoption predominantly adopted by practices with fewer locum staff and better IT systems.
This resulted in improved referral form completion and tracking through the use of the referral form auto-completion function, whereas the decision-support function was underused.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The use of E-SN software in primary care is unlikely to have a significant impact on earlier diagnosis of cancer, but the autocompletion of referral forms and referral tracking are valuable functions that can be used for whole-practice quality improvement.
Future quality improvement and research activities should address the whole practice context and setting of E-SN tools, not just their use in consultation.
Background
Most patients that attend primary care with undifferentiated symptoms leave without a definitive diagnosis, due to the uncertainty of whether the symptoms represent a serious illness or a benign self-limiting disease. The role of primary care is to use clinical judgement to steward healthcare resources in a responsible way, while minimising harm to the patient. In recent years, the role of ‘safety netting’ has received considerable research and policy attention, as a way of supporting and monitoring patients until their symptoms are resolved or diagnosed.1–4 Safety netting describes the management of uncertainty in the diagnosis and management of patients, often by providing information or organising follow-up for patients following consultation. Safety netting may be verbal (during the consultation), or enacted through electronic systems to monitor results and referrals.2 An estimated two-thirds of primary care consultations incorporate spoken safety-netting advice, half of which are documented in the electronic patient record.5 Thorough records of safety netting advice have been encouraged to support continuity of care and medico-legal procedures.1
Electronic health record systems have been developed internationally to support clinicians and managers to look after their patients.6 As part of this development in the UK, electronic safety netting (E-SN) tools have been designed to assist healthcare professionals in managing diagnostic uncertainty either within or separate to the electronic healthcare record.7 Similar product developments are occurring globally that are context-dependent.6 8 There is no consistent definition or approach to E-SN in the UK despite an increasing number of software products identifying as such and being offered to clinical teams, particularly since the COVID-19 pandemic. E-SN tools have been developed to perform a variety of functions including clinician alerts, administrative tasking, decision support and triggering reminder text messages to patients. One evaluation of an E-SN tool reported an increase in urgent referrals, and an increased cancer detection rate, however this was not statistically significant.9 Another evaluation of an E-SN tool was inconclusive about its benefits.10
This evaluation is focused on the C the Signs software, an E-SN software package which supports primary care clinicians to identify patients at risk of cancer at an early stage. The software is available for the main electronic patient records systems in the UK (Emis and SystmOne). The tool has a variety of incorporated functions11:
Cancer risk prediction tool using patient age, gender, risk factors, clinical markers and symptoms.
Referral support including suggested pathways for referral.
Auto-population of referral forms.
Patient education tools including videos and resources that can be sent over text or email.
Automatic tracking of patients on diagnostic or referral pathways using a dashboard with colour coding to identify delays.
Data dashboard to view cancer performance across the organisation (local or region).
Medical record integration.
The medical record integration function fulfils targets within the current financial reward system called the Quality and Outcomes Framework which stipulates that practices need to ‘have a robust and consistent system in place for safety netting’.12 There is a research gap in how E-SN tools work in practice in terms of improvements to patient safety or delays in cancer diagnosis.
Objectives
This paper outlines the results of an independent evaluation of E-SN software in general practices across North East London (NEL). Our objective was to understand the impact of the software on perspectives and practices surrounding early diagnosis of cancer and safety netting behaviours, both individually and at an organisational level.
Methods
Study design and setting
This was a service evaluation conducted within the geographical area covered by the NEL Cancer Alliance. It provided funding for E-SN software to be piloted in five London boroughs (Barking and Dagenham, Havering, Newham, Waltham Forest and Tower Hamlets) between November 2020 and June 2022. With the exception of Havering, these boroughs are relatively deprived communities with Barking and Dagenham, Newham and Tower Hamlets being within the 10% most deprived areas in England.13 The rationale for implementation of the E-SN software was to support primary care to identify patients at risk of cancer at the earliest and most curable stage of the disease. NEL commissioning managers supported the E-SN software deployment team, providing local information about direct access test availability, suspected cancer pathways and related patient information.
The evaluation employed rapid appraisal methodology, with semi-structured interviews used as the key method of data collection.14 15 Rapid appraisal methodology has three key components; (1) a system perspective (eg, attention to local practices and definitions); (2) multiple data sources and (3) iterative data collection and analysis. The latter was possible through structured team interactions and iterative feedback loops between the university research team and NEL Cancer Alliance. This section outlines the recruitment of participants, sampling of organisations and individuals, data collection focus, and data analysis methods.
Sample selection and recruitment
Professionals working in primary care practices within the NEL Cancer Alliance were eligible to participate. Authors DC, SP-D and AC recruited a convenience sample through known contacts with the aim of recruiting professionals from all five London boroughs where the E-SN software had been implemented. All primary care professional roles were included, and participants with varying experiences with the software were contacted.
Participants were identified through email cascades and direct contact with key primary care decision-makers, and approached for participation in the study. Their details were passed on to GBB and SM who interviewed them over the phone or on a virtual platform at a time that was convenient to them. GBB and SM are experienced qualitative interview researchers with a background in primary care and patient safety research. All primary care clinician practices across the North East region of London were invited to participate in the study.
We recruited 21 participants to interview including 15 general practitioners, 5 administrative staff and 1 practice nurse (see table 1). All participants were interviewed alone, apart from one general practitioner and practice nurse who were interviewed together.
Participant characteristics
Data collection
All supporting documents for the interviews, for example, the participant information sheet and the interview guide, can be seen in online supplemental files. The semi-structured interviews reflected the key evaluation questions for this study, which included:
Supplemental material
Supplemental material
How does the E-SN software affect workflow and processes?
How does the E-SN software help ensure referral safety?
How does the E-SN software affect/impact the quality/safety offered for patients who may be at risk of a cancer diagnosis?
Data analysis
All interviews were audio-recorded and transcribed verbatim. The main points were documented and compiled with real-time interview notes and synthesised on a Rapid Appraisal Procedure (RAP) sheet.16 RAP sheets are used in rapid qualitative research to summarise emerging findings so they can be shared while the study is ongoing. After each interview, GBB and SM completed a RAP sheet including the following categories:
General opinion of E-SN software.
Practice ethos/approach to E-SN software.
Uses for E-SN software.
Other safety netting information.
Safety implications of use of E-SN software.
Impact on workflow.
Technical aspects, software glitches etc.
Wider practice issues.
Potential improvements to E-SN software.
After carefully reviewing the RAP sheets and interview transcripts, our research team created an initial coding framework based on our main research questions. We summarised our key findings using this framework in a spreadsheet. By organising the data this way, we could compare information from different participants, allowing us to identify similarities and differences in the data for validation purposes. All authors contributed to discussions based on the completed data set, looking particularly for commonalities and differences in perspectives of the E-SN tool. We also noted ideas or perspectives that had not been discussed. These discussions generated overarching themes relating to the study aims, and recommendations for practice.
Findings
Overall use of the intervention
We found that there were two contrasting approaches to how E-SN software was used by primary care practices which affected its utility: (1) whole practice adoption and (2) self-directed use. Practices who adopted E-SN software at a whole practice level generally identified a higher satisfaction with the general use of the software at an organisational level.
Whole practice adoption of E-SN software: good referral practice for patients with suspected cancers
Participants whose practice had adopted this approach made it mandatory to use E-SN software to facilitate urgent suspected cancer referrals, with a unified approach across the practice and in some cases, across the Primary Care Network. They reported that this made the process more reliable, particularly in being able to access up-to-date forms (preventing rejection by secondary care) and managing referrals using the E-SN software dashboard.
We use it to make all of our cancer referrals within the practice, so we use it as a kind of system by which you know if you want to see whether you know, if someone needs to be referred or how next to manage them, you kind of use it as a troubleshoot page to kind of go through, do I need to refer this patient? Does it suggest any specific investigations? So we typically use it as a kind of go to system to help decision making in whether or not someone needs to be referred on a two week wait pathway and then also the other use of it is to save all of our patients on a safety netting dashboard that you can then review them on. (Participant 13)
These practices used E-SN software as their only means of tracking and safety netting referrals. Practices using this approach were generally well-managed with administrators and practice managers, and tended to be practices with a stable workforce including several partner primary care clinicians. The practices reported positive feedback with the software in improving the referral system, and some participants felt that this resulted in a more reliable and structured system for referring patients. For example, some practices felt that more errors were spotted using the dashboard:
Before, we used to have our receptionist keep a log of all the patients who needed a two week wait appointment. But it was manually updated, and I am sure that patients fell through the gaps at times. [E-SN software] is great because now we have designed human error out of the referral system. There have been cases where the [E-SN software] has picked up that a referral has been held up and we are able to chase it (e.g. with the hospital). We wouldn’t have had the ability to have that oversight with the other handwritten system (Participant 6)
Self-directed use of E-SN software: variance in uptake and satisfaction
In contrast, some participants described that their practice allowed voluntary use of the E-SN software for those who wanted to. These practices were described as being staffed with a high proportion of locum primary care clinicians who found it difficult to access E-SN software. Some participants had alternative/duplicative systems in place for generating and tracking referrals such as manual entry spreadsheets or paper diary methods, with many of these practices ignoring this capability of the software altogether.
I have another dashboard on ERS, my worklist, where I keep looking at referrals so I can’t be looking at too many worklists because if you have three email boxes, you will ignore one. Everyone seems to be into less creative worklists, that’s the new thing so, anyway, the first thing is, I don’t need lots and lots of dashboards and worklists (Participant 3)
For clinicians who had chosen to use the E-SN software in a self-directed way, it created extra work due to the duplicative safety-netting systems as well as limited administrative support for the tracking of referrals using the E-SN dashboard:
Obviously [E-SN software] works really well if it’s a whole team approach. So if the admin team can do the administrative aspect of it and do all that follow up then it works really well. Because here we’ve had to keep a dual system approach, the admin team just are too stretched to do what they’re already doing, keeping a spreadsheet of all the two week wait referrals that anyone ever makes as well as managing [E-SN software]. […] So it puts my workload up massively and it means that I kind of hold the ball for not only my own two week wait referrals but you know for all the other two week wait referrals that are made in the practice via [E-SN software]. (Participant 15)
These practices were characterised by a lack of coordinated workflow between clinicians and administrators, and a lack of security around whether patient delays would be detected and acted on.
Self-directed uptake in the software was associated with poor technology or infrastructure, for example, old computers and low investment in technology which made E-SN software run slowly. Variance in uptake across a practice was also related to poor training across the practice. In some cases, early frustration with E-SN software affected how it was used by some individuals and resulted in some cases with eventual abandonment of E-SN software technology.
Specific use of different components of the E-SN software functions
Practices taking a whole practice approach to use of E-SN software tended to value certain functions more than practices taking a self-directed approach. A full description of the positive and negative experiences associated with each function are described in table 2. A thematic discussion of particular issues associated with these functions follows below.
Electronic safety netting tool functions and associated positive and negative experiences reported by participants
Decision-support tools were appreciated by less experienced practice members, but rarely used by more experienced clinicians
There were varying degrees of engagement with the decision-support tool functions of E-SN software which are designed to offer suggestions for tests or referral pathways based on listed symptoms (see table 1). Twelve clinical participants identified that they made little use of decision-support tools, and only used E-SN software to generate the referral form after the consultation was finished or at the end of it.
I use [E-SN] if I suspect cancer, that is the thing. If I don’t suspect it I won’t use it but if I do use it then that is a very helpful tool because it will tell you what to do or it will say, it’s okay, don’t worry or whatever; 9 times out of 10 it will tell me to refer. Often, it is all down to your own experience, if you get the alarm bells it’s cancer, you’ve got the system and then you will use it. I don’t think it’s been-- it is inclusive, it is for all, for anybody, certainly. (Participant 5)
Use of the E-SN software decision-support tool seemed to be partly dependent on experience, with less enthusiasm from senior primary care clinicians in this respect. Experienced primary care clinicians felt that an algorithmic tool was incapable of appreciating the subtleties of their patients’ needs, and that it would not add anything to their clinical acumen.
I’m an old hat, I’ve been around a long time, and I know the cancer signs. This is more for people who are new and haven’t been around the block a long time I think. I’m experienced, I’ve seen every cancer under the sun. So, I use my experience rather than filling in a tick box form. (Participant 2)
Trainee and newly qualified primary care clinicians were more enthusiastic and reported using it as an educational tool, even if they had already formed a diagnosis:
I guess as a trainee it is useful for me at times. I can think back to a sarcoma where I wasn’t really sure and the symptom decision tool was helpful. But most of the time it is more of a validation that what you thought was correct and it’s rare I would solely use the decision tool without already having a diagnosis in my mind (Participant 1)
Participants who did use the decision-support tool felt that it gave them confidence, for example, when interpreting blood test results. One participant mentioned that the decision-making tool encouraged them to pursue multiple lines of inquiry, for example, referring to colorectal and ovarian pathways at the same time, or eliciting further symptoms from the patient during the consultation. A small number of trainee and newly qualified primary care clinicians agreed that E-SN software helped them to remember the National Institute for Health and Care Excellence guidelines for suspected cancer referral.
Automatic completion of the urgent referral forms was a valued function for practices with unified adoption of E-SN software
Practices that took a whole practice approach to E-SN software particularly valued the function of the software that helped to complete urgent referral forms (see table 1). There were no downsides reported to this function, where it was used. Participants particularly valued avoiding delays in referral associated with incorrect or missing forms, where E-SN software was a much safer system than the previous method of identifying and filling in the correct form.
Due to the ease of completing referral forms, primary care clinicians suggested that it might have increased their willingness to refer patients but others were not sure of its impact. Benefits gained from this aspect of the software were dependent on the practice arrangements for referral; for example, some primary care clinicians sent referral forms to administrators to manage the electronic referral system, whereas other primary care clinicians did it themselves.
Use of the dashboard/tracker was seen to increase patient safety contingent on administrative support
Participants identified that the software provided an easy to access tracker (‘dashboard’) for their patients who had been referred to secondary care or sent for diagnostic investigations. The dashboard had a colour coded system which would identify patients whose appointments or reports had been delayed. This benefit was only realised for practices where administrative teams took a strong lead on monitoring and acting on information displayed on the dashboard. This distribution of responsibility meant that clinicians were unburdened from having to make notes or track their own referrals, confident that the administration team was actively monitoring any delays. Compared with previous software and tracking systems (such as password-protected spreadsheets), the E-SN software benefitted administrators and practice managers as all practice staff had access to the same system. Administrators could track referrals and discharge letters, and individual primary care clinicians who were keen to follow-up on patients of concern could do so.
Yes, it is easy to track all the patients who have done the referral, and what is going on with their referral, like after two weeks, if the doctor wants to check, it is easy to go onto [E-SN software] and get into the consultation and check whether we have received any letters from the hospital about what is going on with the patient; so, it is easy to use. (Participant 4)
Participants identified the dashboard as the E-SN software feature which provided the greatest improvement to patient safety in their everyday practice. For example, patients whose appointments were delayed could be identified more easily:
Recently there was a case where we could see there was a delay in a patient getting their two week wait appointment. Due to the dashboard we could see this and were able to call secondary care quickly, alert them to the problem, and get an appointment out asap to the patient. Is this sort of work our (the administrators) job? No. But at least the software allowed us to know a patient may have slipped through the cracks and we could stop that (Participant 7)
This was expressed as a benefit in terms of early diagnosis, but also as a defence against potential legal ramifications for clinicians:
The biggest concern is a delayed diagnosis, maybe you have not thought about it, I don’t know, but it has a lot of legal ramifications. […] I think doctors have to be aware that you need to protect yourself and do the job properly. I think have in your mind that someone may have cancer and use [E-SN software] because it is a protective tool, it is there for your protection, use it. (Participant 5)
None of the participants we interviewed used the E-SN software tracker for anything other than urgent suspected cancer referrals.
Discussion
This rapid qualitative evaluation of E-SN software demonstrated that it was adopted at a practice level in two distinct ways: (1) whole practice adoption with a unified approach to use of the software to manage referrals and (2) self-directed use which was highly variable. The whole practice approach was predominantly adopted by practices with fewer locum staff and better information technology systems. The whole practice adoption approach was seen as a positive improvement, mainly in terms of improved referral form completion and tracking. The referral form auto-completion was the most used function of the software, valued for being up to date and saving time and repetition. These were only used for urgent suspected cancer referrals. A minority of mainly early career participants used the decision-support tool. Our interpretation is that this reflects both a pattern of variable approaches to organisational leadership, as well as inequalities in information technology resource and capability within primary care.
Key findings in relation to previous research
Our core finding is that the utility of this E-SN tool is dependent on local context: practice leadership around mandatory adoption of the tool, information technology maturity, the presence of locums and trainees, and administrative resources. Evaluations of other decision-support tools suggest that they are also ineffective in practice, which may be due to poor implementation, insufficient uptake or limited contribution of the tools themselves.17
Previous research has also noted that efforts to improve safety in primary care are facilitated by system solutions such as delegation or shared responsibility,18 and hindered by the instinct for general practitioners to work as individuals.19 This has been found to result in slow, duplicative processes around technology use.20
Our study revealed that the decision-support tool functions of the E-SN software were poorly used due to low valuation of their benefit to experienced clinicians. Decision-support tools in all contexts are frequently seen as too simplistic or lacking in credibility for experienced clinicians who employ the ‘art and craft’ of medicine, and are more suitable for trainees.18 21 However, there is evidence that decision support tools can reduce errors that are more easily noticeable, such as medication safety.22
Finally, our findings are similar to studies addressing other forms of safety netting practice, where inconsistency in thoroughness of verbal safety netting is associated with poorer outcomes such as delayed reconsultation.5 23 24 This suggests that consistent, whole practice approaches are beneficial for all types of safety netting.
Strengths and limitations
To our knowledge, this is the first independent evaluation of an E-SN tool in an applied primary care setting. Rapid qualitative methods have enabled data from several practices to be synthesised, increasing the generalisability of the findings. We recruited a relatively modest sample, which may indicate some bias in our findings. However, we obtained a wide range of perspectives suggesting that we had data representing some diversity of experiences with the E-SN tool. A key strength was the incorporation of both clinical and administrative staff perspectives. We could have strengthened our study with a greater number of practice nurses, and the inclusion of wider stakeholders such as the software developers and commissioners.
Recommendations for research and practice
Given that most primary care clinicians are using E-SN software when they have already decided to refer a patient, it is unlikely to have a substantial impact on driving earlier diagnosis of cancer. Nevertheless, our evaluation suggests that autocompletion of referral forms and referral tracking are valued functions and could be used as part of whole-practice quality improvement. Decision-support tools that estimate cancer risk and suggest referral pathways are widely available, but underused.25 26
Future research should look at the impact of specific features of E-SN tools on a range of patient outcomes including process errors such as missing forms, delays in reconsultation, non-attendance at specialist hospital appointments and so on. E-SN should be developed that reflect both safety principles and the needs of patients and primary care clinicians.27 Implementation studies are recommended that address the whole practice context and setting of E-SN tools and not just their use in consultation.28
Conclusion
A commonly available E-SN tool was evaluated in primary care practices in London. There was substantial variance in the way E-SN software was adopted, with meagre indications about the potential for E-SN software to lead to improvements in patient care. Many practices failed to use all the software features or used them in a way which duplicated effort without improving safety. Practices who adopted E-SN software across the whole practice generally identified a higher satisfaction with the general use of the software at an organisational level. The most effectively used functions of E-SN software were the urgent suspected cancer referral forms, suggesting that support with referral form completion and tracking is desirable. The functions of the software were highly dependent on effective information technology and administrative resource.
Data availability statement
Data are available upon reasonable request. The interviews in this study are confidential and will remain with the authors, however, upon reasonable request we may be open to secondary analysis in collaboration with external research teams.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants but following exemption from ethics committee review by the UCL/UCLH Joint Research Office, we obtained Health Research Authority approval for this study (EDGE ID 144185) in accordance with the NHS ethics decision toolkit which deemed NHS ethics approval unnecessary due to the lack of patient participation. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors wish to thank members of the project working group for their support, including Bea Bakshi, Steph Hyams, Shaz Kang, Helen Stedeford, Jyoti Sood, Kanika Rai, Maurice Sanomi, Wayne Douglas, Jake Goodman and Cherie Pope.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @georgiabblack
Contributors All authors devised the project, the main conceptual ideas and proof outline. AC, DC and SP-D led the participant recruitment. GBB and SM collected the interview data and performed the analysis. GBB wrote the initial manuscript draft. All authors contributed to editing the draft and approved the final draft for publication. GBB is the guarantor.
Funding This evaluation was funded by North East London and North Central London Cancer Alliances (no grant award number). GBB acknowledges funding from The Institute of Healthcare Improvement Studies (University of Cambridge) (RG88620/PD-2019-02-004), and the National Institute for Health Research (NIHR) Policy Research Programme, conducted through the Policy Research Unit in Cancer Awareness, Screening and Early Diagnosis, PR-PRU-1217-21601. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Competing interests DC works in the Cancer Alliance where this evaluation took place. AC and SP-D worked in the NEL Commissioning Support Unit at the time this evaluation was undertaken. The results of the evaluation had a direct impact on ongoing funding decisions in relation to the software.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.