Evaluation of a toolkit resource package to support positive workplace behaviours in relation to quality end-of-life care in Australian hospitals
=================================================================================================================================================

* Claire Hutchinson
* Jennifer Tieman
* Kim Devery

## Abstract

**Background** The study aimed to determine the effectiveness of an action-orientated toolkit in supporting behaviour change in relation to quality end-of-life care in acute hospital settings. The toolkit was developed to complement a programme of online end-of-life care education.

**Methods** A toolkit was developed from an international review of peer-reviewed literature on end of life. Toolkits were distributed (n=428) to Australian healthcare professionals over a 4-week period. An online survey was sent to all recipients; 65 responses were received (16% response rate, excluding emails returned as undeliverable). Semistructured interviews (n=10) were conducted using purposeful sampling to ensure a range of views were captured. The focus of the evaluation was on investigating (1) users’ responses to the toolkit and (2) individuals’ reported behaviour change.

**Findings** The toolkit was well received by users who reported increased confidence in communication around end-of-life matters. 59.3% of users reported making a behaviour change over the previous 4 weeks; 70.8% of those who had not made a change reported they intended to in the near future. Against expectation, the toolkit’s appeal went beyond its intended audience in acute hospital settings, for example, personal care workers in aged care settings.

**Conclusions** Despite study limitations (self-report of a small, self-selected sample), these early findings suggest that the toolkit has potential to positively impact on end-of-life care practices. However, additional evaluation is needed to determine whether such a toolkit can positively impact on practice and on patient experience at the end of life.

*   health services research
*   healthcare quality improvement
*   health professions education

## Introduction

Surveys show that most Australians would prefer to die in their own home,1 yet few communicate their wishes to family or healthcare providers.2 In reality, the majority of Australians die in acute hospital settings,3 4 with almost half of these receiving palliative care.5 Notably, the rate of deaths in Australian hospitals is double of that of other similar developed economies of the Organisation for Economic Co-operation and Development, such as New Zealand, France, Ireland and the USA.4 Therefore, as Swerissen and Duckett6 note, dying in Australia is increasingly medicalised and institutionalised.

Despite the majority of Australians dying in hospital, healthcare professionals find initiating discussions regarding end of life challenging.6–11 Clinicians perceive that such discussions are distressing for patients, although research suggests that few patients are actually distressed by such conversations, less than 2% in a US study.8 Instead, clinicians tend to avoid end-of-life conversations by overestimating patients’ chance of recovery12 and continuing treatment, which may be invasive with little chance of prolonging life.6 13 Complaints from patients and families most often relate to poor communication at the end of life.14 15 

End-of-life care is not typically covered in sufficient depth in undergraduate training to prepare doctors and nurses for working with dying patients,16–21 and research has demonstrated that healthcare professionals are well aware of the gaps in their knowledge and skills in this area.22–24 A New Zealand study found that only 19% of clinicians (n=598) had received any formal palliative care training, although 74% would have liked such training.23 The authors observed that those who had received formal training were significantly more confident in delivering end-of-life care than those who had not received such training.23 

Researchers have assessed the impacts of a range of training initiatives to improve end-of-life care. Training interventions have included classroom training25 26 and simulation exercises.27 28 In recent years, online training has been used by many professional groups, including healthcare professions.29 30 Online education has many advantages over classroom training, including flexibility, convenience for users and alignment with the adult learning principles.29 31 It is also a cost-effective way of delivering education to target populations that may be geographically dispersed and where synchronising time for education sessions may be challenging. As in other sectors, online education in the healthcare sector is expanding.30 Research has demonstrated that user satisfaction with online training is high and that such training has been effective in enhancing knowledge.29 32 However, online independent learning does not necessarily test the application of knowledge in the workplace. Indeed, education programmes in general are typically poor at evaluating knowledge transfer to the workplace.33–35 

### The End-of-Life Essentials Project

The Australian Government has committed to funding federal and state projects to implement initiatives to provide high-quality palliative care and end-of-life support for all Australians who require it.36 The Federal Government has provided funding for projects focused on education, training, quality improvements and advance care planning. One such project was the End-of-Life Essentials Project. Phase 1 was to develop six free online education modules. The modules were conceptually developed from the Australian Commission on Safety and Quality in Health Care framework and aimed to increase knowledge and skills in end-of-life care for doctors, nurses and allied health professionals. The modules were launched between June and October 2016, and have been completed by over 3700 doctors, nurses and allied health professionals to date.

Phase 2 of the project was to develop a resource toolkit to complement the online modules. Whereas the online education modules focus on developing knowledge, the focus of the toolkit is to bring about positive change to workplace behaviour at the point of care. Therefore, the toolkit aims to change workplace behaviour at the individual level rather than at the institutional level. As with the online education modules, the toolkit is evidence-based, peer-reviewed, and aims to be generic enough to meet the needs of all healthcare professionals. We could find no evidence of similar resources in end-of-life care nationally, and limited resources internationally, although notably the UK has an online end-of-life toolkit which contains some generic resources as well as those that specifically relate to UK policies, practices and standards around end-of-life care.37 

This paper describes how the toolkit was developed and reports on findings from a mixed methods evaluation. The aims of the evaluation were first to evaluate reactions to the toolkit, that is, the degree to which participants found the resources favourable, engaging and relevant to their jobs; second, to evaluate whether users report making positive changes to workplace behaviours as a result of having the toolkit.

## Methods

The evaluation study was mixed methods using an online survey and telephone interviews as data collection methods.

### Developing My Toolkit

The toolkit was developed following an international review of peer-reviewed literature on end of life in late 2016 and early 2017. From this review, a number of action items were developed and associated resources identified. Feedback on the content, structure and format of the toolkit resources was sought from healthcare academics and practitioners. The toolkit was made available in hard copy format and consisted of a folder containing checklists of 60 suggested actions aligned with the End-of-Life Essentials online education modules. Fifty additional resources in support of the suggested actions were available to users online. These online resources included hyperlinks to policy documents, fact sheets, guidelines, journal papers, professional group-specific resources, and video and audio resources.

We received 252 registrations for the toolkit prior to launch. In the 3 weeks following the launch, a further 236 registrations were received. Registering for the toolkit required people to complete an online form detailing their name, address, profession and email address. Once duplications had been removed, 428 toolkits were distributed prior to the evaluation. All recipients received an email with a link to a short online survey. Recipients were from all states and territories and consisted of 13 doctors, 309 nurses and 66 allied health professionals. Forty toolkit recipients identified as ‘other’, and included educators, managers, volunteers, aged care workers, pastoral support workers, policy makers, project officers, a terminally ill patient and an end-of-life doula. Toolkit recipients’ primary work areas included emergency departments, intensive care units, surgical and general wards, outpatient clinics, aged care, education, gerontology and rehabilitation, advance care planning, oncology and telehealth.

### Participants

#### Online survey

Ninety-three toolkit recipients clicked on the survey link; however, only 65 of these went on to respond to at least some of the survey questions (table 1).

View this table:
[Table 1](http://bmjopenquality.bmj.com/content/7/4/e000286/T1)

Table 1 
Survey sample by profession (frequencies)

Forty-three respondents provided qualitative data to open response questions.

#### Telephone interviews

As the aim was to explore the perspectives of healthcare professionals from across Australia, purposeful stratified sampling38 by state and profession was used to ensure a diversity of views were captured. Only doctors, nurses and allied health professionals were invited to participate in the interviews. Sampling continued until 10 interviews had been conducted. All interviewees were female; seven were nurses and three were allied health professionals. Interviewees were for five different states. The mean age of interviewees was 48.0 years (range 35–62 years) and the mean years of experience as a healthcare professional was 23.7 years (range 13–37 years). Interviews lasted 20 min on average (range 10–40 min) (table 2).

View this table:
[Table 2](http://bmjopenquality.bmj.com/content/7/4/e000286/T2)

Table 2 
Telephone interview sample by profession (frequencies)

### Materials

#### Survey

The first section of the survey contained demographic questions, regarding age, gender, postcode, occupation, years of experience and primary work area. Respondents were then asked 10 questions on the relevancy, content, presentation and usability of the toolkit. They were asked whether using the toolkit had impacted on their confidence in dealing with end-of-life matters and if they had implemented any actions from the toolkit. Responses to these 10 items were on a 5-point Likert scale (strongly agree to strongly disagree). Examples of items are ‘The content of My Toolkit is relevant to me personally’ and ‘Having My Toolkit, I now feel more confident in discussing end-of-life matters with patients’. The remainder of the survey consisted of open response questions for participants to detail changes in workplace behaviour already implemented or planned.

#### Telephone interview

A semistructured interview schedule was developed which covered four broad topic areas with question prompts under each heading. These were (1) introduction/demographic data, (2) the relevance and quality of the materials and resources in the toolkit to their professional practice, (3) whether they believed the toolkit supports positive and effective changes to workplace practice with regard to end-of-life care, and (4) any other comments or feedback.

### Procedure

All toolkit recipients were given 2 weeks to complete the survey. Given that reminders have been shown to improve survey response rates,39 40 a reminder email was sent to all toolkit recipients after 1 week. There were 23 email delivery failures. In total, 65 completed the survey, a response rate of 16%.

Invitations to participate in a telephone interview were sent out concurrently with the survey link. Interview reminders were not sent. Participants formally agreed to participate in the telephone study and to have their interview audio-recorded.

### Analysis

Online responses to the survey were collected via the CareSearch41 Research Data Management System. The data were imported into SPSS V.23 for descriptive analysis. Content analysis to identify key topics of discussion was conducted on interview data, coding directly from the audio files.42 The content analysis was guided by the evaluation aims.

## Results

In this section we outline the findings against the two evaluation questions: (1) users’ reaction to the toolkit and (2) self-reported changes to workplace behaviours as a result of the toolkit.

### Users’ reactions to the toolkit

The survey data showed that users responded positively to the toolkit in terms of relevancy, content and presentation (table 3—items 1–4). For example, 93.7% of the respondents agreed or strongly agreed that the toolkit was relevant to their profession and 92.1% that the toolkit was relevant to them personally (table 3).

View this table:
[Table 3](http://bmjopenquality.bmj.com/content/7/4/e000286/T3)

Table 3 
Survey respondents’ views on *My Toolkit* (%)

In the open response questions, many respondents also reported that they found the toolkit to be clearly laid out, easy to read, well organised and well written, as highlighted in the following quote:

> Clear concise, well laid out and easy to negotiate. Not full of jargon, not cluttered. (State and occupation not reported, Survey Open Response)

The majority of respondents reported that, following use of the toolkit, they felt more confident in discussing end-of-life matters with colleagues, and patients and their families (table 3—items 5 and 6). Overall, 93.6% reported that they would recommend the toolkit to their colleagues, and indeed 82.2% reported that they had already recommended the toolkit to at least one colleague (table 3—items 9 and 10). These findings were also echoed in the survey open responses. For example:

> I have recommended this toolkit to be completed by staff working in acute settings across our local health district and also promoted it with Clinical Nurse Educators. (Nurse, New South Wales, Survey Open Response)

Furthermore, two of the ten interview participants were introduced to the toolkit via a colleague’s recommendation. As with survey respondents, most interview participants were very positive about the toolkit and the potential impact it could have on their practice:

> I think it’s the best thing I have ever read really…as an educational tool…it’s really considered, easy to read, easy to understand, easy to follow…and the resources that you suggested for further exploration is bang on the money, so you are not wasting my time. (Interview 1, Nurse, Australian Capital Territory)

Other participants also found the toolkit to be a valuable resource. Notably, participants found the toolkit format novel, and they appreciated having a resource that was Australian-specific:

> I’ve never seen another toolkit like this…this is a really, really good start…I am not really aware of any Australia stuff, like this otherwise out there. If we can start on this and work on it, I think it’s a great idea. (Interview 4, Clinical Nurse Manager, Queensland)

A participant from a culturally and linguistically diverse background (Interview 6, Social Worker, Victoria) particularly valued the actions and resources in support of cultural sensitivity. In her own personal and professional experiences, she reported that these skills had been lacking among healthcare professionals. Some participants thought the toolkit was more aimed at nurses than allied health professionals or doctors. However, a social worker reported: *“I felt the toolkit was talking to me”* (Interview 6, Social Worker, Victoria). This affirms the relevancy of the materials to different professions within the target audience for the toolkit resource.

### Self-reported changes to workplace behaviour as a result of the toolkit

The primary aim of the toolkit was to change workplace behaviour in relation to end-of-life care. However, given the 4-week follow-up period and available resources, we relied on the self-report of participants to indicate whether they had made any changes to their workplace behaviour or intended to do so. We asked respondents to provide examples of changes to their behaviour and/or intended changes. Overall, 59.3% reported that they had already implemented at least one behaviour change. Of those who had not yet implemented a change, 70.8% reported that they intended to in the near future.

The majority of examples of behaviour change implemented were in the area of communicating about end-of-life matters with colleagues, patients and their families. In the following quote, the respondent first engaged in end-of-life discussions with colleagues, facilitated by the toolkit, before initiating discussions with a patient and their family:

> A recent dying patient was the perfect time to open the toolkit, show colleagues, discuss actions to improve our communication planning and care…I found the toolkit an excellent resource. We initiated the conversations with the patient and his family…everyone involved had their input and needs recorded and acknowledged. After his peaceful and dignified death there were no dramas, no unexpected unhappiness…participation and involvement to each individuals comfort level was excellent. (State and profession not reported, Survey Open Response)

Interview participants also saw communication regarding end-of-life matters as the major area in which the toolkit could support quality end-of-life care practice. Participants observed a great amount of discomfort and denial about the end of life and a tendency to push difficult conversations onto others, as discussed in the following quote:

> ‘No one dies on my shift’…I hear that a lot…[I hear] ‘the social worker deals with that kind of thing. I’ll get the social worker to come and speak to you’…those on the spot questions: ‘Am I going to die?’, ‘Oh Hang on; I’ll get the social worker’. It’s not appropriate. (Interview 3, Nurse, New South Wales)

The toolkit was intended to be a personal resource; however, against expectation, some respondents reported that they were using the toolkit as an educational tool, or intended to use the resource for educational purposes in the future. For example:

> We are currently in the process of organising staff training sessions with the toolkit to educate all doctors and nurses in the department so there is consistency throughout the department. We have some consultants on board who are keen on the process who I have referred to your toolkit also. (Nurse, Victoria, Survey Open Response)

Several of the interview participants also reported that they saw value in the toolkit as an educational resource. Even though a couple of the interviewees felt that the content of the toolkit was too basic for their own needs, they saw it as an invaluable resource for more junior staff or those that were not routinely involved in palliative care. For example, interviewee 7 felt that the Toolkit *“wasn’t extending my practice”* (Interview 7, Nurse, Victoria) as a palliative professional with 34 years’ experience. However, she reported that she would use the toolkit when providing on-the-job education to graduate nurses, and that she *“would definitely recommend it to junior staff or ward staff”* (Interview 7, Nurse, Victoria).

However, other participants reported that the toolkit had a lot to offer a wide cross section of healthcare roles and varying lengths of experience. One participant advised:

> We are not just going to push it [Toolkit] through [the] emergency [department], we are going to push it through the hospital…[I have] managed to get people from a lot of different departments…interested in the topic at the moment…starting on the floor within an ED [Emergency Department] as a trial and filtering out into other departments. I’m excited to get working on the change that is going to happen…it is good to see the consultants jumping on board and it’s not just the nurses. (Interview 9, Nurse, Victoria)

As such, this participant envisaged using the toolkit to bring about positive changes to end-of-life care hospital-wide.

Survey respondents identified a number of intended future actions, including initiating end-of-life conversations with patients, family members and colleagues, sharing and discussing the toolkit at team meetings and workshops, recommending and promoting the toolkit to others, implementing better self-care strategies, and using the toolkit in formal training sessions.

## Discussion

The evaluation findings demonstrate that the toolkit was well received by users, with most finding that it was relevant to both their profession and their current role. Importantly users reported an increase in confidence in discussing end-of-life matters with colleagues, patients and families. This is an important outcome as psychological research has shown that confidence is a prerequisite to behaviour change.43 44 

One aim of the evaluation was to identify the type of changes to workplace behaviour being reported by toolkit users. As with other research into end-of-life training needs, many users reported communication around end-of-life care as a major area of deficit among healthcare professionals19–22 and found initiating discussions around end-of-life care challenging.6–11 However, many users advised that the toolkit was helping to address this deficit, with behaviour change around communication being the most frequently self-reported change by respondents. These examples demonstrated that users were focusing on spending time discussing patients’ wishes, involving family members, and ultimately supporting the dignified end of life for those in their care. However, it should be noted that follow-up was conducted only 4 weeks after receiving the toolkit. It is important to conduct additional follow-up to identify if workplace behaviour has been maintained33 and whether changes positively impact on patients’ experience of end-of-life care.

However, it is important to note that almost 60% of users reported they had implemented a change to their own workplace practice in 4 weeks or less. Of those who had not yet implemented a change—perhaps due to only having recently received the resource—most reported they were planning on making a change to their workplace behaviour in the near future. The theory of planned behaviour posits that intention is the strongest predictor of future behaviour change,45 a theory that has been well supported in the literature (eg, see McEachan et al.46 for a meta-analysis of studies on planned behaviour). This early evaluation evidence therefore indicates that the toolkit has the potential to effect workplace change at the practitioner level.

Notably, two themes emerged from the evaluation that were not anticipated. First, although the toolkit was intended to be a personal resource, the survey and interview data both showed that it was being used as an educational resource both within work units and with healthcare students and recent graduates. One interview participant noted that there was gathering momentum to use the toolkit across the whole hospital and she was personally engaged in this project, starting with the own department. As such, there are indications that the toolkit is being used to improve quality end-of-life care beyond the practitioner level to team and organisational levels.

Second, although the toolkit was intended to be used by doctors, nurses and allied health professionals, we found it was being used by a wider audience; 9.3% of requesters were not doctors, nurses or allied health; they were educators, managers, volunteers, aged care workers, pastoral support workers, policy makers, project officers, a terminally ill patient and an end-of-life doula.

Overall, 15% of survey respondents were aged care workers or healthcare professionals working in the aged care sector. These participants and others saw the toolkit as potentially useful to aged care workers in both community and residential settings. Currently, more than 240 000 workers are employed in direct care worker roles across the residential and community sectors.47 Given that patients nearing the end of life may shift between residential or community settings and acute hospital care,6 extending the toolkit resources to this sector could potentially extend the utility of this resource.

Finally, it is worth acknowledging that the timeframe for the evaluation was more contracted than anticipated. It was hoped that the evaluation would have been extended over a longer period in order to potentially capture a larger sample and a wider range of workplace behaviour changes. This shortened timeframe was due to development issues in late 2016 resulting in additional recruitment to develop the toolkit. Therefore, the evaluation of changes to workplace behaviours following receipt of the toolkit was only able to be established over a maximum period of 4 weeks. Despite this, 59.3% of respondents had made a change to workplace behaviour. That many recipients were able to make behavioural changes within such a short time period is a good indicator of the usability and potential effectiveness.

As with all evaluation studies, there are some notable limitations to this study. First, the sample was self-selected and may reflect a subsample of the target population who have a particular interest in end of life not shared with the target population at large. Second, the sample size was relative small at 16% and therefore may not be representative, although studies have shown that low response rates are not uncommon among healthcare professionals—especially online surveys—due primarily to lack of time.48 49 Third, changes to workplace behaviour were based on self-report and therefore could not be externally validated. Furthermore, the effectiveness of changes to workplace behaviours on patients’ experiences of their end-of-life care is important but was beyond the scope of the current study. Further evaluation should incorporate longer term monitoring of behaviour change and measures of patient experience.

## Conclusions

The toolkit has been well received by users, with many recommending the resource to colleagues. The evaluation data demonstrate that the toolkit had resulted in self-reported changes to workplace practice, with the vast majority of toolkit users reporting that the toolkit made them feel more confident in addressing end-of-life issues with patients and their families. While the toolkit was intended as a resource for individual use in acute hospital settings, we note that the toolkit has had broader appeal both in terms of its applicability in other settings and to other user groups. Some extension of materials would be expected to broaden the appeal of toolkit to other groups such as aged care workers, who represent a significant workers’ population providing care to Australians ageing and nearing the end of life. The toolkit has also been used as an education tool in numerous settings. Finally, the toolkit provides a new resource within the sector, and the Australian-specific nature of the resource has been highly valued by users.

## Acknowledgments

We acknowledge the generous contribution of time and expertise by the members of the Advisory Group, and thank the many members of the clinical community who donated their time to the review of content, resources and products associated with this End-of-Life Essentials Project.

## Footnotes

*   Contributors The toolkit and its evaluation was conceptualised by JT and KD. The toolkit was developed by CH with feedback from JT and KD, as well as a panel of medical experts. CH developed the evaluation materials, collected and analysed the data. Agreement on the themes and key messages was by consensus of all authors. The paper was drafted by CH. All authors commented, reviewed and approved the final draft.

*   Funding The End-of-Life Essentials Project is funded by the Australian Government Department of Health.

*   Competing interests CH and KD have received salary funding from the project.

*   Patient consent Not required.

*   Ethics approval Ethics approval for this project was provided by the Flinders University Social and Behaviour Ethics Committee (7568).

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement Data access can be requested by contacting the corresponding author.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  1. Foreman LM , Hunt RW , Luke CG , et al . Factors predictive of preferred place of death in the general population of South Australia. Palliat Med 2006;20:447–53.[doi:10.1191/0269216306pm1149oa](http://dx.doi.org/10.1191/0269216306pm1149oa) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1191/0269216306pm1149oa&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=16875116&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000238438300009&link_type=ISI) 

2.  2.Australian Senate. Senate community affairs reference committee: palliative care in Australia. Canberra: Commonwealth of Australia, 2012. (accessed 12 Jun 2017).
    
    

3.  3.Australian Commission on Safety and Quality in Health Care. National Consensus Statement: essential elements for safe and high quality end-of-life care. Sydney: ACSQHC, 2015. (accessed 20 Jul 2017).
    
    

4.  4. Broad JB , Gott M , Kim H , et al . Where do people die? An international comparison of the percentage of deaths occurring in hospital and residential aged care settings in 45 populations, using published and available statistics. Int J Public Health 2013;58:257–67.[doi:10.1007/s00038-012-0394-5](http://dx.doi.org/10.1007/s00038-012-0394-5) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1007/s00038-012-0394-5&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=22892713&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000316760300010&link_type=ISI) 

5.  5.Australian Institute of Health and Welfare. Australia’s hospitals 2009–10: at a glance. Health services series no.39 Cat. no. HSE 106. Canberra: AIHW, 2011. (accessed 6 Aug 2017).
    
    

6.  6. Swerissen H , Duckett SJ . What can we do to help Australians die the way they want to? Med J Aust 2015;202:10–11.[doi:10.5694/mja14.01580](http://dx.doi.org/10.5694/mja14.01580) 
    
    

7.  7. Cox K , Moghaddam N , Almack K , et al . Is it recorded in the notes? Documentation of end-of-life care and preferred place to die discussions in the final weeks of life. BMC Palliat Care 2011;10:18.[doi:10.1186/1472-684X-10-18](http://dx.doi.org/10.1186/1472-684X-10-18) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1186/1472-684X-10-18&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=22053810&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

8.  8. Emanuel EJ , Fairclough DL , Wolfe P , et al . Talking with terminally ill patients and their caregivers about death, dying, and bereavement: is it stressful? Is it helpful?  Arch Intern Med. 2004;164–2004.
    
    

9.  9. Keating NL , Landrum MB , Rogers SO , et al . Physician factors associated with discussions about end-of-life care. Cancer 2010;116:998–1006.[doi:10.1002/cncr.24761](http://dx.doi.org/10.1002/cncr.24761) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1002/cncr.24761&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=20066693&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000274315800030&link_type=ISI) 

10. 10. Parker SM , Clayton JM , Hancock K , et al . A systematic review of prognostic/end-of-life communication with adults in the advanced stages of a life-limiting illness: patient/caregiver preferences for the content, style, and timing of information. J Pain Symptom Manage 2007;34:81–93.[doi:10.1016/j.jpainsymman.2006.09.035](http://dx.doi.org/10.1016/j.jpainsymman.2006.09.035) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1016/j.jpainsymman.2006.09.035&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=17531434&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000247985000013&link_type=ISI) 

11. 11. Sulmasy DP , Sood JR , Ury WA . Physicians' confidence in discussing do not resuscitate orders with patients and surrogates. J Med Ethics 2008;34:96–101.[doi:10.1136/jme.2006.019323](http://dx.doi.org/10.1136/jme.2006.019323) 
    
    [Abstract/FREE Full Text](http://bmjopenquality.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToibWVkZXRoaWNzIjtzOjU6InJlc2lkIjtzOjc6IjM0LzIvOTYiO3M6NDoiYXRvbSI7czoyNDoiL2JtanFpci83LzQvZTAwMDI4Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

12. 12. Wolf JH , Wolf KS . The Lake Wobegon effect: are all cancer patients above average? Milbank Q 2013;91:690–728.[doi:10.1111/1468-0009.12030](http://dx.doi.org/10.1111/1468-0009.12030) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/1468-0009.12030&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=24320166&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

13. 13.Australian Commission on Safety and Quality in Health Care. Safety and quality of end-of-life care in acute hospitals: a background paper. 2013 [https://www.safetyandquality.gov.au/wpcontent/uploads/2013/09/SAQ083\_End\_of\_life\_care\_V15\_FILM\_REVISED\_TAGGED.pdf](https://www.safetyandquality.gov.au/wpcontent/uploads/2013/09/SAQ083\_End\_of\_life\_care\_V15_FILM_REVISED_TAGGED.pdf) (accessed 15 June 2017).
    
    

14. 14. Al-Qurainy R , Collis E , Feuer D . Dying in an acute hospital setting: the challenges and solutions. Int J Clin Pract 2009;63:508–15.[doi:10.1111/j.1742-1241.2008.01991.x](http://dx.doi.org/10.1111/j.1742-1241.2008.01991.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1742-1241.2008.01991.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=19222636&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000262953500023&link_type=ISI) 

15. 15.New South Wales Department of Health. Conflict resolution in end of life settings CRELS0 Final CRELS project working group report. 2009 [http://www.health.nsw.gov.au/patients/acp/Pages/conflict-resolution.aspx](http://www.health.nsw.gov.au/patients/acp/Pages/conflict-resolution.aspx) (accessed 4 May 2017).
    
    

16. 16. Field D , Wee B . Preparation for palliative care: teaching about death, dying and bereavement in UK medical schools 2000-2001. Med Educ 2002;36:561–7.[doi:10.1046/j.1365-2923.2002.01232.x](http://dx.doi.org/10.1046/j.1365-2923.2002.01232.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1046/j.1365-2923.2002.01232.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=12047672&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000175907500015&link_type=ISI) 

17. 17. Gibbins J , McCoubrie R , Maher J , et al . Incorporating palliative care into undergraduate curricula: lessons for curriculum development. Med Educ 2009;43:776–83.[doi:10.1111/j.1365-2923.2009.03400.x](http://dx.doi.org/10.1111/j.1365-2923.2009.03400.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2923.2009.03400.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=19659491&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

18. 18. Gibbins J , McCoubrie R , Forbes K . Why are newly qualified doctors unprepared to care for patients at the end of life? Med Educ 2011;45:389–99.[doi:10.1111/j.1365-2923.2010.03873.x](http://dx.doi.org/10.1111/j.1365-2923.2010.03873.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2923.2010.03873.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=21401687&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

19. 19. Johnson A , Chang E , O’Brien L . Nursing the dying: a descriptive survey of Australian undergraduate nursing curricula. Int J Nurs Pract 2009;15:417–25.[doi:10.1111/j.1440-172X.2009.01790.x](http://dx.doi.org/10.1111/j.1440-172X.2009.01790.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1440-172X.2009.01790.x&link_type=DOI) 

20. 20. Lloyd-Williams M , MacLeod RD . A systematic review of teaching and learning in palliative care within the medical undergraduate curriculum. Med Teach 2004;26:683–90.[doi:10.1080/01421590400019575](http://dx.doi.org/10.1080/01421590400019575) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1080/01421590400019575&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=15763870&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000226483100003&link_type=ISI) 

21. 21. Mallory JL . The impact of a palliative care educational component on attitudes toward care of the dying in undergraduate nursing students. J Prof Nurs 2003;19:305–12.[doi:10.1016/S8755-7223(03)00094-2](http://dx.doi.org/10.1016/S8755-7223(03)00094-2) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1016/S8755-7223(03)00094-2&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=14613070&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000186834800011&link_type=ISI) 

22. 22. Bekkema N , de Veer AJ , Albers G , et al . Training needs of nurses and social workers in the end-of-life care for people with intellectual disabilities: a national survey. Nurse Educ Today 2014;34:494–500.[doi:10.1016/j.nedt.2013.07.018](http://dx.doi.org/10.1016/j.nedt.2013.07.018) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1016/j.nedt.2013.07.018&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=23972684&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

23. 23. Frey R , Gott M , Raphael D , et al . Clinical staff perceptions of palliative care-related quality of care, service access, education and training needs and delivery confidence in an acute hospital setting. BMJ Support Palliat Care 2014;4:381–9.[doi:10.1136/bmjspcare-2012-000346](http://dx.doi.org/10.1136/bmjspcare-2012-000346) 
    
    [Abstract/FREE Full Text](http://bmjopenquality.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiYm1qc3BjYXJlIjtzOjU6InJlc2lkIjtzOjc6IjQvNC8zODEiO3M6NDoiYXRvbSI7czoyNDoiL2JtanFpci83LzQvZTAwMDI4Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

24. 24. Pulsford D , Jackson G , O’Brien T , O’Brien T , et al . Classroom-based and distance learning education and training courses in end-of-life care for health and social care staff: a systematic review. Palliat Med 2013;27:221–35.[doi:10.1177/0269216311429496](http://dx.doi.org/10.1177/0269216311429496) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1177/0269216311429496&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=22126845&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

25. 25. Selman L , Robinson V , Klass L , et al . Improving confidence and competence of healthcare professionals in end-of-life care: an evaluation of the ’Transforming End of Life Care' course at an acute hospital trust. BMJ Support Palliat Care 2016;6:231–6.[doi:10.1136/bmjspcare-2015-000879](http://dx.doi.org/10.1136/bmjspcare-2015-000879) 
    
    [Abstract/FREE Full Text](http://bmjopenquality.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiYm1qc3BjYXJlIjtzOjU6InJlc2lkIjtzOjc6IjYvMi8yMzEiO3M6NDoiYXRvbSI7czoyNDoiL2JtanFpci83LzQvZTAwMDI4Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

26. 26. Shifrin MM . An evidence-based practice approach to end-of-life nursing education in intensive care units. J Hospice Palliat Nurs 2016;18:342–8.[doi:10.1097/NJH.0000000000000254](http://dx.doi.org/10.1097/NJH.0000000000000254) 
    
    

27. 27. Gillan PC , Jeong S , van der Riet PJ . End of life care simulation: a review of the literature. Nurse Educ Today 2014;34:766–74.[doi:10.1016/j.nedt.2013.10.005](http://dx.doi.org/10.1016/j.nedt.2013.10.005) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=24183632&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

28. 28. Lippe MP , Becker H . Improving attitudes and perceived competence in caring for dying patients: an end-of-life simulation. Nurs Educ Perspect 2015;36:372–8.[doi:10.5480/14-1540](http://dx.doi.org/10.5480/14-1540) 
    
    

29. 29. Gerkin KL , Taylor TH , Weatherby FM . The perception of learning and satisfaction of nurses in the online environment. J Nurses Staff Dev 2009;25:E8–E13.[doi:10.1097/NND.0b013e318194b6a4](http://dx.doi.org/10.1097/NND.0b013e318194b6a4) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=19182548&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

30. 30. Ray K , Berger B . Challenges in healthcare education: a correlational study of outcomes using two learning techniques. J Nurses Staff Dev 2010;26:49–53. quiz 54-5.[doi:10.1097/NND.0b013e3181d4782c](http://dx.doi.org/10.1097/NND.0b013e3181d4782c) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=20354404&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

31. 31. Brady D , Molzen S , Graham S , et al . Using the synergy of online education and simulation to inspire a new model for a community critical care course. Crit Care Nurs Q 2006;29:231–6.[doi:10.1097/00002727-200607000-00007](http://dx.doi.org/10.1097/00002727-200607000-00007) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=16862024&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

32. 32. Petty J . Interactive, technology-enhanced self-regulated learning tools in healthcare education: a literature review. Nurse Educ Today 2013;33:53–9.[doi:10.1016/j.nedt.2012.06.008](http://dx.doi.org/10.1016/j.nedt.2012.06.008) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1016/j.nedt.2012.06.008&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=22818225&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

33. 33. Chant S , Tim T , Randle J , et al . Communication skills training in healthcare: a review of the literature. Nurse Educ Today 2002;22:189–202.[doi:10.1054/nedt.2001.0690](http://dx.doi.org/10.1054/nedt.2001.0690) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=12027600&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

34. 34. Eraut M . 2.1 Transfer of knowledge between education and workplace settings: a critical perspective. London: Routledge, 2009:65–84.
    
    

35. 35. Heaven C , Clegg J , Maguire P . Transfer of communication skills training from workshop to workplace: the impact of clinical supervision. Patient Educ Couns 2006;60:313–25.[doi:10.1016/j.pec.2005.08.008](http://dx.doi.org/10.1016/j.pec.2005.08.008) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1016/j.pec.2005.08.008&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=16242900&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000236245500008&link_type=ISI) 

36. 36.National Palliative Care Strategy. Supporting Australians to live well at the end of life: commonwealth of Australia. 2010 [https://www.health.gov.au/internet/main/publishing.nsf/Content/EF57056BDB047E2FCA257BF000206168/$File/NationalPalliativeCareStrategy.pdf](https://www.health.gov.au/internet/main/publishing.nsf/Content/EF57056BDB047E2FCA257BF000206168/$File/NationalPalliativeCareStrategy.pdf) (accessed 8 Aug 2017).
    
    

37. 37.Royal College of General Practitioners. Palliative and End of Life Care Toolkit. 2018 [http://www.rcgp.org.uk/clinical-and-research/resources/toolkits/palliative-and-end-of-life-care-toolkit.aspx](http://www.rcgp.org.uk/clinical-and-research/resources/toolkits/palliative-and-end-of-life-care-toolkit.aspx) (accessed 17 Aug 2018).
    
    

38. 38. Palinkas LA , Horwitz SM , Green CA , et al . Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health 2015;42:533–44.[doi:10.1007/s10488-013-0528-y](http://dx.doi.org/10.1007/s10488-013-0528-y) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1007/s10488-013-0528-y&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=24193818&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 

39. 39. Shih T-H , Fan X . Comparing response rates in e-mail and paper surveys: a meta-analysis. Educ Research Review 2009;4:26–40.[doi:10.1016/j.edurev.2008.01.003](http://dx.doi.org/10.1016/j.edurev.2008.01.003) 
    
    

40. 40. Snowball C . How to improve survey response rates. Incite 2010;31:24.
    
    

41. 41. Tieman J . Ensuring Quality in Online Palliative Care Resources. Cancers 2016;8:113.[doi:10.3390/cancers8120113](http://dx.doi.org/10.3390/cancers8120113) 
    
    

42. 42. Elo S , Kyngäs H . The qualitative content analysis process. J Adv Nurs 2008;62:107–15.[doi:10.1111/j.1365-2648.2007.04569.x](http://dx.doi.org/10.1111/j.1365-2648.2007.04569.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2648.2007.04569.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=18352969&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000254192000012&link_type=ISI) 

43. 43. Bandura A . Self-efficacy: the exercise of control: Macmillan, 1997.
    
    

44. 44. Brown RL , Pfeifer JM , Gjerde CL , et al . Teaching patient-centered tobacco intervention to first-year medical students. J Gen Intern Med 2004;19:534–9.[doi:10.1111/j.1525-1497.2004.30144.x](http://dx.doi.org/10.1111/j.1525-1497.2004.30144.x) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1111/j.1525-1497.2004.30144.x&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=15109319&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000221452500011&link_type=ISI) 

45. 45. Ajzen I . From intentions to actions: a theory of planned behavior: action control: Springer, 1985:11–39.
    
    

46. 46. McEachan RRC , Conner M , Taylor NJ , et al . Prospective prediction of health-related behaviours with the Theory of Planned Behaviour: a meta-analysis. Health Psychol Rev 2011;5:97–144.[doi:10.1080/17437199.2010.521684](http://dx.doi.org/10.1080/17437199.2010.521684) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1080/17437199.2010.521684&link_type=DOI) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000297300600001&link_type=ISI) 

47. 47. King D , Mavromaras K , Wei Z , et al . The aged care workforce – final report. 2012 [http://www.agedcarecrisis.com/images/pdf/The\_Aged\_Care\_Workforce\_Report.pdf](http://www.agedcarecrisis.com/images/pdf/The\_Aged_Care_Workforce_Report.pdf) (accessed 28 May 2017).
    
    

48. 48. VanGeest JB , Johnson TP , Welch VL . Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof 2007;30:303–21.[doi:10.1177/0163278707307899](http://dx.doi.org/10.1177/0163278707307899) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1177/0163278707307899&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=17986667&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom) 
    
    [Web of Science](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=000250807600001&link_type=ISI) 

49. 49. Cunningham CT , Quan H , Hemmelgarn B , et al . Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol 2015;15:32.[doi:10.1186/s12874-015-0016-z](http://dx.doi.org/10.1186/s12874-015-0016-z) 
    
    [CrossRef](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=10.1186/s12874-015-0016-z&link_type=DOI) 
    
    [PubMed](http://bmjopenquality.bmj.com/lookup/external-ref?access_num=25888346&link_type=MED&atom=%2Fbmjqir%2F7%2F4%2Fe000286.atom)