Article Text
PDF
Abstract
In healthcare settings, workplace learning is often supported by clinicians who strive to combine service provision and educator roles. We evaluated an international 12-month programme that supports widely distributed critical care health professional educators (HPEs) through a virtual community of practice (vCoP). Specifically, we evaluate if and how the vCoP approach affects learning experiences using an innovative evaluation framework in medical education—the value-creation framework (VCF). We used a mixed-methods approach to evaluation, including an anonymous survey and semistructured interviews. Themes from data sources were identified using the VCF as the common thread. Themes discussed by at least two-thirds of interview participants were analysed using narrative inquiry. 27 of 66 participants responded to the survey, and 15 participated in interviews. Positive and negative indicators of value creation were extracted and organised according to the framework’s eight value cycles. Framework analysis made value-creation and potential flow-on effects in one value-creation cycle to another visible, offering insight into relationships. Themes from narrative inquiry elaborated on the results of the framework analysis. Using the VCF to evaluate the Incubator programme brings to bear the complexity of boundary-crossing HPE faculty development for critical care educators. The framework can be a valuable tool for evaluating a vCoP associated with faculty development programmes.
- Medical education
- Continuing education, continuing professional development
- Health professions education
- Social sciences
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- Medical education
- Continuing education, continuing professional development
- Health professions education
- Social sciences
WHAT IS ALREADY KNOWN ON THIS TOPIC
The value of faculty development programmes (FDPs) that include a community of practice (CoP) aspect is an under-researched phenomenon.
Evaluation tends to focus on developing short-term knowledge gains without addressing identity formation.
The few existing studies highlight the importance and interconnectedness of personal, relational and contextual domains of an educator’s identity.
WHAT THIS STUDY ADDS
This is the first study that uses the value-creation framework (VCF) to evaluate virtual communities of practice and highlights the complexities of identity formation of clinical educators.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
This study outlines an evaluation blueprint for practitioners who want to think holistically about the effectiveness of educational interventions.
Introduction
In healthcare settings, workplace learning is often supported by clinicians who strive to combine service provision and educator roles. In the Australian and New Zealand critical care field, clinicians interested in education are not necessarily trained to educate or might not have the resources to develop these skills. They may also be geographically dispersed, which poses challenges for the delivery of continuing professional development. Community of practice (CoP) is a delivery mode that could support the development of the clinician’s health professional educator (HPE) role. However, evaluating CoPs is challenging because many common evaluation frameworks are ill-suited for measuring holistic impact. This article reports on a year-long faculty development programme (FDP)—Australian and New Zealand Clinician Educator Network (ANZCEN) Clinician Educator Incubator—which employs a virtual CoP approach.1 2 The Incubator aims to develop members’ scholarly teaching practice and the educator component of the clinical role.3 We report on how the value-creation framework (VCF) can be used to evaluate the effectiveness of CoP in clinical education.4 5
Communities of practice
The concept of CoP is based on the understanding that learning is a social enterprise and occurs in situ in established learning communities.6–8 The development of CoP relies on deliberate cultivation, which supports individual and collective learning.1 CoPs create value by guiding innovative processes to solve critical problems, facilitate the transfer of best practices, initiate collective learning and foster a sense of agency and identity for participants.8 9 Through knowledge-sharing and mentoring relationships, a CoP framework can support change towards best practice and enhance the culture of the community.6 10 11
Virtual CoPs allow practitioners who are isolated to access a wider community “to share knowledge, build relationships, and foster innovation” that is “independent of geography or time zones” (Yarris et al, p. 1).2 These qualities allow critical care educators to overcome difficulties such as limited local networks or conditions outside of their control (eg, the need for physical distance imposed by a pandemic). ANZCEN runs a year-long FDP—referred to as the Incubator—that employs a vCoP approach1 2 to develop members’ scholarly teaching practice.3 Although the concept of CoP for faculty development is increasingly applied to FDP, the best strategies for evaluating CoPs are less clear.6
Evaluation of CoPs
The value of FDPs that include a CoP aspect is an under-researched phenomenon. A CoP approach emphasises formal and informal interactions to connect people while fostering shared identity development. Educator identity plays an integral role in HPE motivation to engage in educational activities and participate in professional development, how satisfied they are in their educator roles and academic productivity.12–14 Yet the evaluation of FDPs tends to focus on developing teaching expertise without addressing identity formation. Moreover, the frameworks used could not fully evaluate whether and how FDPs—and communities that emerge from these programmes—add to a clinical role.
Regardless, some studies have attempted to capture the complexities of FDPs and how they could be evaluated.15–22 Studies that attempt to evaluate FDPs with a CoP component highlight that the community aspect was often a source of motivation and engagement.17 21 23–25 They also signal the importance and interconnectedness of personal, relational and contextual domains of an educator’s identity. A sense of belonging to a community and opportunities to share concerns, solutions or discuss ideas played a significant role in solidifying educator identity and participants’ willingness to take risks in their teaching and professional lives.17 24 However, the evaluation procedures used in these studies were resource-intensive, varied and challenging to replicate.
There remains a gap in research concerning interprofessional HPE CoPs and educator identity formation. Smith et al’s critical review of online or blended CoPs concluded that many of the studies lacked practical implications; only 3 out of 17 studies offered evaluative insights that could guide the design of professional development programmes.26 Other authors advocate for the inclusion of community building into the design of longitudinal professional development to foster the development of a dual educator identity —but again do not discuss how the success of community building exercises could be measured. A suitable evaluation framework is needed to account for the complexities of CoPs and their multifaceted nature.
The value-creation framework
The VCF,4 used in a number of studies in other fields, is a tool that can be used in clinical education to understand how communities create value for their members and stakeholders.4 27 Value creation—in an educational context—means the value of the learning enabled by community involvement. Value creation occurs when the community shares information and resources, learn from each other’s experiences, support each other and share additional opportunities with the community. The VCF has eight different cycles of value creation.5 The first four are adaptations of a model developed by Kirkpatrick for training evaluation.28 The additional four cycles were later added to measure the value added in communities and networks.4 5 The values are Immediate, Potential, Realised, Applied, Orienting, Enabling, Strategic and Transformative. The value creation cycles within the framework are presented in figure 1 and elaborated in table 1. The VCF may be a way to capture the complexities of how FDPs with CoPs can develop an educator’s scholarly practice and a dual educator identity. Using the framework may also provide insight into what hinders and enables individual and collective learning and what educators can do to proactively create value in their faculty development activities.4 5
Elaboration of the value-creation cycles in value creation framework5
Value-creation cycles in value creation framework.
This study aims to evaluate if and how a CoP approach to the Incubator FDP enhances the learning experiences of critical care educators, increases the participant’s scholarly teaching practice and develops the participant’s identity as a clinician educator.29
Method
Design
This study uses framework analysis—using the VCF—and narrative inquiry. We used a mixed-methods approach to evaluate the Incubator programme.30 The VCF was initially developed as an evaluation framework for social learning that integrates various data sources from participants, evaluators and stakeholders.4 27 The framework was updated with a broader view considering the various social learning modes.5 The VCF focuses on the value creation experience, acknowledging that what is regarded as value is often different for different participants.5 27 The VCF offers a useful approach to evaluating interventions where the effects of the interventions are not direct and obvious.5 This study uses the VCF as a theoretical and explanatory tool to synthesise qualitative and quantitative data.4 5
We couple this with narrative inquiry, which draws on participant storytelling to identify common themes.31 Stories are a way of making sense of an experience and articulating a complex view of oneself, including reflecting on identity and how individuals position themselves culturally and socially.32 ‘Narrative’ refers to the created structure of a story that has a beginning, middle and end, which can impact the individual and the collective present and future.32 33 The narrative inquiry method can uncover the experience, explore contextual elements and detect changes in individual and collective identities.33–35 Exploring the effectiveness of the Incubator’s CoP using both the VCF alongside a narrative inquiry approach allows us to explore both personal and collective narratives about ‘what a community is doing’ and ‘what counts as value for whom’.4 8
Setting
The setting of this study was ANZCEN’s first year-long Incubator programme. Programme activities occurred online via Slack platform from April 2020 to March 2021, when all participants were unexpectedly and significantly affected by the changed work and home environments from COVID-19 global pandemic. Participants of the Incubator programme consist of critical care educators from Australia, New Zealand, Canada, the USA and Colombia, with varying experience and postgraduate qualifications in health professions education. Those involved in the programme came from medical, nursing, allied health and academia professional groups. Participants were intended to meet in-person throughout the year but were unable to meet because of COVID-19 restrictions (table 2). Online participation was also significantly reduced. For instance, only two out of six ‘Mastermind’ groups met every month, and the quality and quantity of the Slack discussions were significantly reduced compared to pre-COVID offerings of the Incubator.
Incubator context before and during COVID-19 restrictions
Research team: bias and positionality statement
Three people comprise the research team and are listed authors on this paper. Author 1 (MG) is an intensive care specialist and emergency physician and was part of the leadership team of the Incubator programme at the time of data collection. She was supervised by authors 2 (PC-I) and 3 (TJ) during her candidature for a Master of Clinical Education degree at the University of Auckland. This paper is based on research conducted in fulfilment of that degree. Author 2 (PC-I) has a background in the social sciences and health sciences education. Author 3 (TJ) has a background in health sciences education, public health and anthropology. All authors were actively engaged in multiple CoP at the time of designing and conducting this research.
Participants
The target population of this study were participants involved in the Incubator programme, including learners (referred to as Incubatees), faculty members and the community leadership group (n=66). There were 30 Incubatees and 6 leadership group members from Australia and New Zealand. The remaining participants were 30 faculty members (local and international) who facilitated specific topics and groups.
Anonymous survey
After the completion of the programme, all members were invited to complete an anonymous online feedback form (online supplemental appendix A). Completing the online feedback form is part of the Incubator’s standard programme evaluation. The survey asked the participants to rate (5-point scale from negative to positive) how their participation in the Incubator programme affected them as a professional, affected their social connections, helped with their scholarly teaching practice and changed their ability to influence their professional culture. These four areas are chosen for evaluating change in aspects of a clinician educator29 and the participant as a member of the Incubator vCoP.4 5 The participant also had the opportunity to provide free text to elaborate on their ratings and give feedback on the Incubator programme.
Supplemental material
Semistructured interviews
The virtual one-on-one semistructured interviews (online supplemental appendix B) explored concrete examples of how the Incubator affected the participants, both positively and negatively, intending to capture the participant’s value-creation story. One-on-one interviews were used to elicit a personal description of participants’ lived experience of the Incubator programme.
Procedure
An anonymous survey was sent to all members of the Incubator programme. Participants who completed the survey had the option of participating in a semistructured interview. Participants who expressed interest in the interview were then approached directly. Participants were also recruited by advertisement through the Incubator network. Participants gave informed consent and could withdraw from the study at any time. Author 1 (MG) conducted the interviews.
Analysis
Qualitative and quantitative data were integrated using the VCF, using an analysis method of data merging in mixed-methods research.30 The data analysis process was iterative. One research team member (MG) collated, summarised and thematically coded all data by hand. Emergent themes from the anonymous surveys and one-on-one interviews were identified separately and coded in NVivo. Authors 2 and 3 (PC-I and TJ) provided feedback on the emerging themes and codes as they developed. We then used paradigmatic framework analysis31 33 to map emergent themes to the VCF, guided by statements reflecting the presence or absence of value indicators (online supplemental file 1). We also searched for themes that did not map to the framework. Lastly, overarching narratives from the framework analysis—discussed by at least two-thirds of the interview participants—were used to build a collective narrative about the value of the Incubator programme for informing community members’ scholarly teaching practices. These overarching narratives made the identified themes from the framework analysis more meaningful by anchoring them to the participants’ stories.31
Ethical approval
Ethical approval was obtained from the University of Auckland Human Participants Ethics Committee, Reference number UAHPEC2710.
Results
Of the study population of 66 participants, 27 (41%) responded to the anonymous survey, and 15 (23%) participated in the semistructured interviews. The participants’ roles within the Incubator programme and their teaching experience are depicted in table 3.
Participant role in the Incubator and years of teaching in a professional capacity—anonymous survey and semistructured interviews
From the anonymous survey, most participants reported that the Incubator programme positively affected them in four ways—as a professional; through social connection; by developing scholarly teaching practice; and by influencing their professional culture. One participant (faculty) reported a perceived negative impact on their status in the ability to influence their professional culture. In the free text entry section of the survey, the participant did not elaborate on the negative rating of their status and the participant’s responses to other areas were positive. The Incubatee group were more likely to report perceived positive change than the Faculty group. Scholarly teaching practice was the area with the most impact, with all Incubatee participants rating 4/5 or 5/5 on all four aspects of scholarly teaching.
The framework analysis of the study’s qualitative data—from the survey and interviews—showed that the Incubator programme created value, in various ways and to different extents, for the participants despite significant challenges due to the COVID-19 pandemic (table 2). The themes that emerged within each value-creation cycle are shown in figure 2, depicted as + for positive value creation and − for negative value within a theme.
The Incubator’s collective value-creation story showing potential contribution of one value-creation cycle to create value in other value-creation cycles.
The examples of quotes within the IMMEDIATE value themes, showing broad representation from different groups, are provided in online supplemental file 2. Figure 2 represents the Incubator’s collective value-creation story showing the potential contribution of one value-creation cycle—arrows depicting possible flow on effect—to create value in other value-creation cycles.
To provide clarity for the use of VCF and how the framework helps portray the impact of the Incubator on the participant, an example of an individual value-creation story is shown in figure 3. The story, told by an interview participant, is rearranged and placed within the value-creation cycles, showing the flow on effect of value-creation from one cycle to the next.
An example of a value-creation story showing the flow on effect of one value-creation cycle to the next.
Due to space limitations, we will only briefly describe our narrative analysis. We identified four overarching narratives, which include (1) experience of affirmation; (2) a sense of belonging; (3) getting new ideas; and (4) reframing view on education. The themes and subthemes from the narrative analysis, as well as supporting quotes, are presented in online supplemental file 3. The overarching narratives are interconnected in the participants’ evolving sense of identity as clinician educators. From the perspective of a vCoP, the narrative analysis suggests that the Incubator programme contributed to the development of participants’ educator identity. Participants also reported that developing an educator identity positively impacted their local educational contexts.
Discussion
To our knowledge, this is the first study to evaluate a HPE vCoP using the VCF. Our participant-centred data sources give credence to the perspectives and agency of the participants. FDPs using CoP approaches to promote educational scholarship have been described elsewhere2 22 36 37 and have been shown to be successful. However, due to the practicalities of evaluation, studies on FDPs focus on scholarly teaching using quantifiable productivity—often drawing on the number of scientific presentations and publications as a measure of success22 36—and self-report data.24 25 Moreover, these studies do not correlate scholarly teaching practice with observed teaching behaviours. Mapping findings to the value-creation cycles has shed light on ways to measure scholarly practices holistically. Below, we discuss the implications of our findings.
vCoP approach to HPE faculty development is feasible
The Incubator programme aims to transcend professional, disciplinary and geographical boundaries. We found two important dimensions to creating the right environment for participants: the structure and content of the programme; and participant connections and sharing of resources. The programme’s success was positively informed by varied and relevant learning resources and activities and a flexible, participant-centred structure to the course. During COVID-19 restrictions, and as the pandemic progressed, engagement with the Incubator decreased slightly. Yet, participants felt connected to the community through the Incubator despite feeling behind with the programme, and they reported the flexibility of the Incubator structure supported this sense of community. The study’s findings indicate that value was still being created despite an apparent lack of engagement. From the perspective of a vCoP, being deliberate about (re)creating the positive experience generated by the participants in the space will help a vCoP to thrive. A thriving vCoP requires members to find value in participating through social connection, validation, sharing learning resources and ideas, or solving problems together. ENABLING value creation is key to the Incubator vCoP’s sustainability. The leadership group play an important role in cultivating and sustaining the vCoP.
The time and effort required for creating and sustaining a functioning vCoP can be significant. For the programme to be sustainable, educational designers must consider what enables participants to learn and consider possible external barriers that may inhibit participation. Ideally, the participants’ involvement in the programme or community is seen and valued in their workplace. When participants feel supported in their local workplace their motivation to engage in the programme is increased.17 38 For example, when heads of units endorse participant involvement, this signals the value of the programme to others; having ‘protected’ time is usually necessary for successful longitudinal FDPs.24 25 This is not the case for critical care educators in Australasia. At present, the Incubator participants—faculty and Incubatees—do not have protected time to engage, affecting participants’ capacity to participate consistently.
The value creation framework is effective for successful evaluation of CoP in FDPs
The inclusion of community-building elements (CoP) into the design of longitudinal HPE professional development is rapidly becoming standard practice. This inclusion aligns well with educational theory,39 but is seldom matched with strong evaluation strategies. This study establishes that the VCF can be a useful tool for evaluating the added value of a complex intervention such as the vCoP approach to an HPE FDP.
The VCF’s eight value-creation cycles provide categories for evaluating a CoP through an exploration of value-creation—positive, negative or neutral (no value created)—as part of the CoP’s learning. This framework encourages the collection and integration of effect data (indicators of value-creation within a cycle—shown in online supplemental file 1) and contribution data (value-creation stories) to give an evaluation that is both plausible and trustworthy.4 5 For example, although the Incubator cannot claim to cause participant transformations, the richness of the narrative and the flow-on effect seen within the VCF strongly suggest that the community aspect of the Incubator contributed to the forming, development and solidifying of a dual educator identity. The framework helps make the value of the Incubator programme visible.4 5 40 Using the VCF to evaluate the Incubator programme brings to bear the complexity of boundary-crossing HPE faculty development for critical care clinical educators.
Limitations
This mixed-methods study evaluating the Incubator programme has several limitations. First, the findings are based on self-reported data, which is widely acknowledged as having potential limitations of bias and positionality. Second, the study population is small (n=66). Although our survey response rate was reasonable at 41%, it is reflecting the views and experiences of 27 only people, thus limiting us to descriptive analyses. The small sample size may hinder capturing the full spectrum of diverse perspectives or nuanced trends, potentially limits the study’s generalisability. To combat this limitation of the study, we interviewed more than half of the survey respondents (n=15). Interview participants were a self-selected group of motivated people. Author 1, who was part of the Incubator leadership group, conducted the interviews with participants, which may result in a social desirability bias (eg, participants may have overclaimed the programme’s positive effects). To offset author 1’s possible bias, authors 2 and 3—who have no experience of the Incubator programme—scrutinised the study design and findings throughout the data collection and analysis stages, to improve the credibility and trustworthiness of the findings. Lastly, using the VCF to guide the survey and interview questions may have resulted in narratives that over-report positive value and omit negative experiences.
Conclusion
A vCoP approach to the Incubator programme enhances critical care educator learning experiences, increases participants’ scholarly teaching practice and promotes the development of a dual educator identity for critical care clinicians. A vCoP approach to faculty development is feasible, although a significant effort is required. The VCF is useful for formal and informal evaluation of a CoP associated with FDPs. Others seeking to cultivate a similar HPE vCoP across professional, discipline and geographical boundaries can use a similar structure and process, paying attention to potential pitfalls outlined in the discussion. One of the key strengths of this study is the participants’ voices through the stories they have constructed and shared. Their stories speak to how a vCoP, such as the Incubator programme, can transform ways of understanding teaching practice, reimagine what it means to educate in critical care contexts, and affirm and celebrate educator identities. Such transformations offer countless benefits for critical care educators internationally and for the future of critical care education.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
Ethics approval for this research was obtained from the University of Auckland Human Participants Ethics Committee (Reference Number: UAHPEC2710. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors thank all faculty and clinicians who participated in this study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors MG, PC-I and TJ contributed to the study concept, study design and analysis. MG drafted the original manuscript and collected the data, and is the study guarantor. All authors contributed to subsequent revisions and editing.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.