Introduction
Reflective writing is one way that health professionals can document their professional growth, and has become an accepted method for undertaking or participating in continual professional development (CPD).Reference Chapman, Warren-Forward and Dempsey1,Reference French and Dowds2
Reflection and reflective writing can take shape in many different mediums, from guided journals, debriefs and dialoguing to poetry and art.Reference Turley3–Reference Waldo and Hermanns5 Journaling is often used to facilitate reflection in radiotherapy departments where staff and resources are often stretched.Reference Milinkovic and Field6,Reference Cole7 However, traditionally reflective writing is associated with freeform writing, where authors document professional experiences, personal thoughts and learning outcomes in unstructured and unguided journal or diary entries. There is no specific template in freeform writing and the triggers for writing come from personal experience.
The literature provides a broad range of examples within health science where reflective writing has been used to support undergraduate students and qualified staff; such as nursing,Reference Doucet and Wilson8–Reference Fakude and Bruce11 physical therapy,Reference Williams, Wessel, Gemus and Foster-Seargant12,Reference Williams and Wessel13 occupational therapyReference Tryssenaar14 and dentistry.Reference Pee, Woodman, Fry and Davenport15
Recognised problems associated with unstructured freeform writing include that writing can be time consuming due to its lack of direction. Researchers have commented that authors are often unsure what to write and may feel awkward writing about personal issues. Barriers to reflective writing can include a lack of mentoring and support of practitioners wanting to engage in reflective writing for their CPD within the workplace,Reference Brackenridge16 as well as the large workload that clinicians and students already carry, limiting the time to engage with freeform writing. Reflective writing is often seen as having less priority assigned than other professional development activities. The problems associated with freeform writing lend itself to the development and investigation of guided methods of short-form writing.
To assist practitioners to better develop their reflective writing skills, it is important that they are provided with feedback on their writing in the form of evaluation or assessment. This may be in the form of self-evaluation, formal or informal assessment. In order to evaluate freeform or short-form guided reflection a simple and valid tool is required such as the Newcastle Reflective Analysis Tool (NRAT) For a full description of the development and theoretical underpinning of the NRAT please refer to the previous publication by Findlay et al.Reference Findlay, Dempsey and Warren-Forward17
In brief, the NRAT consists of two assessment tools termed the Deep Analytic NRAT and Broad Classification NRAT that can either be used as standalone tools or in combination as a two phase system. The Deep Analytic NRAT is a six level finely demarcated classification system that can be used for formative assessment of reflective writing in academic or research setting. The Broad Classification NRAT has three levels of reflection that can be used more readily in clinical setting or as part of a self-evaluation framework. The NRAT has been previously validated for use in assessing freeform writing for evidence of reflection. For full details of the validation process for use in assessment of freeform writing, please refer to Findlay et al.Reference Findlay, Dempsey and Warren-Forward18
This research reports the development of short-form written guided inventories, known as the Newcastle Reflective Inventories (NRIs) that facilitate the process of reflection for radiation therapists. The validation of the NRIs involved the use of the NRAT, which allowed the determination of the level of reflection documented in the short-form written inventories. The context for this development and validation was within the field of RT. The NRIs were implemented into the CPD protocol of a clinical radiation oncology centre within Australia and used by intern and qualified RT staff as part of their personal and departmental CPD.
This research aims to:
1. Describe the development of the Newcastle Reflective Inventories
2. Validate the NRAT as a tool to evaluate short-form reflective writing such as the NRIs and
3. Validate the NRIs as effective tools to assist practitioners reflective writing ability.
Method
Development of the NRIs
The NRIs consist of a series of questions that prompts or guides the user through the reflective writing process. To provide for the various professional development activities that health professionals may work within, NRIs have been developed for the following three situational contexts:
• post-workshop reflection
• significant clinical event reflection
• post-journal reading reflection
The NRIs were developed based on the authors wide reading of the literature on reflective writing in health care and Boud’sReference Boud, Walker and Keogh19 three stages and six levels of reflection (Table 1). For each of the six levels of reflection, expected outcomes were defined by developing short-form written descriptions of what a practitioner may document for any given situation, in a written reflective piece for each level of reflection, as stated in Boud’s model.
Table 1. Boud’s six levels of reflection Reference Boud, Walker and Keogh19
Questions were then written that could guide reflective responses (Table 2). These were developed by identifying examples of the various levels and outcomes of reflection within previously completed reflective writing of undergraduate RT students and creating the questions that would require a response similar to the example at each level. These questions were then complied into simple and concise inventories that could be used in either hard copy or electronic format. This process was followed for each of the three situational contexts listed above.
Table 2. An example of an expected outcome and required questions for the ‘Association’ level of reflection
Validation of the Deep Analytic NRAT for assessing NRIs
Two coders used the Deep Analytic NRAT to independently evaluate a set of completed NRIs. The NRIs used in the validation of the NRAT when assessing short-form guided writing were obtained from the CPD work of five intern radiation therapists. Each intern completed an NRI after attending a professional development workshop, reading a professional journal article or an experience in their clinical work that they felt was significant to them. No restrictions were placed on the stimulus for the significant clinical event, which could have been a clinical, technical, patient- or staff-focused event. The five interns simply documented their experience using the NRI appropriate to the context. The interns were able to complete the NRI in hard copy or electronic form. All NRIs were completed as part of routine CPD whilst working in the clinical environment over a period of 6 month.
Coder training in the use of the NRAT was provided to the independent coder before commencement of the analysis. Each coder then completed independent coding of the full set of NRIs. Absolute agreement and Cohen’s Kappa statistics were calculated for both the deep analytic and broad classification, results for coder agreement were compared with Wong’sReference Wong, Kember, Chung and Yan20 work.
Validation of the NRIs
The NRIs were validated by analysing the qualitative descriptors contained within the written responses of the NRIs and freeform journals using NRAT. Data were collected from the NRIs and freeform journals of six qualified radiation therapists with varying levels of experience, from recently qualified to very experienced senior staff. Accordingly the sample had large age and qualification variation.
Each was asked to read at least one article from a professional journal of their choice and consider a clinical event that they felt was significant or document their experience following a workshop or course they had attended.
Initially, six participants completed a freeform entry for each of the context. Immediately after completing the freeform journal the participant was provided with the appropriate NRI to complete using the same event as the stimulus. The writing within the NRI was not governed by academic prose or grammatical restrictions, allowing each participant to express themselves in a personal and unrestricted manner. No training or instruction was provided to the participants on reflective writing. Participants were able to complete the NRI in hard copy or electronic form. All journals were completed whilst working in the clinical environment as part of an embedded CPD program over a 1-month period.
Two coders independently analysed the written response in the freeform journals and NRIs for the levels of reflection using the Deep Analytic NRAT. Coding was not completed for the number of times each journal exhibited a specific level of reflection, rather coding illustrated the different levels of reflection that were evident in the writing.
The results of the coding were examined descriptively and categorical statistical analysis was completed to identify whether there was a statistically significant difference in the levels of reflection within the freeform journals, compared with the written responses within the NRIs. Absolute agreement and Kappa statistics were also calculated to determine coder agreement.
Results
Development of the NRIs
Figures 1–3 display the resultant NRIs developed for the three contexts – workshop, significant event and journal reading – used in this research.
Figure 1. Newcastle Reflective Inventory that can be used after completing a workshop or course.
Figure 2. Newcastle Reflective Inventory for use when discussing a journal article that has been read.
Figure 3. Newcastle Reflective Inventory for use when documenting a significant event experienced.
Validation of the Deep Analytic NRAT for assessing NRIs
The sample for this analysis consisted of five interns who completed 30 NRIs in total, (14 Journal Reading NRIs, 8 Significant Event and 8 Workshop NRIs). Figure 4 illustrates the coder profiles of the evidence of reflection found in the NRIs by both coders, broken down by the NRI used and six levels of reflection. For all of the NRIs, the coder profiles show good coder agreement with the exception of the ‘workshop NRI’. The absolute agreement between the two coders when using the Deep Analytic NRAT ranged from 75.0 to 83.3 % and Kappa values ranging from 0.47 to 0.59 (p = 0.001), both measures indicating good intercoder agreement (Figure 4).
Figure 4. Results of the Deep Analytic NRAT analysis (six levels of reflection) of the NRIs, broken down by coders and NRI used. (AA = Absolute Agreement, K = kappa, C1 = coder 1, C2 = coder 2).
When the results of the Deep Analytic NRAT assessment were integrated into the Broad Classification Tool there was a high level of evidence of reflection within the NRI descriptors and excellent intercoder agreement (Figure 5). Coder profiles in all three contexts of the NRI were extremely similar, supported by the excellent absolute agreement (97.3–100.0%) and Kappa statistics (K 0.94–1.00, p < 0.001).
Figure 5. Results of the Broad Classification NRAT analysis (three levels of reflection) for all three NRIs, broken down by coders (AA = Absolute Agreement, K = Kappa).
Validation on the NRIs
Six participants completed a total of 28 journal entries. Of these 16 were significant event entries (8 freeform and 8 NRIs) and 12 journal reading entries (6 freeform and 6 NRIs). When interpreting these results it is important to note that level six (outcome of reflection) of the Deep Analytic NRAT is not a higher level of reflection than that of level 5, as the reflective process does not adhere to a linear model, nor are all steps completed sequentially to reach the end point of reflection. Hence, when assessing written material for evidence of reflection it is important to remember that when there is evidence of level 5 and 6, the practitioners are both reflecting at a critical level of reflection. Interpreting this in relation to this project, Figure 8 illustrates that for coder 1, there was no evidence of level 6 reflectivity using the NRI, compared with 12.5% of entries exhibiting level-6 reflectivity using the freeform method. This does not mean that the freeform entry was more successful at eliciting critical levels of reflection than the NRI.
Figures 6 and 7 illustrate the percentage of participants exhibiting evidence of reflection in the freeform and Journal Reading NRI broken down by coder and level of reflection. The difference in the levels of reflection in the journal reading NRI to that of the freeform entry for both the six and three levels of reflection was highly statistically different. When completing the freeform entry 50% were classified as nonreflective and 33.3% reflective, with <20% being critically reflective. In contrast, the NRI demonstrated that 100% of the participants were classified as critically reflective, with this difference in the evidence of reflection from the NRI to the freeform journal being highly statistically significant (p < 0.001).
Figure 6. Displays the percentage of Journal Reading NRIs and freeform journals that each coder identified evidence of reflection within, broken down by coder and six levels of reflection (K = kappa, AA = absolute agreement, C1 = coder 1, C2 = coder 2).
Figure 7. Displays the percentage of Journal Reading NRIs and freeform journals that each coder identified evidence of reflection within, broken down by coder and three levels of reflection (K = kappa, AA = absolute agreement, C1 = coder 1, C2 = coder 2).
Analysing the results for the freeform and Significant Event NRI (Figures 8 and 9) also revealed a large difference in the evidence of reflection that was highly statistically significant different when assessed using both six and three levels of reflection. When considering the broad three levels of reflection 50% of these freeform journals were classified as having no evidence of reflection (non reflectors), 25% low levels of reflection and 25% as critically reflective. This is contrasted against 100% of journals that showed evidence of critical reflection when completed using the significant event NRI.
Figure 8. Displays the percentage of Significant Event NRIs and freeform journals that each coder identified evidence of reflection within, broken down by coder and six levels of reflection (K = kappa, AA = absolute agreement, C1 = coder 1, C2 = coder 2).
Figure 9. Displays the percentage of Significant Event NRIs and freeform journals that each coder identified evidence of reflection within, broken down by coder and three levels of reflection (K = kappa, AA = absolute agreement, C1 = coder 1, C2 = coder 2).
Discussion
As the completion of this project the Workshop/Conference NRI has been integrated into two large New South Wales teaching hospitals CPD Programs. Although this study covered the development and validation of the NRI in three contexts, further work has been undertaken; the NRIs have been adapted for use in the undergraduate Professional Placement setting in contexts of Personal Interactions and Technical Learning Events. These undergraduate NRIs are currently being integrated into all three Medical Radiation Science (MRS) Disciplines at the University of Newcastle; their use and acceptance by students will be reported at a later date.
Intercoder reliability calculated using Kappa coefficients have illustrated that the NRAT is a reliable tool in assessing written material for evidence of reflection. When the intercoder reliability is compared with the work on WongReference Wong, Kember, Chung and Yan20 on which the NRAT is based, absolute agreement when assessing for the six levels of reflection was 0.83 when utilising the Deep Analytic NRAT compared with 0.5 employing Wong framework alone. The absolute agreement when employing the broader three categories was improved in the study by Wong to achieve 0.88, however, by using the Broad Classification NRAT to guide the assessment of reflective writing achieved an absolute agreement of 0.98.
The results of this study demonstrate that providing radiation therapists with a structured template such as the Newcastle Reflective Inventory is an effective strategy in promoting reflective writing. The study validates the use of the NRI in the context of a significant event journal entry or an entry following the reading of a journal article from a professional publication. However, it should be acknowledged that the use of the NRI may not suit all practitioners in the work place. Those practitioners that are skilled and practiced reflectors may find the format of the reflective inventory too restrictive, as would practitioners that prefer to use an alternate medium to document their reflective process, such as poem or illustration. The evidence of reflection in all the freeform journals supports BoudReference Boud, Walker and Keogh19 commentary, that practitioners have varying skills in the reflective domain and in some cases practitioners do not know how to reflect. It is in these cases that the NRI are most effective, where staff either have little or no knowledge on reflective writing or find it difficult to complete reflective writing exercises. The NRI directs the practitioner through the reflective cycle and allows them to focus more on the subject or experience they are reflecting on and less about what they are expected to be writing.
Conclusion
The results of this study validate the Newcastle Reflective Analysis Tool as a reliable method of assessing short-form guided reflective writing for evidence of reflection. It also clearly validates the Journal Reading and Significant Event NRIs, as effective tools in promoting reflective writing. There are barriers to the incorporation of reflective writing into a practitioners CPD strategy, including resources, knowledge of reflective concepts and support in the clinical environment, incorporation of the NRI may assist in counteracting each of these obstacles. Further research is being undertaken to assess the effectiveness of the Workshop NRI in the wider RT community, the Personal Interaction and Technical Learning Event NRIs in the undergraduate MRS programs and possibly in other allied health disciplines.
