Disasters claim a large number of lives through the destruction of the physical environment and disruption to the social and economic structures of a community. For example, the 2010 Haitian earthquake resulted in more than 200 000 deaths and displacement of another 1.5 million individuals.Reference Doocy, Cherewick and Kirsch1 According to the United Nations Office for Disaster Risk Reduction,2 between 1995 and 2015 more than 4.1 billion individuals have been affected and more than 606 000 have died as a result of disasters. Having a more prepared and resilient community can minimize the impact of disasters, but this requires collaborative planning prior to an event and access to appropriate resources after a disaster occurs.Reference Phillips3, Reference Reynolds and Tamanaha4
An important resource during disasters is having access to critical health information relevant to the event. Without access to such information, emergency responders’ effectiveness is limited. Libraries and librarians play a critical role in emergency management, because the information they provide can improve disaster preparedness and response.Reference Turoff and Starr5 Librarians can serve as institutional supporters, collection managers, information disseminators, internal planners, community supporters, government partners, educators and trainers, and information community builders in any phase of disaster and emergency management.Reference Featherstone, Lyon and Ruffin6 For example, librarians can develop accurate and current disaster and health online resources prior to an event that can be accessed and disseminated to emergency responders, health professionals, and the public during a disaster.Reference Featherstone, Lyon and Ruffin6 As educators and trainers, librarians can also provide training for emergency responders in the use of emergency reference services and tools.Reference Featherstone, Lyon and Ruffin6 Emergency responders’ critical information needs and librarians’ skills in mobilizing and making health information accessible call for the close collaboration between these 2 professions both before and after a disaster occurs.Reference Malizia, Hamilton and Littrell7
The National Library of Medicine (NLM) created the Disaster Information Management Research Center (DIMRC) in 2008 to collect, organize, and disseminate critical health information to emergency responders, health professionals, and the general public during mitigation, preparedness, response, and recovery from disasters.Reference Love, Arnesen and Phillips8 The NLM’s DIMRC resources include training courses, databases, research and after-action reports, guidelines, and preparedness and response tools.9 The NLM also provides training for librarians to become Disaster Information Specialists; this specialized education prepares librarians to serve as support for providing information related to disaster medicine, emergency management, and public health preparedness.Reference Reynolds and Tamanaha4 It is vital that emergency responders and librarians be made aware of and trained in the use of these NLM DIMRC resources so that they can be actively incorporated into disaster-related training and reference materials. However, training is needed to increase awareness of the DIMRC and the free NLM tools.9 In addition, there is a need for emergency responders and librarians to collaborate, so that both professions become aware of the others’ skills and capabilities.9 This multidisciplinary collaboration should result in better prepared emergency responders and more resilient communities.
This study’s primary purpose was to increase awareness and knowledge of NLM resources among emergency responders and librarians and to evaluate the impact of this education on individual competence. Secondary aims were to determine whether dissemination of DIMRC materials could be increased using a train-the-trainer model and to expand the role of Missouri librarians by encouraging them to partner with local emergency responders in disaster planning efforts. This project’s title is the Value of Improved and Sustained Information Access by Library Expertise (VISIBLE).
METHODS
VISIBLE consisted of 4 workshops held in St Louis, Kansas City, Springfield, and Jefferson City, Missouri, in spring of 2016 to increase knowledge of the following NLM DIMRC tools: Wireless Information System for Emergency Responders (WISER), Radiation Emergency Medical Management (REMM), Chemical Hazards Emergency Medical Management (CHEMM), and Toxicology Data Network (TOXNET). The workshops were multidisciplinary and included emergency managers, disaster planners, public health professionals, and librarians. Individuals were recruited with posters/flyers that were emailed to potential participants. Multiple organizations aided in distribution of recruitment materials: St Louis Area Regional Response System, Mid-America Regional Council, Missouri’s State Emergency Management Agency, Boone County Department of Emergency Management, Health Sciences Library Network of Kansas City, Midcontinental Chapter of the Medical Library Association, Missouri Library Association, MOBIUS Consortium, and the St Louis Chapter of the Special Library Association.
Intervention/Workshop
The VISIBLE workshop consisted of 5 scenarios, the first of which was conducted “cold” (ie, prior to education about the NLM tools). Next, workshop attendees were provided information about the NLM tools and walked through examples of how and when to use them. Participants were then given 4 additional scenarios to work through that required use of the tools. The scenarios consisted of a cesium 137 theft (the cold scenario), a train crash involving release of radiological material, a chemical suicide in a car, sarin gas release on a bus, and a dirty bomb explosion in an urban area. The scenarios were customized for each workshop through incorporation of local landmarks to make the scenarios more realistic. Workshop seating was arranged so that each group was multidisciplinary and facilitated small group discussions that forced team members to work together. The workshop utilized a train-the-trainer (TTT) format that consisted of providing the workshop’s slides, handouts, and scenario information to the participants with the hope that they would share the workshop materials with others formally or informally.
Data Collection and Variables of Interest
Data were collected both before and after the VISIBLE workshop in a preintervention/postintervention study design. Postintervention data were collected during 3 time frames: immediately after the workshop, 3 months later, and 1 year later. Data collected included the following: (1) awareness and use of the NLM DIMRC materials; (2) perceived confidence in accessing and using the NLM tools; (3) knowledge of the NLM WISER, REMM, CHEMM, and TOXNET tools; (4) workshop participants’ intent to disseminate the knowledge and extent to which they did disseminate the knowledge through formal and informal trainings; and (5) reasons why the information was not shared, when applicable. Awareness and perceived confidence were measured on a 5-point Likert-scale. Knowledge was measured by 16 true-false and/or multiple-choice questions, and all knowledge-related question content was covered during the workshop. In addition, participants were asked 8 questions related to the perceived quality and usefulness of the workshop; items were measured using 5-point Likert scales. The workshop was pilot-tested with a group of 31 graduate public health students and faculty from Saint Louis University prior to study implementation. The Saint Louis University Institutional Review Board approved this study.
Data Analysis
All data analyses were performed using Statistical Package for the Social Sciences (SPSS®) 24.0. Knowledge questions were scored as 1 point per question, with the subsequent calculation of a knowledge score that could range from 0 to 16 points. Descriptive statistics were conducted with all variables. Wilcoxon signed rank tests were run to compare preworkshop responses to immediate and 1-year postworkshop responses for awareness and perceived confidence. Chi-square tests were used to compare perceived confidence across use of the NLM tools. A paired t test was conducted to assess a change in knowledge from before the workshop to immediately after the workshop. Chi-square tests were used to compare intent to share training materials formally versus informally at each time period; when cell counts were below 5, a Fisher exact test was used. A Bonferroni correction was used for the pre-post awareness and perceived confidence comparisons, which resulted in an adjusted critical value of 0.008 for these tests (calculated as .05/6 comparison tests). A critical P value of .05 was used for all other analyses.
RESULTS
Four workshops were hosted, with a total of 74 attendees; the number of participants ranged from 11 to 28 per workshop. Just over half (56.8%, n = 42) were male and most (90.5%, n = 67) were white. Twenty percent (n = 15) of the attendees were librarians; the remaining 80% were first responders, public health professionals, emergency managers, and health care professionals. More than half (62.2%, n = 46) identified as being in a management role. A little more than a third (37.9%, n = 29) had a master’s degree or more education or a bachelor’s degree (37.9%, n = 29). Approximately one-half (51.4%, n = 38) had 11 or more years of work experience. Of the 15 librarians, one-third (n = 5) were reference librarians and the rest were public librarians. None of the librarians reported holding a certificate in Disaster Information Specialization from the Medical Library Association. Of the first responders, public health professionals, emergency managers, and health care professionals (n = 59), about a quarter (28.8%, n = 17) were current or past members of hazardous materials (HAZMAT) teams. Participant demographics are outlined in Table 1. The response rate for the surveys ranged from 50% to 100%. All participants (N = 74) completed a preworkshop and immediate-postworkshop surveys; 58% (n = 43) completed a 3-month postworkshop survey, and 50% (n = 37) completed a 1-year postworkshop survey.
TABLE 1 Participant Demographics (N = 74)
Perceived Quality and Relevance of the Workshop Materials
Immediately following the workshop, participants were asked questions about the perceived quality of the workshop, whether they believed they accomplished the workshop’s goals, and perceived relevance and usefulness of the workshop materials for themselves or when they train others. The questions were measured using Likert scales related to perceived quality (1 = poor to 5 = excellent) and perceived relevance (1 = strongly disagree to 5 = strongly agree). Average scores on all perceived relevance and quality questions were high (range of average scores: 3.95 to 4.53), meaning that most participants found the workshop materials to be of high quality, useful, and relevant to them. Table 2 outlines participants’ responses to questions related to perceived quality and usefulness of the workshop materials.
TABLE 2 Participants’ Perception of the Quality and Usefulness of the Workshop
a Likert scales used: quality scale, 1 = poor to 5 = excellent; perception scale, 1 = strongly disagree to 5 = strongly agree.
Awareness and Use of the NLM Tools Before Versus After the Workshop
“Awareness and use of the NLM tools” was measured on a 5-point scale ranging from 1 to 5 (1 = “never heard of them”; 5 = “aware and used them often”). There was a significant increase in awareness from before to after the workshop, and this awareness remained increased 1 year after the workshop. There was a significant increase in awareness from before the intervention to after the intervention (average 1.9 preworkshop vs 3.5 immediately postworkshop and 3.1 1 year postworkshop; P < .001 for both pre/post comparisons). Almost none of the participants reported having ever received any training on REMM, CHEMM, or TOXNET prior to the workshop (91.9%, 91.9%, and 90.5%, respectively). Most (80.0%, n = 59) reported having received no training on WISER, though 10% (n = 8) had received 30 minutes or less training and the remaining 10% (n = 7) had received 31 minutes or more WISER training. Almost none of the participants reported having ever used REMM, CHEMM, or TOXNET prior to the workshop (95.9%, 93.2%, and 91.9%, respectively). Very few participants indicated that, prior to the workshop, they had used WISER often (2.7%, n = 2) or sometimes (8.1%, n = 6). The rest had either never used it (78.4%, n = 58) or used it rarely (10.8%, n = 8). The librarians (n = 15) were asked if an emergency responder had ever approached them for assistance in identifying literature relevant to disaster planning or response; 93.3% (n = 14) indicated that they had never been asked by an emergency responder to find such resources.
Perceived Confidence in Using the NLM Tools Before Versus After the Workshop
Participants were asked about their perceived confidence in using 4 NLM tools: WISER, CHEMM, REMM, and TOXNET, using a 5-point Likert-type scale ranging from 1 (very unconfident) to 5 (very confident). Changes in perceived confidence from preworkshop to immediately postworkshop were statistically significant and remained statistically significant at the 1-year follow-up for all 4 NLM tools: CHEMM, REMM, TOXNET, and WISER (Table 3). Participants were significantly more likely to report being confident in the use of WISER than in the use of the other NLM tools (P < .001 for all comparisons).
TABLE 3 Participants’ Perceived Confidence in Using the National Library of Medicine Tools Before the Workshop Versus After the Workshop
Abbreviations: CHEMM, Chemical Hazards Emergency Medical Management; NLM, National Library of Medicine; REMM, Radiation Emergency Medical Management; TOXNET, Toxicology Data Network; WISER, Wireless Information System for Emergency Responders.
a Perceived confidence measured on a 5-point Likert-type scale: 1 = very unconfident; 5 = very confident.
b Determined by a Wilcoxon signed rank test with Bonferroni correction.
Knowledge of the NLM Tools Before Versus After the Workshop
Sixteen questions were used to assess participants’ knowledge of the NLM tools, and 1 point was assigned to each item; therefore, participants’ knowledge scores could range from 0 to 16. The average preworkshop knowledge score was 7.2 and the average immediately postworkshop score was 11.9. The mean pre-to-post change of 4.7 was statistically significant (t = 15, P < .001), with a 95% confidence interval of 4.1 to 5.3. Knowledge scores did not differ with occupation, age, gender, education level, years of work experience, perceived confidence in using the tools, or past reported use of the tools.
Intent to and Extent to Which Participants Disseminated Materials After the Workshop
Immediately after the workshop, about a third of participants (36.5%, n = 27) indicated that they planned to host a formal training using the TTT materials in the next month; significantly more individuals (54.1%, n = 40; P < .001 for all comparisons) indicated that they planned such a formal training in the next 6 months or year. About three-quarters (78.4%, n = 58) reported that they planned to share the TTT materials informally with others in the next month. Participants were significantly more likely to indicate that they planned to offer informal training than formal training for each of the time frames (Table 4). There was no association between occupation and intent to train formally or informally.
TABLE 4 Intent to Share Workshop Materials Formally Versus Informally in the Short-term and Long-term
a Participants could answer “yes” to more than 1 time frame.
b Determined by the X 2 test; Fisher exact test was used when cell sizes were ≤ 5.
Three (7%) of the 43 participants who responded to the 3-month postworkshop survey reported that they had provided formal training to others using the TTT materials since the workshop; in total, they reported providing formal training to approximately 31 individuals. Twenty-four (55.8%) of the 43 participants who responded to the 3-month postworkshop survey reported that they had provided informal training to others since the workshop; in total, they reported having informally trained approximately 157 individuals. Librarians and first responders were equally likely to have provided informal training.
Among the respondents who answered the 1-year postsurvey (n = 37), significantly more reported having informally trained at least 1 person since the workshop than indicated that they had provided at least 1 formal training session (56.8% vs 8.1%; P < .001). In these reported trainings by workshop attendees, approximately 42 individuals received formal training and 203 individuals received informal training. Those who responded to the 1-year postworkshop survey who indicated that they had not used the TTT materials to provide formal training (n = 34) were asked to identify the reason(s) they did not provide such training to others; they could select more than 1 reason. The most frequently reported reasons were a lack of time (61.8%, n = 21) and the belief that it was not their responsibility to train others (52.9%, n = 18). About a third (32.4%, n = 11) indicated that they lack the resources to host a formal training, 26.5% (n = 9) lacked confidence in their knowledge of the tools, and 11.8% (n = 4) believed it would be too expensive.
Of those who responded to the 1-year postworkshop survey (n = 37), about a quarter (24.3%, n = 9) indicated that they planned to host a formal training session using the TTT materials in the next month to year. Even more reported planning to host informal training sessions on the NLM tools in the next month to year (45.9%, n = 17; formal vs informal comparison, X 2 = 8.8, P < .01; Table 4). There was no association between occupation and intent to train formally or informally.
Qualitative Comments During and After the Workshop
In addition to the quantitative data collected and analyzed, qualitative data were collected via the postworkshop surveys and through the research team’s observations during the workshops. A recurring theme identified was the participants’ desire for more training on radiological events. Two of the scenarios involved radiological events, with the intent of having participants gain mastery over the NLM tools by having them access and use the tools to respond to the scenarios. Though this was successful, multiple workshops almost became severely sidetracked when participants requested specific information regarding radiological disasters that was outside the workshop’s scope. Another common theme expressed by attendees was the unanticipated benefit of having librarians and emergency responders partner together in the workshop. Many attendees, both librarians and emergency responders, commented that they had not understood librarians’ role in disaster preparedness prior to the workshop, but afterwards expressed the benefit of this collaboration.
DISCUSSION
The primary purpose of the VISIBLE program was to increase awareness and knowledge of NLM resources among emergency responders and librarians. To that end, this project was highly successful as evidenced by the significant increase in awareness of, perceived confidence in the use of, and knowledge of the NLM tools covered by the workshop. In addition to the short-term change commonly seen in education-related intervention studies,Reference Abatemarco, Beckley and Borjan10, Reference Foster, Campbell and Lee11 this study found that the attendees maintained their increased awareness and perceived confidence in using the NLM tools for 1 year after the workshop. Though studies assessing the long-term outcomes of public health disaster preparedness educational interventions are uncommon, 2 other studies reported similar findings: 1 year after a training, attendees had retained their knowledge and/or confidence in the content provided.Reference Lehman-Huskamp, Rebmann and Walter12, Reference O’Brien, Broom and Ullah13 The VISIBLE project focused on 4 tools available for free through the NLM: WISER, REMM, CHEMM, and TOXNET. These tools can be used for both pre-event planning and for responding to actual events. The sustained strong confidence in using these tools increases the likelihood that emergency responders will use the NLM resources to develop more comprehensive disaster plans and/or have a more successful response to an actual event. This should result in more resilient communities.
Findings from this study indicate that the TTT format resulted in dissemination of the VISIBLE program workshop materials to others by both the librarian and emergency responder participants. In total, approximately 245 individuals received training on the NLM tools during the “second generation” of this project in the 1 year after the workshop. In addition, a quarter of the respondents reported that they plan to host a formal training session and almost half indicated that they would disseminate information on the NLM tools informally to others in the upcoming year. Assuming that they follow through with these intentions, information on the NLM tools will be greatly disseminated beyond the participants of the initial workshops held in this study. This supports the intention of using a TTT format: to spread the training more widely than could be done by only hosting a few workshops.Reference Cross, Cerulli and Richards14, Reference Koffel and Reidt15
It is difficult to quantify the expected percentage of participants who would disseminate the information to others or the number of individuals that would be trained formally or informally after a TTT program, as a lack of long-term follow-up to document exactly how many individuals have been trained in the second generation is a common limitation in TTT programs.Reference Cross, Cerulli and Richards14, Reference Orfaly, Frances and Campbell16 Only 3 fully evaluated TTT programs related to disaster preparedness could be identified. A TTT program in New York focused on a disaster mental health education intervention reported that 60% of the 140 participants had provided some training to others in the year after the intervention, and that, in total, 559 individuals were trained in the second generation.Reference Cross, Cerulli and Richards14 An study on an online TTT program aimed at Bangladeshi health professionals reported that there was almost no dissemination of knowledge beyond the initial groups of individuals trained; the reason for the lack of follow-up was postulated to be a lack of confidence in the knowledge gained.Reference O’Brien, Broom and Ullah13 A state-wide TTT program focused on public health preparedness in Maine involved 118 individuals in the initial training, and only 20% of those disseminated their training to others.Reference Orfaly, Frances and Campbell16 The findings from the VISIBLE program are in line with the better results reported in the New York TTT study in terms of the percentage of workshop attendees who disseminated the training to others and the approximate number of people educated in the second generation. Further research should be conducted to examine the long-term outcomes of other TTT programs to better quantify the extent of dissemination of programs or the reasons why some programs are more successful than others.
Findings from this study indicate that VISIBLE program workshop participants were significantly more likely to offer informal training than formal training on the NLM tools. One explanation for this could be the time and resource commitment needed to provide formal training. For example, 2 commonly reported reasons for not providing formal training in this study included a lack of time and lack of resources. Another possible reason why participants in this study offered more informal than formal training is that those who attended the workshop were not aware before arrival that the program was intended to be a train-the-trainer format. It is possible that those who attended were seeking training for themselves only and are not in a position to host formal training for others. There is some evidence for this: half of those who did not offer formal training reported the belief that it was not their responsibility to train others. In general, TTTs are designed to provide education to instructor-trainees who have the responsibility to train others.Reference Cross, Cerulli and Richards14, Reference Koffel and Reidt15 If workshop attendees do not perceive themselves to be instructor-trainees, it lessens the likelihood that they will use the TTT materials to disseminate to others. It may be prudent to target advertising for future workshops to indicate the TTT aspect of the program. Dissemination of TTT materials may also be increased in future workshops by more heavily emphasizing the importance of sharing the TTT materials and offering ongoing assistance to attendees if they request it.
A secondary aim of the VISIBLE program was to expand the role of Missouri librarians by partnering them with local emergency responders in an educational setting to improve disaster planning efforts. Though this aim was not measured quantitatively through the study, there is qualitative evidence that both emergency responders and the participating librarians benefited from working together at these workshops. Attendees commented at the workshop and in the qualitative comments section of the surveys about the benefits of having this diverse group of professionals working together on these complex scenarios. It seems that, prior to this workshop, neither group understood how librarians could contribute to disaster preparedness. Given that none of the librarians in this study reported holding a certificate in Disaster Information Specialization from the Medical Library Association,Reference Foster, Campbell and Lee11 this workshop may have served as an introduction to the librarian’s role in disaster preparedness for many of the participating librarians. It is important to build this awareness, because librarians can play a significant role in preparing health care facilities and emergency response agencies for disasters.Reference Lehman-Huskamp, Rebmann and Walter12 In particular, librarians could aid emergency responders in identifying critical literature needed to develop more comprehensive disaster plans.Reference Lehman-Huskamp, Rebmann and Walter12 It is notable that, in this study, only 1 of the librarians had ever been asked by an emergency responder to assist in identifying disaster-related literature. Perhaps this is because emergency responders are not aware of this resource in their community. Future studies and educational programs should be focused on increasing awareness about these resources and providing more collaborative opportunities for emergency responders and librarians.
This study had multiple strengths. It was the first known study to partner librarians and emergency responders in an educational setting specifically to enhance the benefits of each group. TTT formats have long been used to aid in disseminating training materials, but formal evaluation, especially long-term evaluation, of these programs is rare.Reference Cross, Cerulli and Richards14, Reference Koffel and Reidt15 This study included an extended postworkshop follow-up time period (ie, 1 year), which allowed for examination of longer-term outcomes, which is a major strength. The main limitation to this study was the lower response rate at the 3-month and 1-year postworkshop time periods; however, the response rate remained reasonably high at 58% (3 months) and 50% (1 year) in the follow-up time periods and is similar to the response rate seen in other studies using a TTT approach.Reference Cross, Cerulli and Richards14–Reference Orfaly, Frances and Campbell16 The lower response rate in the follow-up surveys may introduce some response bias in those who chose to complete a 3-month or 1-year postworkshop survey. It is possible that nonresponders may have answered differently than those who chose to complete a survey. This study may also have social bias, as the reporting of postworkshop training dissemination was self-reported. Lastly, because this study was only conducted in Missouri, the results may not be generalizable to other regions.
CONCLUSIONS
This study used an educational intervention to increase awareness of, confidence in the use of, and knowledge of free tools available through the National Library of Medicine for improving disaster preparedness and response among emergency responders and librarians. Attendees perceived the workshop to be of high quality and useful to their careers, and both short- and long-term positive changes were found. In addition, the train-the-trainer model resulted in further dissemination of the workshop information through both formal and informal training. The NLM tools are free of charge and could be used to host TTT programs in other communities.
Acknowledgements
This study was partially funded by the National Library of Medicine (contract No. HHSN276201500660P). The workshop materials were developed using this funding, and are available free of charge. Those interested in obtaining a copy of the materials may contact the primary author.
