Healthcare-associated infections (HAIs) affect 5% to 10% of patients in acute-care hospitals and up to 33% of those admitted to the intensive care unit (ICU).Reference Yokoe and Classen 1 The Big Four infection types account for >80% of all HAIs: (1) ventilator-associated pneumonia (VAP), (2) central line-associated bloodstream infection (CLABSI), (3) surgical site infection (SSI) and (4) catheter-related urinary tract infection (CAUTI). 2 Their impacts in terms of excess morbidity and expenditures have led to a transition from accepting HAIs as an inevitable outcome of hospital admission to a goal of zero tolerance.Reference Yokoe and Classen 1 , Reference Roberts, Scott and Hota 3
Compliance with current evidence-based guidelines (EBGs) could prevent up to 70% of cases of CLABSI and CAUTI as well as 55% of cases of VAP and SSI.Reference Umscheid, Mitchell, Doshi, Agarwal, Williams and Brennan 4 – Reference Yokoe, Mermel and Anderson 6 Unfortunately, adherence rates among HCWs are often low,Reference Rello, Lorente, Bodì, Diaz, Ricart and Kollef 7 , Reference Ricart, Lorente, Diaz, Kollef and Rello 8 as are their levels of guideline knowledge.Reference Blot, Labeau, Vandijck, Van Aken and Claes 9 – Reference Labeau, Vandijck and Rello 12 The results of various multiple-choice knowledge tests (MCTs) completed by >3,000 European intensive care unit (ICU) nurses never exceeded the conventional 50% threshold to pass a test.Reference Labeau, Vandijck and Rello 10 , Reference Labeau, Vandijck and Rello 12
There is good evidence that effective educational interventions help to facilitate guideline implementation.Reference Bero, Grilli, Grimshaw, Harvey, Oxman and Thomson 13 E-learning, a method that integrates information technology and the learning process using material delivered through the Internet,Reference Reime, Harris, Aksnes and Mikkelsen 14 has been acknowledged as a valuable educational tool.Reference Choules 15
In this paper, we report the development of an e-course that bundles the essentials of evidence-based HAI prevention and its contribution to the acquisition and retention of knowledge regarding evidence-based strategies for infection prevention among an international cohort of HCWs.
METHODS
Course Development
The e-course was developed in the Dutch language using open-source software eXe, release 1.04.0.3532 (http://exelearning.org). The layout was embellished by a web designer. Forward and backward translations of the course were effectuated in English, Portuguese, Spanish, and Turkish languages. To optimize accessibility, only a computer with Internet access and a web browser were needed to study the course; no plug-ins, additional software, or downloads were required.
The validity of the course content was assessed and approved by an international team of experts in infection prevention (S.I.B., D.M.V., J.L., G.D., J.R.). A sample of 50 potential users acknowledged the face validity of the course and its usability according to the Software Usability Measurement Inventory (SUMI), a proven method of measuring software quality from the end user’s point of view.Reference Kirakowski and Corbett 16 In addition to a general score, SUMI measures 5 subdimensions of software usability: (1) the affect subscale quantifies the user’s emotional reaction to the software; (2) efficiency reflects the software’s transparency; (3) helpfulness measures the degree to which the software is self-explanatory and has “help” facilities; (4) control reveals the extent to which users feel in control of the software; and finally, (5) learnability indicates the speed and ease of mastering the system.
Course Contents
The course comprised 7 chapters. The overall focus of the course was evidence-based practice, and the first chapter was dedicated to this concept. The second chapter introduced the problem of HAIs and emphasized the importance of prevention. Hand hygiene, which is key to preventing infection, was discussed in the third chapter. Chapters 4–7 were focused on each of the Big Four, respectively. Each chapter could be studied separately. Different types of exercises with immediate feedback, such as case studies, cloze exercises, and MCTs were integrated to allow self-evaluation during the learning process.
Recruitment of the Sample
An international sample of voluntary learners was recruited through repeated international promotional campaigns: e-flyers were mailed to professional organizations and members of existing networks; flyers were distributed at an (inter)national congresses; and e-mails were sent to all members of the Flemish Society for Intensive Care Nurses (VVIZV) and the European Society for Intensive Care Medicine (ESICM), which endorsed the study.
As an incentive, a certificate of participation issued by the ESICM was acquired upon completion of the entire study path.
Inclusion and Exclusion Criteria
The course was originally designed for HCWs dealing with critically ill patients, but with the exception of VAP prevention, all topics included were valid for non-ICU clinicians as well. Because numerous healthcare professionals working outside the ICU environment also explicitly showed interest in the course, involvement in inpatient care was set as the only requirement for study participation; no exclusion criteria were defined. As such, enrollment was open to all HCWs and students.
Enrollment
A study website (www.evidenceproject.org) was created to provide information about the study design, to allow participants to grant informed consent, enroll, and access the course. The site also provided information on the background, aims, and design of the study. The website was open for registration from October 30, 2010, until December 31, 2011. All transactions on the study site were closed on July 15, 2012.
Study Path
An automated e-mail system guided registered participants through the study path (Fig. 1): (1) An initial e-mail invited participants to log in to the study site and complete a 50-item MCT. (2) After electronic submission of this pretest (T0; measured prior knowledge), participants were automatically granted access to the course for a maximal period of 8 weeks, but students who felt that mastering the course earlier could end the study period as soon as they wanted. After 6 weeks, an automated e-mail alert about the imminent end of the study period was sent. (3) Immediately after the study period, an automated e-mail invited the participants to complete a second 50-item MCT (posttest1; T1; measured immediate learning effect). These questions were identical to those of T0 but were ranked differently.Reference Swygert, Balog and Jobe 17 (d) At 12 weeks after submitting T1 and without further access to the course, participants were invited by automated e-mail to complete a third and final 50-item MCT (T2; measured residual learning effects). Again, these questions were identical to those of T0 and T1 but, again, were ranked differently.Reference Swygert, Balog and Jobe 17
FIGURE 1 Study path. Study website open for course enrollment: October 30, 2010 to December 31, 2011. Final date to submit posttest T1: March 31, 2012. Final date to submit posttest T2: July 15, 2012. Study closed: July 15, 2012.
MCT
The MCT consisted of 1 correct answer, 2 distractors, and the answering option “I do not know” to restrain participants from guessing (see Supplementary Material). The MCT underwent face and expert content validation, and its reliability was assessed by means of item analysis using methods similar to those described elsewhere.Reference Labeau, Witdouck and Vandijck 11 , Reference Labeau, Vandijck, Claes, Van Aken and Blot 18 , Reference Labeau, Vereecke, Vandijck, Claes and Blot 19 Face and content validity were achieved for all items and the results of the item analysis supported the questionnaire’s reliability. Test scores were calculated as follows: correct answer=1 point; wrong answer or “I do not know”=0 points. There was no correction for guessing. Correct answers and feedback on test scores were not provided until the end of the study.
Study Time Measurement
Depending on the participant’s prior knowledge, it was estimated that it would take 3–5 hours to master the course. Each participant’s actual study time was automatically logged. Time-outs requiring an additional log in occurred following 5 minutes of computer inactivity and with a warning popped up on the screen whenever such a timeout was near.
Dropout Analysis
Following the end of the study, a 1-question survey was e-mailed to all participants who had not completed the entire study path to identify their reasons for opting out. The participants were offered 14 answering options to choose from to indicate the reasons for not completing the study. They could indicate as many answering options as applicable for their individual situation, or they could complete the reason in written text.
Statistical Analysis
For statistical analysis, SPSS for Windows 20.0 (IBM, Armonk, New York) was used. Tests were 2-tailed, and statistical significance was set at P<0.05. Continuous variables with non-normal distribution are described as median (interquartile range; IQR). Univariate analysis was performed using the Mann-Whitney U test, Friedman test, or Kruskal-Wallis test as appropriate. For data analysis, the continuous variables “age” and “study time” were conveniently categorized in quartiles. The immediate learning effect was calculated by subtracting the T0 scores (%) from the T1 scores; the residual learning effect was calculated by subtracting the T0 scores from the T2 scores.
Multivariate linear regression analysis using the “Enter” method was conducted to assess independent relationships with immediate and residual learning effects. To avoid spurious associations, variables with a plausible relationship with learning effects and/or a statistical relationship in univariate analysis (P<.20) were included in the regression model. Multicollinearity was assessed. A stepwise elimination of variables with P>.15 was predefined to develop the final model.
To evaluate the course’s effect in relation to human development level of a participant’s country, the 2011 Education and Health Human Development Report of the United Nations Development Program was used to ranks countries into very high, high, medium, or low with regard to the human development index (HDI). 20
Data were analyzed by S.L. and S.B. and were reviewed by a biostatistician at Ghent University.
Ethical Considerations
Upon enrollment, informed consent was given by ticking a box in the electronic registration form. The study was reviewed and approved by the ethics committee at Ghent University Hospital (registration codes B67020072039 and B67020108358).
RESULTS
Face Validity and Usability of the Course
All SUMIReference Kirakowski and Corbett 16 items scored well above the expected average score of 50. Because they demonstrated user satisfaction with the course in general as well as with each of its subdimensions measured (Table 1), no software adaptations were made following this study.
TABLE 1 Results of Software Usability Measurement Inventory (SUMI) Testing
Description of the Sample
A total of 3,587 HCWs representing 79 nationalities enrolled. Of these, 2,590 HCWs (72.2%) submitted T0; 1,410 HCWs (39.8%) actually studied the course and submitted T1; and 1,011 HCWs (28.2%) also submitted T2, thus completing the entire study path. The study flow chart is shown in Figure 2; participant characteristics are listed in Table 2.
FIGURE 2 Study flow chart.
TABLE 2 Median Test Scores and Learning Effects According to Learner Characteristics
NOTE. IQR, interquartile range; n, no. of participants; ICU, intensive care unit; NA, not applicable; HDI, human development index. Data are reported as median % (IQR). For study time and age data, intervals are based on quartiles.
a Median pretest scores (IQR) of participants who submitted only the pretest.
b Median pretest scores (IQR) of participants who submitted pretest + posttest T1,
c Median pretest scores (IQR) of participants who submitted pretest, posttest T1 + posttest T2 (entire study path).
d Immediate effect=Δ pretest – posttest T1.
e P value indicating difference in immediate learning effect between subgroups.
f Residual effect=Δ pretest – posttest T2.
g P value indicating difference in residual learning effect between subgroups.
h Low HDI=participants from Afghanistan, Angola, Bangladesh, Pakistan, Sudan. Medium HDI=participants from Egypt, El-Salvador, Honduras, India, Jordan, Occupied Palestinian Territory, Peru, Philippines, South Africa, Thailand.
i High HDI=Participants from Bolivarian Republic of Venezuela, Brazil, Bulgaria, Colombia, Cuba, Ecuador, Former Yugoslav Republic of Macedonia, Georgia, Islamic Republic of Iran, Kuwait, Malaysia, Mexico, Romania, Russian Federation, Saudi Arabia, Turkey, Ukraine. Very high HDI=participants from and Argentina, Australia, Austria, Belgium, Canada, Chile, Croatia, Cyprus, Denmark, Estonia, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Luxembourg, Malta, The Netherlands, New Zealand, Norway, Poland, Portugal, Singapore, Spain, Sweden, Switzerland, the United Arab Emirates, the United Kingdom, and the United States.
The median study time (n=1,410) was 194 minutes (IQR, 96–306 minutes). Convenient grouping of this variable according to its quartiles resulted in the following categories: <100 minutes (median 45 minutes; IQR, 24–70 minutes; n=371), 100–200 minutes (median, 156 minutes; IQR, 127–178 minutes; n=353), 201–300 minutes (median, 243 minutes; IQR, 223–269 minutes; n=318), and >300 minutes (median, 405 minutes; IQR, 344–502 minutes; n=368).
The median age of the 1,410 participants was 33 years (IQR, 28–40 years). This variable was conveniently categorized based on its quartiles as follows: <30 years (median, 26 years; IQR, 23–28 years; n=441), 30–35 years (median, 32 years; IQR, 31–34 years; n=429), 36–40 years (median, 38 years; IQR, 36–39 years; n=210), and >40 years (median, 46 years; IQR, 43–50 years; n=330).
Participants included 1,034 ICU nurses, 267 clinical nurse specialists, 261 general care nurses, 189 intensivists, 182 microbiologists, 115 head nurses, 115 nurse practitioners, 100 nursing students, 98 pharmacists, 73 emergency room nurses, 71 anaesthesiologists, 41 other physicians, 18 physician trainees, 8 respiratory therapists, 6 paramedics, and 4 physiotherapists.
Test Scores and Learning Effects
The overall median score at T0 was 52% (IQR, 44%–62%; n=2,590), increased to 80% (IQR, 68%–88%) at T1 (n=1,410), and was 74% (IQR, 64%–84%) at T2 (n=1,011). Among the participants who completed the entire study path (n=1,011), median scores were 54% (IQR, 46%–64%) at T0, 82% (IQR, 72%–90%) at T1, and 74% (IQR, 64%–84%) at T2. Table 2 shows the median scores according to student characteristics and related learning effects.
The overall immediate learning effect was 24% (IQR, 12%–34%; P<.001; n=1,410), and the overall residual learning effect was 18% (IQR, 8%–28%; P>.001; n=1,011). The immediate learning effect obtained by the students who completed the entire study path was 21% (IQR, 14%–34%; P<.001; n=1,011).
For all course topics, positive immediate and residual learning effects were found (Table 3). Gains in knowledge increased with study time. The immediate learning effect reached a maximum at 200 study minutes (28%), while the residual effect was greater once study time exceeded 300 minutes (Table 2).
TABLE 3 Median Test Scores and Learning Effects for the Total Course and Per Category of Questions
NOTE. n, no. of participants; IQR, interquartile range; Q, questions; CAUTI, catheter-associated urinary tract infection; CLABSI, central line-associated bloodstream infection; VAP, ventilator-associated pneumonia; SSI, surgical site infection. Data are reported as median % (IQR).
a Median pretest scores (IQR) of participants who submitted only the pretest.
b Median pretest scores (IQR) of participants who submitted pretest+posttest T1,
c Median pretest scores (IQR) of participants who submitted pretest, posttest T1 + posttest T2 (entire study path).
d Immediate effect=Δ pretest – posttest T1.
e Residual effect=Δ pretest – posttest T2.
P value for all subgroup analyses: P<.001.
Multivariate linear regression (Table 4) identified longer study time, longer work experience, and living in a country with high or very high HDI as being associated with a better immediate learning effect; a higher score on the pretest, female gender, and higher age category were independently associated with lower learning effects (R2=0.36).
TABLE 4 Multivariate Linear Regression Analysis for Independent Relationships with Learning Effects
NOTE. ICU, intensive care unit; HDI, human development index.
a R2=0.36.
b Study time <100 min, 101–200 min, 201–300 min, or >300 min.
c Work experience <1 year, 1–5 years, 6–10 years, or >10 years.
d Age <30 y, 30–35 y, 36–40 y, or >40 y.
e R2=0.29.
For the residual effect, longer study time, female gender, and longer work experience were identified as independently associated with better learning effects; a higher score on the pretest and higher age category were associated with lower effects (R2=0.29). Multicollinearity analysis detected no correlations between the variables entered.
Dropout Analysis
Of all 1,579 dropouts contacted, 525 completed the survey (33% response rate). In addition to a lack of time (n=211; 40%), the main reasons for opting out included having forgotten course enrollment (n=146; 28%), problems with computer or Internet connection (n=52; 10%), length of the study path (n=21; 4%), and unavailability of the course in the respondent’s mother tongue (n=9; 1%).
DISCUSSION
In the present study, we found that limited time invested in studying an e-course on the essentials of HAI prevention with good usability and exercises for self-evaluation yielded significant increases in immediate (+24%) and residual (+18%) learning effects among HCWs. Although the course was originally developed for ICU professionals, it also showed to benefit HCW in non–ICU-related settings.
Cook et alReference Cook 21 systematically reviewed 126 articles published between 1990 and 2007 that evaluated knowledge outcomes of Internet-based instructional methods compared to no intervention. Overall, e-learning improved knowledge by 12%. Our study resulted in an overall immediate learning effect of 24%, thereby doubling the expected improvement. However, our students were volunteers, and selection bias might have influenced our results. Our participants may have been particularly motivated or interested in the topic, and the most motivated or interested may have completed the entire study path, thus generating better learning effects than if participation had been imposed on a random sample. Yet, during the study course, numerous clinicians indicated that they participated specifically to obtain the certificate of participation issued by the ESICM. This motive for participation might at least partially alleviate selection bias caused by voluntary enrollment. Also, we found that greater prior knowledge was associated with lower immediate and residual learning effects. This can be considered a logical finding: the greater the prior knowledge, the less room for improvement. If these participants’ prior knowledge was higher than that of the general population of HCWs, learning effects in a random sample might exceed those identified in the current cohort.
Interpretation of the results in association with participant characteristics (Table 2) shows that males and females obtained identical immediate learning results (P=.14). Residual effects were, however, significantly higher among women (P=.001; Table 2), and men displayed a greater decrease in knowledge after 3 months without course access (−10%). The current study did not focus on gender differences, and more research into the relationship between healthcare professionals’ gender and learning styles, preferences, and outcomes might help to explain these findings.
Physicians and nurses demonstrated identical immediate learning effects (24%). Physicians nevertheless obtained higher crude pretest and posttest T1 scores with somewhat narrower interquartile ranges, which might be suggestive for a more common general knowledge of EBGs for HAI prevention than among nurses. Not unexpectedly, trainees obtained the lowest pretest scores and the highest immediate learning effect (30%). Their scores on posttest T1 (77%), as well as the scores of pharmacists, microbiologists, respiratory therapists, and paramedics (78%, other HCWs) were relatively low compared with physicians and nurses (84% and 80%, respectively). This result and the finding that more experienced participants obtained better scores on the pretest than their less experienced colleagues suggest a potential positive association between knowledge and daily practice experience.
Of all course topics (Table 3), SSI prevention was associated with the lowest baseline test scores and the highest immediate learning effect. Possibly, HCWs involved in surgical patient care are specifically familiar with this problem compared with those working in other types of settings. Remarkably, the limited learning effects and relatively low crude posttest scores obtained for the topic of hand hygiene reflect the challenge of finding an effective educational approach to increase HCW interest in hand hygiene guidelines and, consecutively, compliance.
The dropout rate from our course was high. Of all HCWs enrolled, <40% actually studied the course. Isolation of e-learners has been identified as a common reason for high dropout rates.Reference Ellaway and Masters 22 Our dropout survey identified a lack of time as the main reason for opting out. Because our sample merely consisted of volunteers involved in inpatient care, a high dropout rate was hardly unexpected; work, family life and personal commitments are easily and understandably prioritized over continuing eduation. Potentially, dropouts were less motivated than HCWs who completed the entire study, which may have contributed to the positive study results.
Additional research comparing different web-based interventions is needed to elucidate how to implement e-learning most effectively. In the meantime, our study suggests that moderate time invested in a low-cost e-course with good usability features and exercises for self-evaluation can enhance knowledge on HAI prevention. We therefore encourage institutional decision makers and professional societies to consider translating their recommendations in e-learning modules in parallel with published guidelines.
ACKNOWLEDGMENTS
The authors wish to acknowledge the valuable help of Prof. Jurek Kirakowski, School of Applied Psychology, University College Cork, Ireland, for free use of the SUMI software to evaluate the course’s usability and for the related data analysis. We thank Prof. Georges Van Maele at University of Ghent for reviewing our data analysis. We are grateful to the VVIZV and the ESICM for endorsing the study and all support, and to all professional organizations that supported our recruitment of participants for this project. Last but not least, we sincerely thank all HCWs who participated in the project.
Financial support: This study was supported by a doctoral research grant from University College Ghent and an unrestricted grant from the ESICM and Edwards (ECCRN-Edwards Nursing Science Award 2008). The study was endorsed by the ESICM. Funding sources had no involvement in the design or analysis of this study.
Potential conflicts of interest: All authors report no conflicts of interest relevant to this article.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/ice.2016.107
