Introduction
The U.S. Army and Department of Defense (DoD) have implemented prevention programs for several negative outcomes (Department of the Army, 2015b ), including suicide (Department of the Army, 2015a ), workplace violence (Department of Defense, 2014b ), and sexual assault (Department of Defense, 2014a ), that have high prevalence in the early years of the Army career (Kaufman et al. Reference Kaufman, Brodine and Shaffer2000; Department of the US Army, 2010, 2012). These preventive interventions are mostly universal; that is, all personnel are required to participate and the interventions are relatively non-intensive. More intensive interventions exist and could be implemented (Vungkhanching et al. Reference Vungkhanching, Heinemann, Langley, Ridgely and Kramer2007; Parkkari et al. Reference Parkkari, Taanila, Suni, Mattila, Ohrankammen, Vuorinen, Kannus and Pihlajamaki2011; Shea et al. Reference Shea, Lambert and Reddy2013; Rudd et al. Reference Rudd, Bryan, Wertenberger, Peterson, Young-McCaughan, Mintz, Williams, Arne, Breitbach, Delano, Wilkinson and Bruce2015; Senn et al. Reference Senn, Eliasziw, Barata, Thurston, Newby-Clark, Radtke and Hobden2015), but would be cost-effective only if targeted at high-risk personnel (Foster & Jones, Reference Foster and Jones2006; Golubnitschaja & Costigliola, Reference Golubnitschaja and Costigliola2012). This targeting would require valid risk prediction tools. Recent studies have shown that Army and DoD administrative data can be used to develop such tools to predict negative soldier outcomes such as suicide (Kessler et al. Reference Kessler, Warner, Ivany, Petukhova, Rose, Bromet, Brown, Cai, Colpe, Cox, Fullerton, Gilman, Gruber, Heeringa, Lewandowski-Romps, Li, Millikan-Bell, Naifeh, Nock, Rosellini, Sampson, Schoenbaum, Stein, Wessely, Zaslavsky and Ursano2015), violent crime perpetration (Rosellini et al. Reference Rosellini, Monahan, Street, Heeringa, Hill, Petukhova, Reis, Sampson, Bliese, Schoenbaum, Stein, Ursano and Kessler2016), and sexual assault victimization (Street et al. Reference Street, Rosellini, Ursano, Heeringa, Hill, Monahan, Naifeh, Petukhova, Reis, Sampson, Bliese, Stein, Zaslavsky and Kessler2016), but these models are limited by the fact that administrative data only become available over the course of time and are unavailable when preventive interventions might most logically be implemented at the beginning of service. An alternative would be to implement a risk factor survey at the beginning of service to target new recruits for preventive interventions. The current report presents the results of an attempt to develop risk models for a number of high-priority negative outcomes using data collected in such a survey of new U.S. Army soldiers subsequently followed over the first 2 years of service. If successful, the logic of this approach might be generalizable to a wide range of other workplace settings.
Methods
Sample
The survey was the New Soldier Survey (NSS) of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) (Ursano et al. Reference Ursano, Colpe, Heeringa, Kessler, Schoenbaum and Stein2014).The NSS was implemented April 2011–November 2012 in a representative sample of new U.S. Army soldiers prior to beginning Basic Combat Training (BCT) at Fort Benning, GA, Fort Jackson, SC, and Fort Leonard Wood, MO. Recruitment began by selecting weekly samples of 200–300 new soldiers at each installation to attend an informed consent presentation within 48 h of reporting for BCT. The presentation explained study purposes, confidentiality, voluntary participation, and answered all attendee questions before seeking written informed consent for a self-administered computerized questionnaire (SAQ) and neurocognitive tests and to link these data to the soldier's administrative records. These study recruitment and consent procedures were approved by the Human Subjects Committees of all Army STARRS collaborating organizations. The 21 832 NSS respondents considered here represent all Regular Army soldiers who completed the SAQ and agreed to administrative data linkage (77.1% response rate) (Rosellini et al. Reference Rosellini, Heeringa, Stein, Ursano, Chiu, Colpe, Fullerton, Gilman, Hwang, Naifeh, Nock, Petukhova, Sampson, Schoenbaum, Zaslavsky and Kessler2015). Data were doubly-weighted to adjust for differences in survey responses among the respondents who did v. did not agree to administrative record linkage and differences in administrative data profiles between the latter subsample and the population of all new soldiers.
Outcomes
Outcome data were abstracted from 14 administrative databases through December 2013 (13–33 follow-up months after NSS completion) to operationalize 12 high-priority outcomes involving mental–physical disorders (Canham-Chervak et al. Reference Canham-Chervak, Hooper, Brennan, Craig, Girasek, Schaefer, Barbour, Yew and Jones2010; Department of the US Army, 2010; Institute of Medicine, 2010), violent crime perpetration–victimization (Institute of Medicine, 2010; Department of the US Army, 2012), and career problems (Kubisiak et al. Reference Kubisiak, Lentz, Horgen, Bryant, Connell, Tuttle, Borman, Young and Morath2009; Kapp, Reference Kapp2013). Dichotomous dependent variables were defined for first occurrence of each of the following outcomes:
Mental and physical disorders
Suicide attempt was defined based on the DoD Suicide Event Reporting system and International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) treatment codes. A positive test for illicit drug use was defined based on the Army substance abuse tracking database. Three health outcomes were defined from electronic medical records: mental hospitalization, traumatic brain injury (TBI); and any other severe injury exclusive of TBI (e.g. blindness, deafness, amputation, severe burns, paralysis), all based on ICD-9-CM codes.
Violent crime
DoD criminal justice databases were used to define three measures of violent crime perpetration [major physical, sexual assault, and minor violence (e.g. harassment)] and two of victimization (minor violence, sexual assault) coded according to the Bureau of Justice Statistics National Corrections Reporting Program classification system (U.S. Department of Justice, 2011) and of sufficient frequency to develop prediction models. The perpetration outcomes were defined from records of ‘founded’ offenses (i.e. where the Army found sufficient evidence to warrant full investigation). The victimization outcomes were defined using any officially reported victimization regardless of evidence.
Army career problems
Premature attrition from service due to career or personal problems (e.g. a character or behavior disorder, disability) was defined using an Army personnel database that tracks reasons for separating from service. Demotion was defined using information in the Army master personnel file.
Independent variables
Our goal was to optimize classification of soldiers who subsequently had the outcomes rather than to test specific hypotheses. As a result, we considered all potential predictors for which there was any evidence in the literature; e.g. risk-protective factors for suicidality (Nock et al. Reference Nock, Deming, Fullerton, Gilman, Goldenberg, Kessler, McCarroll, McLaughlin, Peterson, Schoenbaum, Stanley and Ursano2013; Afifi et al. Reference Afifi, Taillieu, Zamorski, Turner, Cheung and Sareen2016), mental hospitalization (Iribarren et al. Reference Iribarren, Sidney, Jacobs and Weisner2000; Rytila-Manninen et al. Reference Rytila-Manninen, Lindberg, Haravuori, Kettunen, Marttunen, Joukamaa and Frojd2014), substance problems (Kilpatrick et al. Reference Kilpatrick, Acierno, Saunders, Resnick, Best and Schnurr2000; Kirst et al. Reference Kirst, Mecredy, Borland and Chaiton2014), TBI (Cassidy et al. Reference Cassidy, Carroll, Peloso, Borg, von Holst, Holm, Kraus and Coronado2004; Elmasry et al. Reference Elmasry, Boivin, Feng, Packnett and Cowan2017), other severe physical injuries (Bulzacchelli et al. Reference Bulzacchelli, Sulsky, Rodriguez-Monguio, Karlsson and Hill2014; Theodoroff et al. Reference Theodoroff, Lewis, Folmer, Henry and Carlson2015), violence perpetration (Dahlberg, Reference Dahlberg1998; Elbogen et al. Reference Elbogen, Fuller, Johnson, Brooks, Kinneer, Calhoun and Beckham2010), violence victimization (Suris & Lind, Reference Suris and Lind2008; Turchik & Wilson, Reference Turchik and Wilson2010), and career problems (Knapik et al. Reference Knapik, Jones, Hauret, Darakjy and Piskator2004; Booth-Kewley et al. Reference Booth-Kewley, Highfill-McRoy, Larson and Garland2010). In total, 727 independent variables were operationalized from the SAQ in addition to eight performance-based neurocognitive test measures assessed in conjunction with the SAQ and 37 basic administrative variables recorded for all new soldiers at the beginning of service (772 total variables) (online Supplementary Appendix Table 1 summarizes all independent variables).
The SAQ variables were in six categories that included: sociodemographics (e.g. age, sex, race-ethnicity), self-reported lifetime history of DSM-IV mental disorders (ADHD, bipolar disorder, conduct disorder, generalized anxiety disorder, major depressive disorder, oppositional-defiant disorder panic disorder, PTSD, substance abuse-dependence), exposure to stressors (childhood adversities, other lifetime traumatic stressors, past-year stressful life events and difficulties), personality (e.g. neuroticism, impulsivity, secure attachment), social networks (e.g. number of friends, number of sexual partners), and lifetime suicidality/non-suicidal self-injury (referred to henceforth as ‘self-harm’).
The neurocognitive variables, described in more detail elsewhere (Moore et al. Reference Moore, Gur, Thomas, Brown, Nock, Savitt, Keilp, Heeringa, Ursano and Stein2017), assessed seven constructs: mental flexibility, attention, working memory, impulse control, facial memory, emotion identification, and bias toward negative emotions. A standardized efficiency score (the average of test accuracy and speed) was defined for each neurocognitive construct along with a composite overall efficiency score across the seven constructs. The administrative variables included Armed Forces Qualification Test (AFQT) scores, physical profile system (PULHES) scores, enlistment military occupational specialty classifications, and a series of indicators of enlistment waivers.
Analysis methods
Analysis was carried out remotely by Harvard Medical School analysts on the secure University of Michigan Army STARRS Data Coordination Center server. Given that respondents differed in number of months of follow-up, we calculated survival curves for each outcome based on observed outcome distributions to estimate number of respondents who would have each outcome if they were all followed 24 months after enlistment (24-month morbid risk) using the actuarial method (Halli & Rao, Reference Halli and Rao1992) implemented in SAS Proc LIFETEST (SAS Institute Inc., 2010). We projected morbid risk to 24 months despite our data going to 33 months because the number of soldiers followed beyond 24 months was too small for projection.
Discrete-time survival analysis with person-month the unit of analysis and a logistic link function (Willett & Singer, Reference Willett and Singer1993) was used to develop a separate prediction model for first occurrence of each outcome. As noted above, our goal was to maximize classification rather than to test hypotheses about specific predictors, leading us to consider all potential predictors in the models. The major danger in doing this was overfitting (Ritchie, Reference Ritchie2005; Upstill-Goddard et al. Reference Upstill-Goddard, Eccles, Fliege and Collins2013). We addressed this problem by using the elastic net penalized regression method (Zou & Hastie, Reference Zou and Hastie2005) implemented in the R-package glmnet (Friedman et al. Reference Friedman, Hastie and Tibshirani2010) to select an optimal subset of predictors for each final model. Penalized regression methods are designed either to use shrinkage to include multiple highly correlated predictors in a single model, to select the most stable single predictor in each highly correlated set to represent all predictors in the set, or to use some combination of both approaches in the service of maximizing out-of-sample classification accuracy at the expense of coefficient accuracy. Given that performance-based neurocognitive test data collection was time-consuming (20 min administration time) and required special software, elastic net was implemented both with and without the neurocognitive measures to evaluate their incremental importance.
Prior to using elastic net, univariate associations of each potential predictor were estimated with each outcome controlling BCT site and time of data collection in SAS proc logistic (SAS Institute Inc., 2010). Functional forms of significant non-dichotomous predictors were transformed to capture simple nonlinearities. The elastic net analysis was then limited to significant univariate predictors. Once final elastic net models were estimated, individual-level predicted probabilities were calculated for each outcome and area under the receiver operating characteristic curve (AUC) was generated from these predicted probabilities to evaluate overall model fit. The sample was then divided into 20 groups of equal size (ventiles) for each outcome based on predicted probabilities of the outcome. When concentration of risk (COR; the observed proportion of realizations of a given outcome in a given ventile) was at least 15% among soldiers in the top-risk ventile (i.e. three times the expected value), we examined model coefficients and inspected COR across all 20 ventiles of predicted risk of that outcome.
Results
Morbid risk and correlations among outcomes
The most common outcome was attrition, with a 24-month morbid risk of 189.8/1000 soldiers (Table 1). Demotion had the next highest morbid risk (60.8/1000 soldiers), followed by sexual assault victimization (37.7/1000 female soldiers), mental hospitalization (34.0/1000 soldiers), severe injury (29.9/1000 soldiers), positive drug test (18.9/1000 soldiers), and minor violence perpetration (11.2/1000 soldiers). All other outcomes had 24-month morbid risk <10.0/1000 soldiers. Several outcomes were strongly inter-correlated (Table 2), with tetrachoric correlations of 0.84 for suicide attempt with mental hospitalization, 0.48–0.76 for major physical perpetration with the other two perpetration outcomes, 0.54 for minor violence perpetration with minor violence victimization, and 0.55–0.70 for positive drug test with attrition and demotion.
Table 1. 24-month morbid risk per 1000 soldiers and incidence per 1000 person-years of adverse outcomes in the New Soldiers Study (n = 21832) a
s.e., standard error; n, number of observed cases of the outcome in the sample.
a Although the same sample of new soldiers was used for all outcomes, the number of person-months varied across outcomes because we predicted first occurrence of each outcome and each sample was censored separately either after the month when the outcome first occurred, termination of Regular Army service, or December 2013, whichever came first. The range of person-months was between 52 842 (to predict sexual assault victimization among women) and 4 20 706 (to predict attrition).
b Morbid risk was estimated as the number of new soldiers predicted to have each outcome within 24 months of the beginning of service based on an actuarial projection (Halli & Rao, Reference Halli and Rao1992). The projection was made to 24 months even though we had data for up to 33 months for some respondents because the number of respondents became too small for estimation beyond 24 months.
c Incidence was estimated from the person-month file as the observed proportion of person-months with a realization of the outcome multiplied by 12 000 (month-to-year conversion × 1000). There is no necessary relationship between incidence and morbid risk, as the former is based on a dataset that extends for between 13 and 33 months and depends on the timing of occurrences, whereas the former is projected only to 24 months and would have the same value whether outcomes occurred early or late in that time period.
Table 2. Tetrachoric correlation matrix for all 12 observed outcomes in the total sample (n = 21 832) a
a Correlations are estimated at the person-level ignoring duration of the follow-up period.
b These correlations could not be estimated because these involve opposite-sex sex-specific outcomes.
Model accuracy
The number of predictors selected by elastic net was 3–29 (median = 14) across outcomes. AUC for the models ranged between 0.62 (mental hospitalization, severe injury) and 0.80 (TBI) (Table 3). Focusing on the nine outcomes with top-ventile COR above the minimum pre-specified level of 15%, 32.1–38.2% of new soldiers with three outcomes (major physical and sexual assault perpetration, TBI) and 21.6–29.8% of those with four other outcomes (minor violence perpetration, sexual assault victimization, suicide attempt, positive drug test) were among the 5% in the highest predicted risk ventiles for those outcomes (Fig. 1). The 40.5–46.5% of new soldiers with each of four outcomes (major physical and sexual assault perpetration, suicide attempt, TBI) were among the 10% in the two highest ventiles for those outcomes; and 55.8–57.2% of new soldiers with two outcomes (sexual assault perpetration, TBI) were among the 15% in the three highest ventiles for those outcomes. Morbid risk among new soldiers in the highest-risk ventile ranged widely due to the variations in overall morbid risk and COR, from a high of 501.1/1000 soldiers for attrition to a low of 19.9/1000 for TBI.
Fig. 1. Proportion of outcome observations within each ventile of predicted risk derived from final modelsFootnote † Footnote 1 .
Table 3. Performance of the final discrete-time survival model for each outcome a
AUC, area under the receiver operating characteristic curve.
a Each ventile represents 5% of the soldiers in the sample ranked in terms of their predicted risk of each outcome
b Concentration of risk refers to the percent of all observed occurrences of the outcome in a ventile or ventiles of the predicted risk distribution.
c Morbid risk was defined as the number of new soldiers within a ventile of predicted risk subsequently predicted to have each outcome within 24 months of the beginning of service based on the actuarial method. The projection was made to 24 months even though we had data for up to 33 months for some respondents because the number of respondents became too small for estimation beyond 24 months.
Model predictors
As noted above in the section on analysis methods, penalized regression maximizes classification accuracy at the expense of coefficient accuracy, making it important to focus more on model performance than on the specific predictors that entered the models. It is nonetheless noteworthy that 40.7% of the 772 potential predictors had significant (0.05-level, two-sided test) univariate associations with mental hospitalization, followed by 25.9–26.2% with suicide attempt and positive drug test, 16.7–17.9% with minor violence perpetration and sexual assault victimization, and no more than chance (4.8–6.3%) with the other outcomes (Table 4). As the potential predictors were unequally distributed across classes, we focused on observed/expected (O/E) predictor ratios in final models. Personality measures were proportionally overrepresented in the final models for seven outcomes, sociodemographics for six, stressors and administrative variables for three, mental disorders for two, and the other classes of predictors for 0–1 outcomes (odds ratios and 95% confidence intervals of predictors in final models are available on request).
Table 4. Proportion of significant univariate predictors and final-model predictors across the eight predictor categories a
Neurocog, neurocognitive; Admin, adminstrative Army career variables; U, univariate; O/E, observed number of final model predictors divided by expected number of final model predictors; FM, final model; P, proportion; N, number.
a Results are shown for the nine outcomes that achieved a concentration of risk in the top-ventile of risk at least three times the expected value (i.e. 15.0%⩽; see Table 3). Empty cells (–) indicate predictor categories that had no significant univariate associations or predictor categories that were not represented in a final model. Coefficients (odds ratios) for the final model predictors are available on request.
b These columns report the percent of predictors within each predictor category with significant univariate associations with outcome (0.05 level, two-sided tests).
c This ratio refers to the observed representation of the column predictor category divided by the expected representation of the column predictor category. For example, there was one sociodemographic variable selected among the 10 total final model variables for major physical perpetration – 10% representation. However, 114 of all predictors were sociodemographic variables. – 14.8%. This ratio (10.0/14.8) is equal to 0.7.
d This column reports the total proportion of all predictors with significant univariate associations with each outcome.
e This column reports the total number of final model predictors selected by elastic net.
Several predictors selected by elastic net emerged in two or more final models. The common sociodemographics associated with elevated risk included minority status (predicting all violence perpetration outcomes, TBI, positive drug test), female sex (minor violence victimization, suicide attempt), low education (minor violence perpetration, positive drug test), and high religiosity (sexual assault perpetration, positive drug test). The most important mental disorder predictors associated with increased outcome risk were anger attacks (predicting major physical and minor violence perpetration, positive drug test), substance use disorders (minor violence perpetration, mental hospitalization, positive drug test), insomnia (minor violence perpetration and victimization, positive drug test), childhood behavioral disorders (minor violence perpetration, sexual assault perpetration and victimization, TBI, positive drug test), anxiety disorders (sexual assault perpetration, TBI, mental hospitalization, suicide attempt), total number of lifetime disorders (major physical and minor violence perpetration and victimization, mental hospitalization, suicide attempt), and lifetime treatment of mental disorders (major physical perpetration, suicide attempt).
The most important stressors included various chronic strains that occurred in the year prior to enlistment (predicting major physical perpetration, minor violence victimization, TBI), childhood physical abuse and physical assault victimization (sexual assault perpetration, TBI, positive drug test), family history of mental illness (sexual assault perpetration and victimization, TBI, mental hospitalization, suicide attempt), living in a foster home as a child (minor violence and sexual assault victimization), and number of lifetime stressors (all mental–physical disorder outcomes). Although a broad range of personality variables entered the different final models, only two were included in 2+ models: extroverted personality (predicting elevated risk in all three perpetration models); and non-secure attachment styles (predicting elevated risk of sexual assault perpetration, minor violence victimization, TBI, mental hospitalization, suicide attempt).
The remaining four groups of predictors were much less prominent in the final models. The only consistent predictor involving social networks was information about number of sexual partners (predicting elevated risk of minor violence perpetration, sexual assault perpetration, and positive drug tests). Indicators of self-harm were associated with elevated risk of sexual assault perpetration, mental hospitalization, and suicide attempt. The only administrative variable entering the models for multiple outcomes was the enlistment AFQT score, which inversely predicted risk of major physical and minor violence perpetration, TBI, and suicide attempts. Only one neurocognitive measure entered any final model: composite test efficiency predicted increased risk of TBI and improved top-ventile COR by 8.0% (from 38.2% to 46.2%).
Cross-outcome risk
Not surprisingly given that some outcomes were significantly inter-correlated, some composite predicted risk scores based on the final models were also significantly inter-correlated (detailed results available on request), raising a question whether predicted risk scores of Outcome A ever incrementally predicted Outcome B over the predict risk score for Outcome B. We explored this possibility by estimating models where each outcome was regressed on the predicted risk score for that outcome in addition to the predicted risk scores for the other outcomes. Only a handful of cross-outcome predictions were statistically significant and none improved COR in high-risk ventiles (detailed results available on request).
Discussion
We found that self-report questionnaire data collected at the beginning of service can be used to develop risk models with high top-ventile concentrations of risk for a number of subsequent negative soldier outcomes in the early years of service. Of particular note: 40.5% of suicide attempts occurred among the 10% of new soldiers with highest predicted risk of that outcome, 34.0% of male major physical perpetrators were among the 5% with highest predicted risk of that outcome, 57.2% of male sexual assault perpetrators were among the 15% with highest predicted risk of that outcome, and 35.5% of female sexual assault victims were among the 10% with highest predicted risk of that outcome.
Caution is needed in interpreting the importance of the specific predictors in our final models because penalized regression methods maximize overall model performance at the expense of individual coefficient accuracy. Nonetheless, three observations are noteworthy about these predictors.
First, the vast majority of the signs of the associations between predictors and outcomes are consistent with those in prior studies of military personnel (Suris & Lind, Reference Suris and Lind2008; Elbogen et al. Reference Elbogen, Fuller, Johnson, Brooks, Kinneer, Calhoun and Beckham2010; Turchik & Wilson, Reference Turchik and Wilson2010; Nock et al. Reference Nock, Deming, Fullerton, Gilman, Goldenberg, Kessler, McCarroll, McLaughlin, Peterson, Schoenbaum, Stanley and Ursano2013; Afifi et al. Reference Afifi, Taillieu, Zamorski, Turner, Cheung and Sareen2016) and civilians (Dahlberg, Reference Dahlberg1998; Kilpatrick et al. Reference Kilpatrick, Acierno, Saunders, Resnick, Best and Schnurr2000; Kirst et al. Reference Kirst, Mecredy, Borland and Chaiton2014; Rytila-Manninen et al. Reference Rytila-Manninen, Lindberg, Haravuori, Kettunen, Marttunen, Joukamaa and Frojd2014). The major exception is that religiosity is positively associated with risk of sexual assault perpetration and positive drug tests. Religiosity is usually (Miller et al. Reference Miller, Davies and Greenwald2000; Nonnemaker et al. Reference Nonnemaker, McNeely and Blum2003; Salas-Wright et al. Reference Salas-Wright, Vaughn, Hodge and Perron2012) found to be protective against these outcomes. However, at least three previous studies found religiosity to be associated with increased risk of similar outcomes (Jeffords, Reference Jeffords1984; Schensul & Burkholder, Reference Schensul and Burkholder2005; Herman-Stahl et al. Reference Herman-Stahl, Krebs, Kroutil and Heller2007). An understanding of why this might be the case would require more focused investigation.
Second, personality is the only predictor category with an O/E ratio consistently greater than 1.0 across outcomes. Few previous studies have examined personality predictors of negative soldier outcomes (MacManus et al. Reference MacManus, Dean, Al Bakir, Iversen, Hull, Fahy, Wessely and Fear2012a , Reference MacManus, Dean, Iversen, Hull, Jones, Fahy, Wessely and Fear b ). Given our results, though, additional research might be warranted to compare predictive validity of the personality measures in the NSS with personality measures collected independently by the Army (Drasgow et al. Reference Drasgow, Stark, Chernyshenko, Nye, Hulin and White2012) as pre-accession screens for negative outcomes (Niebuhr et al. Reference Niebuhr, Gubata, Oetting, Weber, Feng and Cowan2013).
Third, despite some evidence that objective measures of psychological characteristics have incremental validity over self-reports (Fuentes et al. Reference Fuentes, Tavares, Artes and Gorenstein2006; Back et al. Reference Back, Schmukle and Egloff2009; Huntjens et al. Reference Huntjens, Rijkeboer, Krakau and de Jong2014), the performance-based neurocognitive test variables considered here did not figure prominently in our final models despite having high proportions of significant univariate associations. This raises a question about the value of including these tests in future surveys of new soldiers.
Three study limitations are noteworthy. First, the NSS was described to new soldiers as an independent survey in which individual-level responses would not be shared with Army leaders. If model results of the sort reported here are used to target preventive interventions in future cohorts of new soldiers, results would have to be shared with Army leaders and respondents would have to be made aware of this fact before participating. This shift in auspices might alter reports in ways that reduce model performance. The STARRS results are nonetheless valuable in showing that respondent self-reports unencumbered by concerns about disclosure are capable of predicting the outcomes considered here with good accuracy. Based on these results, future studies might experiment with different approaches to motivate honest reporting in the context of respondents being told that results will be used to select soldiers for preventive interventions.
Second, our model results might not generalize beyond the 13–33 months of follow-up considered here and perhaps not beyond the 24 months for which we have a substantial sample. Replication of the current analyses over longer follow-up periods is needed to investigate this issue. Importantly, administrative data become richer over time, making it of interest in future long-term studies to examine the joint associations of baseline self-report measures and ongoing administrative measures in predicting the outcomes considered here.
Third, our use of administratively-recorded outcomes means that we excluded outcomes not reported to authorities (e.g. unreported sexual assault victimizations) and not detected by authorities (e.g. crime perpetrators who eluded authorities). Concern about this limitation is dampened, though, by two considerations. One is that administratively-recorded cases are often more severe than unrecorded cases. This is true, for example, of suicide attempts, where 100% of the severe attempts (i.e. those requiring hospitalization) need medical attention and are recorded in administrative records, even though some unknown number of less severe attempts are not known to authorities. The other is that administratively-recorded prevalence of these outcomes is high enough that prevention only of those cases would be of considerable value. Such interventions would also be expected to prevent some unreported cases, leading to conservative estimates of intervention cost-effectiveness based on administrative outcomes alone.
Conclusions
Within the context of these limitations, our results show that small subsets of new soldiers are responsible for high proportions of many of the negative outcomes considered here and that these high-risk new soldiers could be pinpointed at the beginning of their Army careers with models based on self-report data and basic administrative variables. These results argue strongly for the potential value of using self-report surveys with new soldiers to target preventive interventions if the issue of confidentiality could be addressed successfully. Of course, the ultimate value of prediction models based on such surveys depends on the broadly-defined costs (both direct costs and competing risks) of these outcomes to the Army and the individual soldier and the effectiveness of such interventions in terms of number needed to treat (NNT; i.e., the number of high-risk new soldiers who would need to be treated to prevent one instance of a focal outcome). However, as noted in the introduction, a number of interventions exist that are very promising both in terms of costs and effectiveness when considered for targeted implementation, making the results reported here of considerable value.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S003329171700071X.
Acknowledgements
Army STARRS was sponsored by the Department of the Army and funded under cooperative agreement (R.J.U. & M.B.S., cooperative agreement number U01MH087981) (2009–2015) with the U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (NIH/NIMH); Subsequently, STARRS-LS was sponsored and funded by the Department of Defense (R.J.U. & M.B.S., USUHS grant number HU0001-15-2-0004); This research was also funded by the Department of Defense, Office of the Assistant Secretary for Defense for Health Affairs, Defense Health Program (OASD/HA), awarded and administered by the U.S. Army Medical Research & Materiel Command (USAMRMC), at Fort Detrick, MD (R.C.K., under contract numbers W81XWH-12-2-0113, W81XWH-13-1-0037); and Dr Street's effort was supported with resources and the use of facilities at VA Boston Healthcare System, Boston, MA.
Declaration of Interest
Dr Stein reports being paid as a consultant by Actelion Pharmaceuticals, Dart NeuroScience, Janssen Pharmaceuticals, Resilience Therapeutics, and Oxeia Biopharmaceuticals; and for editorial work for UpToDate and the journal Biological Psychiatry. Dr Monahan is co-owner of Classification of Violence Risk (COVR), Inc. In the past 3 years, Dr Kessler received support for his epidemiological studies from Sanofi Aventis; was a consultant for Johnson & Johnson Wellness and Prevention, Shire, Takeda; and served on an advisory board for the Johnson & Johnson Services Inc. Lake Nona Life Project. Kessler is a co-owner of DataStat, Inc., a market research firm that carries out healthcare research. The remaining authors declare no conflict of interest.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Group Information
The Army STARRS Team consists of Co-Principal Investigators: Robert J. Ursano, MD (Uniformed Services University of the Health Sciences) and Murray B. Stein, MD, MPH (University of California San Diego and VA San Diego Healthcare System). Site Principal Investigators: Steven Heeringa, Ph.D. (University of Michigan) and Ronald C. Kessler, Ph.D. (Harvard Medical School). National Institute of Mental Health (NIMH) collaborating scientists: Lisa J. Colpe, Ph.D., MPH and Michael Schoenbaum, Ph.D. Army liaisons/consultants: COL Steven Cersovsky, MD, MPH [USAPHC (Provisional)] and Kenneth Cox, MD, MPH [USAPHC (Provisional)]. Other team members: Pablo A. Aliaga, MA (Uniformed Services University of the Health Sciences); COL David M. Benedek, MD (Uniformed Services University of the Health Sciences); K. Nikki Benevides, MA (Uniformed Services University of the Health Sciences); Paul D. Bliese, Ph.D. (University of South Carolina); Susan Borja, Ph.D. (NIMH); Evelyn J. Bromet, Ph.D. (Stony Brook University School of Medicine); Gregory G. Brown, Ph.D. (University of California San Diego); Laura Campbell-Sills, Ph.D. (University of California San Diego); Catherine L. Dempsey, Ph.D., MPH (Uniformed Services University of the Health Sciences); Carol S. Fullerton, Ph.D. (Uniformed Services University of the Health Sciences); Nancy Gebler, MA (University of Michigan); Robert K. Gifford, Ph.D. (Uniformed Services University of the Health Sciences); Stephen E. Gilman, ScD (Harvard School of Public Health); Marjan G. Holloway, Ph.D. (Uniformed Services University of the Health Sciences); Paul E. Hurwitz, MPH (Uniformed Services University of the Health Sciences); Sonia Jain, Ph.D. (University of California San Diego); Tzu-Cheg Kao, Ph.D. (Uniformed Services University of the Health Sciences); Karestan C. Koenen, Ph.D. (Columbia University); Lisa Lewandowski-Romps, Ph.D. (University of Michigan); Holly Herberman Mash, Ph.D. (Uniformed Services University of the Health Sciences); James E. McCarroll, Ph.D., MPH (Uniformed Services University of the Health Sciences); James A. Naifeh, Ph.D. (Uniformed Services University of the Health Sciences); Tsz Hin Hinz Ng, MPH (Uniformed Services University of the Health Sciences); Matthew K. Nock, Ph.D. (Harvard University); Rema Raman, Ph.D. (University of California San Diego); Holly J. Ramsawh, Ph.D. (Uniformed Services University of the Health Sciences); Anthony Joseph Rosellini, Ph.D. (Harvard Medical School); Nancy A. Sampson, BA (Harvard Medical School); CDR Patcho Santiago, MD, MPH (Uniformed Services University of the Health Sciences); Michaelle Scanlon, MBA (NIMH); Jordan W. Smoller, MD, ScD (Harvard Medical School); Amy Street, Ph.D. (Boston University School of Medicine); Michael L. Thomas, Ph.D. (University of California San Diego); Leming Wang, MS (Uniformed Services University of the Health Sciences); Christina L. Wassel, Ph.D. (University of Vermont); Simon Wessely, FMedSci (King's College London); Christina L. Wryter, BA (Uniformed Services University of the Health Sciences); Hongyan Wu, MPH (Uniformed Services University of the Health Sciences); LTC Gary H. Wynn, MD (Uniformed Services University of the Health Sciences); and Alan M. Zaslavsky, Ph.D. (Harvard Medical School).
Disclaimer
The contents are solely the responsibility of the authors and do not necessarily represent the views of the Department of Health and Human Services, NIMH, or the Department of the Army, or the Department of Defense. No official endorsement should be made.
