Older data are becoming increasingly disfavored for publication. For example, JAMA states in its author instructions for submission of articles that manuscripts with most recent collection of data >5 years ago will have lower publication priority (JAMA Network, 2021), and authors are asked to ‘provide a detailed explanation of the relevance of the information in light of current knowledge and medical practice as well as the most recent date(s) of analysis of the study’. The preferred age limit of data collection for the American Journal of Public Health (2021) is <3 years, which deems manuscripts with most recent data collection of data beyond this time window to have ‘insufficient priority.’ A recent analysis of clinical trials published in high-impact journals (Welsh et al., Reference Welsh, Lu, Dhruva, Bikdeli, Desai, Benchetrit and Krumholz2018) did not define the duration of ‘older’ data but stated that recent data are preferable.
Like older data, older references are also disfavored. As with older data, consensus is lacking on what constitutes ‘old’ references, with definitions ranging from 10 to 15 and 20 years old (Patsopoulos & Ioannidis, Reference Patsopoulos and Ioannidis2009; Verstak et al., Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014). One journal's guidelines include recommendations to focus on recent articles with the strict instruction that references to studies ‘older than five years should not be included except for an overriding purpose’ (The Pharmaceutical Journal, 2014). An editor of another journal provided advice in a document entitled ‘Eight reasons why I rejected your article’, beginning with a complaint about ‘very old’ references (Thrower, Reference Thrower2012). Editors may put policies in place that favor very recent citations because this increases impact factor in general, and their journal's impact factor in particular when the citations are to their own journal. Many academic writers find such limitations to be restrictive, arbitrary, or even misleading about the source of prior knowledge. Future research comparing the effect of recency policies on impact factor and proper citation of research may be warranted. Reflecting such preferences for newer research, search functions in PubMed and Google Scholar now provide standard filters for inclusion of references by publication age, and the default setting in PubMed allows options for users to select a recency range of 1, 5, or 10 years.
The rationale for favoritism toward newness of data and references reflects a bias against aging data. An overriding assumption is that new data and new references are inherently superior to their older counterparts. The value of newer v. older data and references is pertinent to the reasoning by philosophers of science such as Popper, Kuhn, and Lakatos on the differentiation of true from false science. Popper (Reference Popper1983) held that data and theories must be testable by observation or testing to determine if they are empirically false, a process he termed the test of falsification, and any data or theory surviving this test is ‘good’ and those that cannot be tested in this way (e.g. many psychological theories) are pseudoscience. A conservative and heuristic view of Kuhn (Reference Kuhn1962) was that data and theories were useful for solving existing problems and puzzles, and new paradigms were needed only when major revolutionary changes occurred, leading to abandonment of prior research. Lakatos (Reference Lakatos1999) pointed out that theory often proceeds data. His perspective may have relevance to modern problems of the current COVID-19 pandemic, as some important elements of the previously existing theory on viruses did not apply to COVID-19. In response to urgent needs for scientific advances, COVID-19 science developed so rapidly that prior research was inadequate and in many respects outdated for this emergent situation.
Dichotomizing research into ‘old’ and ‘new’ creates an arbitrary duality that promotes dismissal of valuable knowledge from earlier time periods and an artificial limitation of accepted knowledge based on the value of newness. A myopic focus on newer research risks losing sight of important research questions already addressed by now-invisible older studies. This creates a ‘Groundhog Day’ effect as illustrated by the 1993 movie of this name in which the protagonist has to relive the same day (Groundhog Day) over and over and over within a world with no memory of it.
Scientific research publications that include modern methods, presentation, and interpretation have been available for more than 50 years for most medical specialties. Older publications provide a valuable basis for accepted facts, seminal observations, and insights deserving of citation, and new research builds from currently existing knowledge contributed by older studies. Older references and older data may have appropriate uses, but biases against them are increasingly apparent in the preferences of journal editors and reviewers for citations to newer publications. Given these perspectives, the purpose of this article is therefore to examine the consequences of the recent emphasis on the value of newer data and references in current publication practices and to stimulate new consideration of the utility of selected older data and references for the advancement of scientific knowledge.
The status of older data
Bias against older data assumes the data have no relevance or usefulness because of its age, without regard to any methodological strengths or unique design characteristics. A pivotal question for many reviewers and readers is: how can data collected years ago still be useful or relevant? The following comment by a reviewer to the authors of this manuscript illustrates this concern and criticism well: ‘The focus on DSM-III criteria is antiquated as the field is well onto DSM-5….Some of the diagnostic criteria…have changed since DSM-III was published 40 years ago….and [are] significantly outdated.’ (Additionally, there is no strong evidence that subsequent editions substantially improved the validity or reliability of the criteria, and even small changes in diagnostic criteria made in each new edition hinder the comparison of older and newer data collected with different versions of the criteria and augment the insecurity of psychiatric nosology.) The reviewer went on to state, ‘If the authors choose to continue using this dataset for publication, it should be clearly stated as a limitation with a rationale for this decision. This sample does not seem easily generalizable to modern samples given the time period….’
An important example of older research data with continuing relevance and foundational importance to the field of medicine is the Framingham Heart Study (Dawber, Meadors, & Moore, Reference Dawber, Meadors and Moore1951; Dawber, Moore, & Mann, Reference Dawber, Moore and Mann1957; Feinleib, Reference Feinleib1985). This epidemiologic investigation of a cohort of 4739 individuals, initiated in 1948 by the US Public Health Service to prospectively investigate the epidemiology and risk factors for cardiovascular disease, continued longitudinal data collection over 35 years through 1983. The impact of the Framingham study was captured by the title of a 2018 video by Boston University School of Medicine, ‘70 Years, 15,000 Participants and 3716 Research Papers: Celebrating the Framingham Heart Study’ (Back2BU, 2018).
An example of such a foundational dataset in psychiatry is the landmark Epidemiologic Catchment Area (ECA) study (Regier et al., Reference Regier, Myers, Kramer, Robins, Blazer, Hough and Locke1984; Robins et al., Reference Robins, Helzer, Weissman, Orvaschel, Gruenberg, Burke and Regier1984). This longitudinal study, initiated in 1980 with subsequent waves of follow-up over 24 years, remains the first, and an unduplicated, study of the population prevalence of psychiatric disorders with methodologic rigor. It examined the prevalence of psychiatric disorders in the American general population in a sample of 20 000 adults systematically recruited from residential and institutional sources in urban and rural settings, using structured interviews to assess psychiatric disorders according to established diagnostic criteria. The database from this classic study has been utilized for secondary analysis of a vast array of topics in articles published to this very year on many topics, including mood disorders, posttraumatic stress disorder, panic attacks, alcohol and drug dependence, homelessness, antisocial personality disorder, homosexuality, gastrointestinal disease, diabetes, cancer, asthma, stroke, Parkinson's disease, head injury, headaches, sleep disturbance, stress and cognition, obesity, suicidality, social support, healthcare utilization, and mortality (Crum, Storr, Chan, & Ford, Reference Crum, Storr, Chan and Ford2004; Edwards & Mezuk, Reference Edwards and Mezuk2012; Fleury, Grenier, Bamvita, Perreault, & Caron, Reference Fleury, Grenier, Bamvita, Perreault and Caron2014; Glass & Bucholz, Reference Glass and Bucholz2011; Goodwin & Eaton, Reference Goodwin and Eaton2005; Gross, Gallo, & Eaton, Reference Gross, Gallo and Eaton2010; Henriksen, Mather, Mackenzie, Bienvenu, & Sareen, Reference Henriksen, Mather, Mackenzie, Bienvenu and Sareen2014; Krasnova, Eaton, & Samuels, Reference Krasnova, Eaton and Samuels2019; Kubzansky, Koenen, Jones, & Eaton, Reference Kubzansky, Koenen, Jones and Eaton2009; Larson, Owens, Ford, & Eaton, Reference Larson, Owens, Ford and Eaton2001; Lauterbach, Freeman, & Vogel, Reference Lauterbach, Freeman and Vogel2004; Maulik, Eaton, & Bradshaw, Reference Maulik, Eaton and Bradshaw2009; Mezuk, Lohman, Dumenci, & Lapane, Reference Mezuk, Lohman, Dumenci and Lapane2013; Munro et al., Reference Munro, Wennberg, Bienko, Eaton, Lyketsos and Spira2019; North, Alpers, Thompson, & Spitznagel, Reference North, Alpers, Thompson and Spitznagel1996; North, Dinwiddie, Cottler, & Spitznagel, Reference North, Dinwiddie, Cottler and Spitznagel1991; North, Eyrich, Pollio, & Spitznagel, Reference North, Eyrich, Pollio and Spitznagel2004; North, Kotamarti, & Pollio, Reference North, Kotamarti and Pollio2021; Ramsey, Leoutsakos, Mayer, Eaton, & Lee, Reference Ramsey, Leoutsakos, Mayer, Eaton and Lee2010; Silver, Kramer, Greenwald, & Weissman, Reference Silver, Kramer, Greenwald and Weissman2001; Woolley, Fredman, Goethe, Lincoln, & Heeren, Reference Woolley, Fredman, Goethe, Lincoln and Heeren2008).
The status of older references
If older references are considered inferior, they might be expected to be eclipsed by the growing numbers of newer citations in more recent publications. Logically, older publications have a natural inherent advantage in their likelihood of citation in academic literature, as the earlier an article was published, the more time there has been for it to be cited (Aksnes, Langfeldt, & Wouters, Reference Aksnes, Langfeldt and Wouters2019; Mazhari, Reference Mazhari2013). However, because important randomized clinical trials have an average publication delay of 3 years, they may receive sparse or no citation (Aksnes, Reference Aksnes2005). Consensus is lacking as to the primacy of older and newer references in general scientific citation.
Jemielniak, Masukume, and Wiliamowski (Reference Jemielniak, Masukume and Wiliamowski2019) observed that Wikipedia citations of recently published general medicine articles have increased disproportionately relative to older publications. They referred to this phenomenon as ‘recentism’, describing it as ‘where an article has an inflated or imbalanced focus on recent events. It is writing without an aim toward a long-term historical view’. A review of meta-analytic studies in the Cochrane Database of medical interventions by Patsopoulos and Ioannidis (Reference Patsopoulos and Ioannidis2009) pointed out that citations of older articles may have inherent problems in their obsolescence or irrelevance. They noted, however, that older studies do not necessarily suffer in quality, sample size, or external validity compared to newer studies. They cautioned that such ‘evidence should not be undervalued simply because of its age’ (p. 66) and that studies should instead be judged by careful scrutiny of their research methods.
There is some evidence that citation of older references may be expanding in science and medicine specifically as well as more broadly in other fields (Sluyter, Reference Sluyter2005). In contrast to the preference for more recent citations described by Jemielniak et al. (Reference Jemielniak, Masukume and Wiliamowski2019), Verstak et al. (Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014) observed that citations of older publications in Google Scholar Metrics increased disproportionately relative to newer publications between 1990 and 2013 across the humanities, business, social science, physics, life sciences, medical sciences, computer sciences, engineering, and chemical sciences. References older than 10 years increased from 10% to 56% in the business field, from 29% to 34% in the engineering literature, and from 3% to 33% in medical sciences over this 23-year period. In their article, Verstak et al. (Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014) suggested that the availability of search engines and on-line references or full-text articles has made older publications more widely known, available, and readily cited.
The literature is thus not clear as to the primacy of citations of older v. newer studies, but older landmark studies such as the Framingham Study and the ECA Study remain important to science today. Such historical studies are not likely to be conducted again and their original findings will retain their importance and value, thus deserving ageless citation. The most cited article in science, describing a fundamental laboratory procedure for measuring proteins, goes back to 1951 (Lowry, Rosebrough, Farr, & Randall, Reference Lowry, Rosebrough, Farr and Randall1951) with >300 000 citations (van Noorden, Maher, & Nuzzo, Reference van Noorden, Maher and Nuzzo2014). The natural citation advantage for older articles was likely an important contributor for this 70-year-old article. The most cited article in psychiatry, introducing a cognitive screening test that came into wide usage, was published in 1975 (Folstein, Folstein, & McHugh, Reference Folstein, Folstein and McHugh1975), with >28 000 citations (Mazhari, Reference Mazhari2013).
Illustrative examples of citation and data age in recent scientific articles
To illustrate the recency of references in published original medical research articles in a selected sample of recent academic medical journals, original research articles were examined (in April 2021, the current month at the time of the inception of this exercise) in the most recent issues in selected psychiatry, medicine, and surgery journals. This exercise is not intended as empirical evidence, but rather as illustrative of novel points this article advances independently. The total numbers of references in 5-year time categories in the last 15 years listed in the bibliographies of all original articles in each issue (or every other article if issues included >20 articles) were tabulated and summarized as raw counts and proportions. The years of the data collection in the articles included were also noted if stated in the article, and the journal impact factors for each journal were recorded if provided in the website. The psychiatry journals sampled for this effort were JAMA Psychiatry (eight articles with a total of 393 references) and the American Journal of Psychiatry (three articles with a total of 138 references). The surgery journals considered were the Journal of Orthopedic Research (11 articles with a total of 459 references) and the Journal of Gastrointestinal Surgery (13 articles with a total of 404 references). The general medicine journals examined were the New England Journal of Medicine (NEJM) (five articles with a total of 125 references) and the Lancet (three articles with a total of 101 references).
Of the articles reviewed, data collection had been completed within 3 years by 12 studies, within 5 years by four studies, between 6 and 9 years by nine studies, and none >9 years; 12 did not state the age of the data (not shown in figure). Additionally, two articles used pre-existing databases of varied or uncertain age, stating that they had analyzed the data within 1 year. Figure 1 lists the publication recencies of the references listed in the selected articles. The recency patterns were consistent across the different journals. In general, although approximately one-half of the references in the articles were from the past 5 years, fewer were 5–10 years old, and even fewer were 10–15 years old; the majority of the references in these articles had publication dates within the last 10 years and the vast majority in the last 15 years.
Fig. 1. Recencies of publication dates for references in journal articles sampled.
Groundhog Day
The illustrative examination undertaken above is exemplary of the bias we and others have observed toward recency in data and references asserted in this article, also reflecting the examples from journal guidelines to authors. Understandably, recent publications and newly breaking research findings, especially in high-impact journals, are valued in the medical literature. However, older data and references to older studies can also be very important for science and its advancement. Failure to utilize existing datasets sufficiently and discounting older research articles both promote loss of knowledge that was established long ago, leading to collection of the same data again for the same purposes. Repeating the same actions again and again is the basis of the plot in the movie Groundhog Day – the essence of the title of this article. In this movie, the protagonist learns from his Groundhog Day experience and makes changes the next time around, but the townsfolk cannot benefit from this experience because they never remember it. Essentially, every repeated day is a new experiment. In much the same way, failing to employ older data and reference older studies leaves readers uninformed. Older data and older references are necessary and of great value to avoid endless repetition of the same data collection and investigation of the same research questions.
Value of older data
There are four main types of older data that may have important value and utility. The first involves three types of longitudinal studies. These may include examination of changes over time within a single sample in prospective longitudinal study designs known as cohort studies. They may also include retrospective study designs starting with current samples of cohorts to identify earlier events or risk factors associated with current outcomes in case-control studies. Also, epidemiologic studies may examine different representative samples of a population over time to identify trends in selected variables of interest. Although these studies may span several decades and thus much of the longitudinal data in them may be quite old, if the completion of the study data collection was recent, articles written from such data are not as likely to be rejected or assigned low priority by journals for publication.
A second type of older studies includes classic or legacy studies that were conducted using rigorous research methods with far-reaching implications whose value has stood the test of time. Two examples of such studies were introduced earlier in this article. The famous population-based, 35-year longitudinal Framingham Study had second- and third-generation cohort studies launched from the original cohort, and many articles have been published from secondary analyses of this dataset. The ECA study, an epidemiologic population study of psychiatric disorders in America, has also had numerous articles published from secondary analysis of its data as well as analysis of the data in relation to externally available data on variables of interest in public sources. Although the data from these studies were not old in articles published shortly after the data collection was completed, in the years that have elapsed, the data have aged considerably. Despite the aging of these datasets, however, the findings remain respected, the original articles are still cited, and robust applications of the data continue to generate newly published articles to this day.
A third type of older studies is original research conducted by investigators who for various reasons did not generate published articles from the research, and thus the findings were not disseminated through publication of the work unless other researchers later managed to analyze the data and write up the results for publication after the data were considered ‘old’. A recent example was a remarkably well-conducted study of families with children who were exposed to dioxin contamination and other disasters in Times Beach and nearby areas in eastern Missouri in the early-to-mid 1980s. The death of the principal investigator interrupted progress in analysis and publication of the study, and only decades later was the project resurrected by the principal investigator's mentee and a mentee of the principal investigator's mentee who completed the data analysis and writing of a manuscript to document the findings (Lee et al., Reference Lee, Pfefferbaum, Portley, Kotamarti, Canan and North2021). Another study of patients hospitalized for alcoholism in 1967 conducted a prospective longitudinal follow-up of their outcomes over two decades, and although some of the main results were published in a handful of articles within a few years of the completion of data collection, colleagues resurrected the project many years later to continue publication of additional main findings of this study (Lewis & North, Reference Lewis and North2021).
A fourth type of older studies includes datasets from research projects that have exhausted publication of the main findings but have utility for further analysis and publication of additional findings outside of the study's main objectives. These datasets may be subjected to later analysis by the principal investigator who was unable to conduct additional analyses for publication in addition to those of the main study objectives until time has gone by, or may be utilized by trainees as sources of data for testing hypotheses that may be peripheral or even unrelated to the study's main purposes and generation of published manuscripts from them. A recently increasing practice is for sharing databases from published studies to allow analysis and use by colleagues and other researchers and the public.
An often-overlooked contribution of older studies is the new knowledge they may provide that has not been reported even by more recent publications, despite the dated nature of the data. Just because the data were collected years ago does not automatically render the findings obsolete. Another advantage of available data from older studies is that analysis and publication of data from them is efficient and cost-effective, as new grant applications and funding are not needed, and the expense and burden of recruiting new research samples and collection of new data are not needed. Investigators who receive major federal grants do not often have enough time to complete all the data collection and analyze and publish all the main findings before they must write and obtain more grants requiring them to collect more data that they cannot publish before the end of the grant. This process may go on for decades, leaving investigators with a great deal of excellent but unpublished data.
As available research funding has become more competitive and younger investigators lack research funding and effort, older datasets can be leveraged by applying new methods and techniques in fresh attempts to solve longstanding medical problems using existing databases by conducting secondary analyses, reducing the tragic waste of the accumulation of unpublished data. This can aid research trainees and young faculty members who do not have their own independent research, but do not have time to complete the labor-intensive design and implementation of an original study with new data collection. With projects involving secondary analysis, trainees and young faculty can proceed directly to data analysis and writing up the results.
Recognizing these research efficiencies, many federal funding agencies and journals request that data collected with federal funds or published in academic journals be made available to qualified researchers. A further use of older data is to provide a source of material for the testing of new hypotheses and exploring phenomena outside of current understanding, such as symptoms outside of existing diagnostic constructs, and for exploring novel constructs. Lacking modern laboratory and imaging diagnostic methods, older research in psychiatry elaborated rich clinical detail. Reliance on signs and symptoms remains the basis of diagnosis more for psychiatry than for other medical disciplines that have increasingly incorporated biological evidence, making the older studies particularly relevant for the field of psychiatry. Older datasets have advantages not realized by more recent studies that have substituted self-report symptom checklists for full diagnostic assessments used in many older studies. New insights from older datasets may also emerge from the application of more recent advances in statistical methods, including multiple regression analysis; factor, cluster, and latent class analysis; path analysis; and neural network modeling, machine learning, and artificial intelligence techniques. It may not be necessary for these older studies to be lost in the fog of time if the data can be revisited with newer statistical methods to extract new findings.
Value of older references
Older references may be underappreciated for the important conceptual, historical, and philosophical contributions to the progression of scientific knowledge and advances they may bring. Older articles provide a historical context, and the references they contain may offer specific details demonstrating additional relevance to current problems. For example, the COVID-19 pandemic has introduced urgent pressures for the development of effective treatments and preventive measures. The mRNA technology used for creating vaccines was established several decades ago. Had scientists not known of this technology published in older literature, the field would not have critical knowledge that it needed to produce the needed vaccines with sufficient timeliness. Historically, literature from more than a century ago regarding the Spanish flu pandemic has direct relevance to the present pandemic. The St. Louis Department of Health was credited as significantly reducing influenza transmission and mortality by mandating nonpharmaceutical interventions such as social distancing, banning large public gatherings, and closing schools, churches, and theaters (McKinsey, McKinsey, & Enriquez, Reference McKinsey, McKinsey and Enriquez2018). Modern policymakers have used this history to refine public health approaches for contemporary pandemic influenza preparedness.
Older literature may provide additional context as it heralds newly emerging paradigm shifts. In psychiatry, the release of DSM-III (American Psychiatric Association, 1980) represented a paradigm shift from theoretically-driven and etiologic approaches to diagnosis in DSM-II to empirically-based and scientifically-validated diagnostic criteria (Kendler, Munoz, & Murphy, Reference Kendler, Munoz and Murphy2010; Mayes & Horwitz, Reference Mayes and Horwitz2005; Surís, Holliday, & North, Reference Surís, Holliday and North2016). The context of this basic shift is remarkably relevant as the field continues to grapple with ongoing pressures to anoint new paradigms such as Research Domain Criteria (RDoC) and the Hierarchical Taxonomy of Psychopathology (HiTOP) (Michelini, Palumbo, DeYoung, Latzman, & Kotov, Reference Michelini, Palumbo, DeYoung, Latzman and Kotov2021). Following strict Kuhnian thinking, some of the resources of older paradigms may be lost. For example, with the paradigm shift that occurred with the introduction of operational definitions of psychiatric disorders in DSM-III, the rich clinical descriptions of published research in prior centuries and information contained in it may be forgotten because researchers may lack training in this art, lack time for it, or no longer value it.
There are good examples of older scientific work that have not held up over time. These include the initially popular dopamine theory of schizophrenia and the use of the dexamethasone suppression test in the diagnosis of depression that were eventually invalidated by subsequent research. An example of old references that continue to have relevance is found in the classic textbook Goodwin and Guze's Psychiatric Diagnosis – especially in the earlier editions – contains many very old references from different classical eras, including the 1960s and 1970s work of Eli Robins and Sam Guze in St. Louis, older psychiatric works of Briquet and Kraepelin, as well as sources from antiquity including Plato and even ancient Egyptian text written on papyrus (North & Yutzy, Reference North and Yutzy2018).
Conclusion
The dichotomization of old v. new data and old v. recent references is overly simplistic. Current data do not automatically lose value through aging in the way that ‘old milk’ becomes ‘spoiled’ after its ‘use by’ date. The current valuation of newer data above older data fails to appreciate the fact that new data eventually become old, and that old data were once new. Old data have an obvious context in that what has changed since the data were collected is known, but new data do not have this context because it is still evolving. Thus, the value of newer and older research is determined by the ‘context’ of when the data were collected and the references were provided. The contextual recommendations of this article are reflected in some of Meehl's stated reasons for lack of scientific progress (Meehl, Reference Meehl1978). Late-breaking laboratory research and current clinical data have value within the context of the current progress of the field, but not to the exclusion of older research findings.
The bias demonstrated in arbitrary rules pertaining to older data and older references can be addressed by instituting comparable treatment of older and newer data and references. Older research has the advantage of a context that is known at the point in time that the research becomes available, whereas the context of newer research can only be speculative and ephemeral. The latest research at any given time lacks the ability to be contextualized within the course of interventions being developed beyond the time frame of the research. For example, the delta variant of COVID-19 provided a different context for research investigation from that of the original (alpha) form of the virus. Even as this discussion is awaiting publication, the context of COVID-19 continues to demonstrate rapid metamorphosis, which may render this example more or less relevant, depending on what happens. This leads to two conclusions: (1) that context needs to be incorporated in all considerations of publication of all research, regardless of the age of the research, and (2) that research needs to be valued based on its contribution rather than its recency. Winston Churchill's time-honored advice that ‘Those who fail to learn from history are condemned to repeat it’ seems to apply here. Thus, rather than asking authors to provide justification of older data and references, a more balanced approach by journals might be to have authors learn from the historical context of the research in their manuscripts.
Older data are becoming increasingly disfavored for publication. For example, JAMA states in its author instructions for submission of articles that manuscripts with most recent collection of data >5 years ago will have lower publication priority (JAMA Network, 2021), and authors are asked to ‘provide a detailed explanation of the relevance of the information in light of current knowledge and medical practice as well as the most recent date(s) of analysis of the study’. The preferred age limit of data collection for the American Journal of Public Health (2021) is <3 years, which deems manuscripts with most recent data collection of data beyond this time window to have ‘insufficient priority.’ A recent analysis of clinical trials published in high-impact journals (Welsh et al., Reference Welsh, Lu, Dhruva, Bikdeli, Desai, Benchetrit and Krumholz2018) did not define the duration of ‘older’ data but stated that recent data are preferable.
Like older data, older references are also disfavored. As with older data, consensus is lacking on what constitutes ‘old’ references, with definitions ranging from 10 to 15 and 20 years old (Patsopoulos & Ioannidis, Reference Patsopoulos and Ioannidis2009; Verstak et al., Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014). One journal's guidelines include recommendations to focus on recent articles with the strict instruction that references to studies ‘older than five years should not be included except for an overriding purpose’ (The Pharmaceutical Journal, 2014). An editor of another journal provided advice in a document entitled ‘Eight reasons why I rejected your article’, beginning with a complaint about ‘very old’ references (Thrower, Reference Thrower2012). Editors may put policies in place that favor very recent citations because this increases impact factor in general, and their journal's impact factor in particular when the citations are to their own journal. Many academic writers find such limitations to be restrictive, arbitrary, or even misleading about the source of prior knowledge. Future research comparing the effect of recency policies on impact factor and proper citation of research may be warranted. Reflecting such preferences for newer research, search functions in PubMed and Google Scholar now provide standard filters for inclusion of references by publication age, and the default setting in PubMed allows options for users to select a recency range of 1, 5, or 10 years.
The rationale for favoritism toward newness of data and references reflects a bias against aging data. An overriding assumption is that new data and new references are inherently superior to their older counterparts. The value of newer v. older data and references is pertinent to the reasoning by philosophers of science such as Popper, Kuhn, and Lakatos on the differentiation of true from false science. Popper (Reference Popper1983) held that data and theories must be testable by observation or testing to determine if they are empirically false, a process he termed the test of falsification, and any data or theory surviving this test is ‘good’ and those that cannot be tested in this way (e.g. many psychological theories) are pseudoscience. A conservative and heuristic view of Kuhn (Reference Kuhn1962) was that data and theories were useful for solving existing problems and puzzles, and new paradigms were needed only when major revolutionary changes occurred, leading to abandonment of prior research. Lakatos (Reference Lakatos1999) pointed out that theory often proceeds data. His perspective may have relevance to modern problems of the current COVID-19 pandemic, as some important elements of the previously existing theory on viruses did not apply to COVID-19. In response to urgent needs for scientific advances, COVID-19 science developed so rapidly that prior research was inadequate and in many respects outdated for this emergent situation.
Dichotomizing research into ‘old’ and ‘new’ creates an arbitrary duality that promotes dismissal of valuable knowledge from earlier time periods and an artificial limitation of accepted knowledge based on the value of newness. A myopic focus on newer research risks losing sight of important research questions already addressed by now-invisible older studies. This creates a ‘Groundhog Day’ effect as illustrated by the 1993 movie of this name in which the protagonist has to relive the same day (Groundhog Day) over and over and over within a world with no memory of it.
Scientific research publications that include modern methods, presentation, and interpretation have been available for more than 50 years for most medical specialties. Older publications provide a valuable basis for accepted facts, seminal observations, and insights deserving of citation, and new research builds from currently existing knowledge contributed by older studies. Older references and older data may have appropriate uses, but biases against them are increasingly apparent in the preferences of journal editors and reviewers for citations to newer publications. Given these perspectives, the purpose of this article is therefore to examine the consequences of the recent emphasis on the value of newer data and references in current publication practices and to stimulate new consideration of the utility of selected older data and references for the advancement of scientific knowledge.
The status of older data
Bias against older data assumes the data have no relevance or usefulness because of its age, without regard to any methodological strengths or unique design characteristics. A pivotal question for many reviewers and readers is: how can data collected years ago still be useful or relevant? The following comment by a reviewer to the authors of this manuscript illustrates this concern and criticism well: ‘The focus on DSM-III criteria is antiquated as the field is well onto DSM-5….Some of the diagnostic criteria…have changed since DSM-III was published 40 years ago….and [are] significantly outdated.’ (Additionally, there is no strong evidence that subsequent editions substantially improved the validity or reliability of the criteria, and even small changes in diagnostic criteria made in each new edition hinder the comparison of older and newer data collected with different versions of the criteria and augment the insecurity of psychiatric nosology.) The reviewer went on to state, ‘If the authors choose to continue using this dataset for publication, it should be clearly stated as a limitation with a rationale for this decision. This sample does not seem easily generalizable to modern samples given the time period….’
An important example of older research data with continuing relevance and foundational importance to the field of medicine is the Framingham Heart Study (Dawber, Meadors, & Moore, Reference Dawber, Meadors and Moore1951; Dawber, Moore, & Mann, Reference Dawber, Moore and Mann1957; Feinleib, Reference Feinleib1985). This epidemiologic investigation of a cohort of 4739 individuals, initiated in 1948 by the US Public Health Service to prospectively investigate the epidemiology and risk factors for cardiovascular disease, continued longitudinal data collection over 35 years through 1983. The impact of the Framingham study was captured by the title of a 2018 video by Boston University School of Medicine, ‘70 Years, 15,000 Participants and 3716 Research Papers: Celebrating the Framingham Heart Study’ (Back2BU, 2018).
An example of such a foundational dataset in psychiatry is the landmark Epidemiologic Catchment Area (ECA) study (Regier et al., Reference Regier, Myers, Kramer, Robins, Blazer, Hough and Locke1984; Robins et al., Reference Robins, Helzer, Weissman, Orvaschel, Gruenberg, Burke and Regier1984). This longitudinal study, initiated in 1980 with subsequent waves of follow-up over 24 years, remains the first, and an unduplicated, study of the population prevalence of psychiatric disorders with methodologic rigor. It examined the prevalence of psychiatric disorders in the American general population in a sample of 20 000 adults systematically recruited from residential and institutional sources in urban and rural settings, using structured interviews to assess psychiatric disorders according to established diagnostic criteria. The database from this classic study has been utilized for secondary analysis of a vast array of topics in articles published to this very year on many topics, including mood disorders, posttraumatic stress disorder, panic attacks, alcohol and drug dependence, homelessness, antisocial personality disorder, homosexuality, gastrointestinal disease, diabetes, cancer, asthma, stroke, Parkinson's disease, head injury, headaches, sleep disturbance, stress and cognition, obesity, suicidality, social support, healthcare utilization, and mortality (Crum, Storr, Chan, & Ford, Reference Crum, Storr, Chan and Ford2004; Edwards & Mezuk, Reference Edwards and Mezuk2012; Fleury, Grenier, Bamvita, Perreault, & Caron, Reference Fleury, Grenier, Bamvita, Perreault and Caron2014; Glass & Bucholz, Reference Glass and Bucholz2011; Goodwin & Eaton, Reference Goodwin and Eaton2005; Gross, Gallo, & Eaton, Reference Gross, Gallo and Eaton2010; Henriksen, Mather, Mackenzie, Bienvenu, & Sareen, Reference Henriksen, Mather, Mackenzie, Bienvenu and Sareen2014; Krasnova, Eaton, & Samuels, Reference Krasnova, Eaton and Samuels2019; Kubzansky, Koenen, Jones, & Eaton, Reference Kubzansky, Koenen, Jones and Eaton2009; Larson, Owens, Ford, & Eaton, Reference Larson, Owens, Ford and Eaton2001; Lauterbach, Freeman, & Vogel, Reference Lauterbach, Freeman and Vogel2004; Maulik, Eaton, & Bradshaw, Reference Maulik, Eaton and Bradshaw2009; Mezuk, Lohman, Dumenci, & Lapane, Reference Mezuk, Lohman, Dumenci and Lapane2013; Munro et al., Reference Munro, Wennberg, Bienko, Eaton, Lyketsos and Spira2019; North, Alpers, Thompson, & Spitznagel, Reference North, Alpers, Thompson and Spitznagel1996; North, Dinwiddie, Cottler, & Spitznagel, Reference North, Dinwiddie, Cottler and Spitznagel1991; North, Eyrich, Pollio, & Spitznagel, Reference North, Eyrich, Pollio and Spitznagel2004; North, Kotamarti, & Pollio, Reference North, Kotamarti and Pollio2021; Ramsey, Leoutsakos, Mayer, Eaton, & Lee, Reference Ramsey, Leoutsakos, Mayer, Eaton and Lee2010; Silver, Kramer, Greenwald, & Weissman, Reference Silver, Kramer, Greenwald and Weissman2001; Woolley, Fredman, Goethe, Lincoln, & Heeren, Reference Woolley, Fredman, Goethe, Lincoln and Heeren2008).
The status of older references
If older references are considered inferior, they might be expected to be eclipsed by the growing numbers of newer citations in more recent publications. Logically, older publications have a natural inherent advantage in their likelihood of citation in academic literature, as the earlier an article was published, the more time there has been for it to be cited (Aksnes, Langfeldt, & Wouters, Reference Aksnes, Langfeldt and Wouters2019; Mazhari, Reference Mazhari2013). However, because important randomized clinical trials have an average publication delay of 3 years, they may receive sparse or no citation (Aksnes, Reference Aksnes2005). Consensus is lacking as to the primacy of older and newer references in general scientific citation.
Jemielniak, Masukume, and Wiliamowski (Reference Jemielniak, Masukume and Wiliamowski2019) observed that Wikipedia citations of recently published general medicine articles have increased disproportionately relative to older publications. They referred to this phenomenon as ‘recentism’, describing it as ‘where an article has an inflated or imbalanced focus on recent events. It is writing without an aim toward a long-term historical view’. A review of meta-analytic studies in the Cochrane Database of medical interventions by Patsopoulos and Ioannidis (Reference Patsopoulos and Ioannidis2009) pointed out that citations of older articles may have inherent problems in their obsolescence or irrelevance. They noted, however, that older studies do not necessarily suffer in quality, sample size, or external validity compared to newer studies. They cautioned that such ‘evidence should not be undervalued simply because of its age’ (p. 66) and that studies should instead be judged by careful scrutiny of their research methods.
There is some evidence that citation of older references may be expanding in science and medicine specifically as well as more broadly in other fields (Sluyter, Reference Sluyter2005). In contrast to the preference for more recent citations described by Jemielniak et al. (Reference Jemielniak, Masukume and Wiliamowski2019), Verstak et al. (Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014) observed that citations of older publications in Google Scholar Metrics increased disproportionately relative to newer publications between 1990 and 2013 across the humanities, business, social science, physics, life sciences, medical sciences, computer sciences, engineering, and chemical sciences. References older than 10 years increased from 10% to 56% in the business field, from 29% to 34% in the engineering literature, and from 3% to 33% in medical sciences over this 23-year period. In their article, Verstak et al. (Reference Verstak, Acharya, Suzuki, Henderson, Iakhiaev, Chiung Yu Lin and Shetty2014) suggested that the availability of search engines and on-line references or full-text articles has made older publications more widely known, available, and readily cited.
The literature is thus not clear as to the primacy of citations of older v. newer studies, but older landmark studies such as the Framingham Study and the ECA Study remain important to science today. Such historical studies are not likely to be conducted again and their original findings will retain their importance and value, thus deserving ageless citation. The most cited article in science, describing a fundamental laboratory procedure for measuring proteins, goes back to 1951 (Lowry, Rosebrough, Farr, & Randall, Reference Lowry, Rosebrough, Farr and Randall1951) with >300 000 citations (van Noorden, Maher, & Nuzzo, Reference van Noorden, Maher and Nuzzo2014). The natural citation advantage for older articles was likely an important contributor for this 70-year-old article. The most cited article in psychiatry, introducing a cognitive screening test that came into wide usage, was published in 1975 (Folstein, Folstein, & McHugh, Reference Folstein, Folstein and McHugh1975), with >28 000 citations (Mazhari, Reference Mazhari2013).
Illustrative examples of citation and data age in recent scientific articles
To illustrate the recency of references in published original medical research articles in a selected sample of recent academic medical journals, original research articles were examined (in April 2021, the current month at the time of the inception of this exercise) in the most recent issues in selected psychiatry, medicine, and surgery journals. This exercise is not intended as empirical evidence, but rather as illustrative of novel points this article advances independently. The total numbers of references in 5-year time categories in the last 15 years listed in the bibliographies of all original articles in each issue (or every other article if issues included >20 articles) were tabulated and summarized as raw counts and proportions. The years of the data collection in the articles included were also noted if stated in the article, and the journal impact factors for each journal were recorded if provided in the website. The psychiatry journals sampled for this effort were JAMA Psychiatry (eight articles with a total of 393 references) and the American Journal of Psychiatry (three articles with a total of 138 references). The surgery journals considered were the Journal of Orthopedic Research (11 articles with a total of 459 references) and the Journal of Gastrointestinal Surgery (13 articles with a total of 404 references). The general medicine journals examined were the New England Journal of Medicine (NEJM) (five articles with a total of 125 references) and the Lancet (three articles with a total of 101 references).
Of the articles reviewed, data collection had been completed within 3 years by 12 studies, within 5 years by four studies, between 6 and 9 years by nine studies, and none >9 years; 12 did not state the age of the data (not shown in figure). Additionally, two articles used pre-existing databases of varied or uncertain age, stating that they had analyzed the data within 1 year. Figure 1 lists the publication recencies of the references listed in the selected articles. The recency patterns were consistent across the different journals. In general, although approximately one-half of the references in the articles were from the past 5 years, fewer were 5–10 years old, and even fewer were 10–15 years old; the majority of the references in these articles had publication dates within the last 10 years and the vast majority in the last 15 years.
Fig. 1. Recencies of publication dates for references in journal articles sampled.
Groundhog Day
The illustrative examination undertaken above is exemplary of the bias we and others have observed toward recency in data and references asserted in this article, also reflecting the examples from journal guidelines to authors. Understandably, recent publications and newly breaking research findings, especially in high-impact journals, are valued in the medical literature. However, older data and references to older studies can also be very important for science and its advancement. Failure to utilize existing datasets sufficiently and discounting older research articles both promote loss of knowledge that was established long ago, leading to collection of the same data again for the same purposes. Repeating the same actions again and again is the basis of the plot in the movie Groundhog Day – the essence of the title of this article. In this movie, the protagonist learns from his Groundhog Day experience and makes changes the next time around, but the townsfolk cannot benefit from this experience because they never remember it. Essentially, every repeated day is a new experiment. In much the same way, failing to employ older data and reference older studies leaves readers uninformed. Older data and older references are necessary and of great value to avoid endless repetition of the same data collection and investigation of the same research questions.
Value of older data
There are four main types of older data that may have important value and utility. The first involves three types of longitudinal studies. These may include examination of changes over time within a single sample in prospective longitudinal study designs known as cohort studies. They may also include retrospective study designs starting with current samples of cohorts to identify earlier events or risk factors associated with current outcomes in case-control studies. Also, epidemiologic studies may examine different representative samples of a population over time to identify trends in selected variables of interest. Although these studies may span several decades and thus much of the longitudinal data in them may be quite old, if the completion of the study data collection was recent, articles written from such data are not as likely to be rejected or assigned low priority by journals for publication.
A second type of older studies includes classic or legacy studies that were conducted using rigorous research methods with far-reaching implications whose value has stood the test of time. Two examples of such studies were introduced earlier in this article. The famous population-based, 35-year longitudinal Framingham Study had second- and third-generation cohort studies launched from the original cohort, and many articles have been published from secondary analyses of this dataset. The ECA study, an epidemiologic population study of psychiatric disorders in America, has also had numerous articles published from secondary analysis of its data as well as analysis of the data in relation to externally available data on variables of interest in public sources. Although the data from these studies were not old in articles published shortly after the data collection was completed, in the years that have elapsed, the data have aged considerably. Despite the aging of these datasets, however, the findings remain respected, the original articles are still cited, and robust applications of the data continue to generate newly published articles to this day.
A third type of older studies is original research conducted by investigators who for various reasons did not generate published articles from the research, and thus the findings were not disseminated through publication of the work unless other researchers later managed to analyze the data and write up the results for publication after the data were considered ‘old’. A recent example was a remarkably well-conducted study of families with children who were exposed to dioxin contamination and other disasters in Times Beach and nearby areas in eastern Missouri in the early-to-mid 1980s. The death of the principal investigator interrupted progress in analysis and publication of the study, and only decades later was the project resurrected by the principal investigator's mentee and a mentee of the principal investigator's mentee who completed the data analysis and writing of a manuscript to document the findings (Lee et al., Reference Lee, Pfefferbaum, Portley, Kotamarti, Canan and North2021). Another study of patients hospitalized for alcoholism in 1967 conducted a prospective longitudinal follow-up of their outcomes over two decades, and although some of the main results were published in a handful of articles within a few years of the completion of data collection, colleagues resurrected the project many years later to continue publication of additional main findings of this study (Lewis & North, Reference Lewis and North2021).
A fourth type of older studies includes datasets from research projects that have exhausted publication of the main findings but have utility for further analysis and publication of additional findings outside of the study's main objectives. These datasets may be subjected to later analysis by the principal investigator who was unable to conduct additional analyses for publication in addition to those of the main study objectives until time has gone by, or may be utilized by trainees as sources of data for testing hypotheses that may be peripheral or even unrelated to the study's main purposes and generation of published manuscripts from them. A recently increasing practice is for sharing databases from published studies to allow analysis and use by colleagues and other researchers and the public.
An often-overlooked contribution of older studies is the new knowledge they may provide that has not been reported even by more recent publications, despite the dated nature of the data. Just because the data were collected years ago does not automatically render the findings obsolete. Another advantage of available data from older studies is that analysis and publication of data from them is efficient and cost-effective, as new grant applications and funding are not needed, and the expense and burden of recruiting new research samples and collection of new data are not needed. Investigators who receive major federal grants do not often have enough time to complete all the data collection and analyze and publish all the main findings before they must write and obtain more grants requiring them to collect more data that they cannot publish before the end of the grant. This process may go on for decades, leaving investigators with a great deal of excellent but unpublished data.
As available research funding has become more competitive and younger investigators lack research funding and effort, older datasets can be leveraged by applying new methods and techniques in fresh attempts to solve longstanding medical problems using existing databases by conducting secondary analyses, reducing the tragic waste of the accumulation of unpublished data. This can aid research trainees and young faculty members who do not have their own independent research, but do not have time to complete the labor-intensive design and implementation of an original study with new data collection. With projects involving secondary analysis, trainees and young faculty can proceed directly to data analysis and writing up the results.
Recognizing these research efficiencies, many federal funding agencies and journals request that data collected with federal funds or published in academic journals be made available to qualified researchers. A further use of older data is to provide a source of material for the testing of new hypotheses and exploring phenomena outside of current understanding, such as symptoms outside of existing diagnostic constructs, and for exploring novel constructs. Lacking modern laboratory and imaging diagnostic methods, older research in psychiatry elaborated rich clinical detail. Reliance on signs and symptoms remains the basis of diagnosis more for psychiatry than for other medical disciplines that have increasingly incorporated biological evidence, making the older studies particularly relevant for the field of psychiatry. Older datasets have advantages not realized by more recent studies that have substituted self-report symptom checklists for full diagnostic assessments used in many older studies. New insights from older datasets may also emerge from the application of more recent advances in statistical methods, including multiple regression analysis; factor, cluster, and latent class analysis; path analysis; and neural network modeling, machine learning, and artificial intelligence techniques. It may not be necessary for these older studies to be lost in the fog of time if the data can be revisited with newer statistical methods to extract new findings.
Value of older references
Older references may be underappreciated for the important conceptual, historical, and philosophical contributions to the progression of scientific knowledge and advances they may bring. Older articles provide a historical context, and the references they contain may offer specific details demonstrating additional relevance to current problems. For example, the COVID-19 pandemic has introduced urgent pressures for the development of effective treatments and preventive measures. The mRNA technology used for creating vaccines was established several decades ago. Had scientists not known of this technology published in older literature, the field would not have critical knowledge that it needed to produce the needed vaccines with sufficient timeliness. Historically, literature from more than a century ago regarding the Spanish flu pandemic has direct relevance to the present pandemic. The St. Louis Department of Health was credited as significantly reducing influenza transmission and mortality by mandating nonpharmaceutical interventions such as social distancing, banning large public gatherings, and closing schools, churches, and theaters (McKinsey, McKinsey, & Enriquez, Reference McKinsey, McKinsey and Enriquez2018). Modern policymakers have used this history to refine public health approaches for contemporary pandemic influenza preparedness.
Older literature may provide additional context as it heralds newly emerging paradigm shifts. In psychiatry, the release of DSM-III (American Psychiatric Association, 1980) represented a paradigm shift from theoretically-driven and etiologic approaches to diagnosis in DSM-II to empirically-based and scientifically-validated diagnostic criteria (Kendler, Munoz, & Murphy, Reference Kendler, Munoz and Murphy2010; Mayes & Horwitz, Reference Mayes and Horwitz2005; Surís, Holliday, & North, Reference Surís, Holliday and North2016). The context of this basic shift is remarkably relevant as the field continues to grapple with ongoing pressures to anoint new paradigms such as Research Domain Criteria (RDoC) and the Hierarchical Taxonomy of Psychopathology (HiTOP) (Michelini, Palumbo, DeYoung, Latzman, & Kotov, Reference Michelini, Palumbo, DeYoung, Latzman and Kotov2021). Following strict Kuhnian thinking, some of the resources of older paradigms may be lost. For example, with the paradigm shift that occurred with the introduction of operational definitions of psychiatric disorders in DSM-III, the rich clinical descriptions of published research in prior centuries and information contained in it may be forgotten because researchers may lack training in this art, lack time for it, or no longer value it.
There are good examples of older scientific work that have not held up over time. These include the initially popular dopamine theory of schizophrenia and the use of the dexamethasone suppression test in the diagnosis of depression that were eventually invalidated by subsequent research. An example of old references that continue to have relevance is found in the classic textbook Goodwin and Guze's Psychiatric Diagnosis – especially in the earlier editions – contains many very old references from different classical eras, including the 1960s and 1970s work of Eli Robins and Sam Guze in St. Louis, older psychiatric works of Briquet and Kraepelin, as well as sources from antiquity including Plato and even ancient Egyptian text written on papyrus (North & Yutzy, Reference North and Yutzy2018).
Conclusion
The dichotomization of old v. new data and old v. recent references is overly simplistic. Current data do not automatically lose value through aging in the way that ‘old milk’ becomes ‘spoiled’ after its ‘use by’ date. The current valuation of newer data above older data fails to appreciate the fact that new data eventually become old, and that old data were once new. Old data have an obvious context in that what has changed since the data were collected is known, but new data do not have this context because it is still evolving. Thus, the value of newer and older research is determined by the ‘context’ of when the data were collected and the references were provided. The contextual recommendations of this article are reflected in some of Meehl's stated reasons for lack of scientific progress (Meehl, Reference Meehl1978). Late-breaking laboratory research and current clinical data have value within the context of the current progress of the field, but not to the exclusion of older research findings.
The bias demonstrated in arbitrary rules pertaining to older data and older references can be addressed by instituting comparable treatment of older and newer data and references. Older research has the advantage of a context that is known at the point in time that the research becomes available, whereas the context of newer research can only be speculative and ephemeral. The latest research at any given time lacks the ability to be contextualized within the course of interventions being developed beyond the time frame of the research. For example, the delta variant of COVID-19 provided a different context for research investigation from that of the original (alpha) form of the virus. Even as this discussion is awaiting publication, the context of COVID-19 continues to demonstrate rapid metamorphosis, which may render this example more or less relevant, depending on what happens. This leads to two conclusions: (1) that context needs to be incorporated in all considerations of publication of all research, regardless of the age of the research, and (2) that research needs to be valued based on its contribution rather than its recency. Winston Churchill's time-honored advice that ‘Those who fail to learn from history are condemned to repeat it’ seems to apply here. Thus, rather than asking authors to provide justification of older data and references, a more balanced approach by journals might be to have authors learn from the historical context of the research in their manuscripts.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/S0033291722000216
Financial support
None.
Conflict of interest
None.