Five years ago, Jeffrey Lieberman, the chair of the department of psychiatry at Columbia University, New York, and a past president of the American Psychiatric Association, published a book surveying what he acknowledged was the profession's at times dismal past. According to Lieberman, that litany of failure and mistakes stood in stark contrast to psychiatry's glorious present. Between 1980 and the present, he proclaimed, psychiatry had ‘matured from a psychoanalytic cult of shrinks into a scientific medicine of the brain.’ In the first two decades of the twenty-first century, ‘sweeping progress…has transformed psychiatry.’ ‘Most mental illness,’ he averred, ‘can be diagnosed and treated very effectively…we have entered a period of scientific advances that will produce a stream of innovations more dazzling than any that have come before.’ Psychiatrists now basked in an era of ‘clinical triumphs.’ For many of those with serious mental illness, these advances have produced, not merely the relief of a patient's symptoms, but ‘the utter transformation of a person's life.’ ‘The modern psychiatrist,’ it appears, ‘now possesses the tools to lead any person out of a maze of mental chaos into a place of clarity, care, and recovery’ (Lieberman & Ogas, Reference Lieberman and Ogas2015).

Such is one view of the state of American psychiatry at the dawn of a new millennium. It is obviously not an assessment shared by all psychiatrists, but Lieberman's comments are certainly not those of someone on the margins of the profession. Sadly, I shall suggest here, his conclusion is a fantasy, one sharply at odds with the crisis that increasingly envelopes the profession, and it obscures the dismal realities that confront those unfortunate enough to suffer from the more serious forms of mental disturbance. Given the huge influence that American psychiatry exerts internationally, I suggest it is vital to document the vast gap which exists between that fantasy and reality.

The collapse of public psychiatry that accompanied deinstitutionalization (a development that occurred largely behind the profession's back, and for which it assuredly does not bear primary responsibility) left those with serious mental illness to struggle in a world that deified the marketplace and had a little place and less sympathy for those who lacked the resources or capacity to purchase the services they needed (Lerman, Reference Lerman1982; Scull, Reference Scull1977). The malign neglect that now passes for public policy in this area constitutes a powerful reason for dismal fate that is the lot of those with serious mental illness. But the shortcomings of contemporary psychiatry undeniably must also shoulder a good deal of the blame for a situation where the life expectancy of someone with psychosis is decades shorter than that of the rest of us, and where that abbreviated life too often consists of an alternation between the jail, the flophouse, and the gutter − with all-too-brief psychiatric interventions largely confined to the prescription of antipsychotic medications.

Lieberman was president of the American Psychiatric Association when the latest edition of the profession's diagnostic manual, DSM 5, was finally published in May 2013. Its architects had originally intended to introduce a radically new approach to diagnosing mental illness. At the outset, they had acknowledged the parlous situation they confronted: ‘Despite many proposed candidates, not one laboratory marker has been found to be specific in identifying any of the DSM-defined syndromes. Epidemiologic and clinical studies have shown extremely high rates of comorbidities among the disorders, undermining the hypothesis that the syndromes represent distinct etiologies. Furthermore, epidemiological studies have shown a high degree of short-term diagnostic instability for many disorders.’ But this time around, they promised, things would be different (Kupfer, First, & Regier, Reference Kupfer, First and Regier2002: xviii, 19).

Confident that advances in neuroscience and genetics were on the brink of revealing the biological bases of mental disorders, the architects of the most recent iteration of the DSM aimed to move beyond a diagnostic system based purely on symptoms. DSM's III and IV had placed all the emphasis on reliability, attempting to ensure that psychiatrists reached the same diagnosis when confronted by the same patient. Now, validity would take centre stage. Rather than a series of categories that might or might not identify natural kinds, science and the emerging knowledge of the biological basis of mental illness would drive the process and transform the ways mental illness was categorized. Henceforth, they asserted, the way forward was ‘to recognize the most prominent syndromes that are actually present in nature’ (Regier, Narrow, Kuhl, & Kupfer, Reference Regier, Narrow, Kuhl and Kupfer2009: 646; Regier, Narrow, Rae, & Rubio-Stiper, Reference Regier, Narrow, Rae, Rubio-Stiper, Zorumski and Rubin2005).

Controversy dogged the development of the new manual. Long-time critics of the DSM contended that the new manual would, like its predecessors, expand the definitions of what constituted mental disorder, pathologizing the normal (Boysen & Ebersole, Reference Boysen and Ebersole2014; Horwitz, Reference Horwitz2015; Horwitz & Wakefield, Reference Horwitz and Wakefield2007; Taylor, Reference Taylor2013; Wakefield, Reference Wakefield2012). At the opposite extreme, when word leaked that the DSM Task Force was considering eliminating the diagnosis of Asperger's syndrome and tightening the criteria for a diagnosis of autism, it provoked massive pushback from the parents of children with these diagnoses, for whom these labels were a sine qua non for obtaining access to educational and social services (Carey, Reference Carey2012).

The Task Force beat a hasty retreat. It nominally abolished the previous diagnosis of Asperger's syndrome, but, in reality, it simply relocated it into a new category called autism spectrum disorder. That made sense given that the Task Force concluded that the existing dividing lines were hard to justify and difficult to draw, but patients' families were clearly concerned that insurance companies and government entities might stop paying for services for those at the milder end of the spectrum. To head off further protests, the Task Force issued a ‘clarification’: ‘Individuals with a well-established DSM-IV diagnosis of autistic disorder, Asperger's disorder, or pervasive developmental disorder not otherwise specified should be given the diagnosis of autism spectrum disorder.’ (Diagnostic and Statistical Manual 5: 51). As Allan Horwitz dryly comments, ‘these diagnoses had become such valuable commodities that no one would lose them even if they no longer met [the new] criteria’ (Horwitz, Reference Horwitz2021).

More damaging still, the primary architects of three previous editions of the DSM, Robert Spitzer and Allen Frances, became increasingly vociferous critics of the work of the Task Force. Spitzer objected particularly stridently to what he claimed was the secrecy with which it conducted its business, including the confidentiality agreements members had been forced to sign. I regard these complaints as something of a red herring. Spitzer's protestations notwithstanding, the deliberations of the two task forces he had run had likewise taken place behind closed doors. His claims of openness and dedication to science were disingenuous when his own committee's deliberations were riven with arm-twisting and horse-trading (Davies, Reference Davies2017; Decker, Reference Decker2013). As a good politician, Spitzer did periodically discuss the progress of the manual with different audiences, but the same can be said of work on DSM 5, where the availability of the internet also allowed work groups to post draft documents and to solicit outside comments, which were often extensive. But while the objection about secrecy may have been misplaced or overblown, there can be no doubting its effectiveness as a public relations tactic.

Frances adopted a different line of attack. He echoed the concerns of some lay detractors, and lamented the vast expansion of the psychiatric imperium that would result if some leaked proposals about changes to the manual were eventually implemented. Both men's objections were hammered home in a campaign that addressed both their professional colleagues and the public at large (Frances, Reference Frances2009, Reference Frances2010, Reference Frances2013a, Reference Francesb; Spitzer, Reference Spitzer2009) and provoked an ad hominem attack from the president of the American psychiatric association, Alan Schatzberg, and from the leaders of the Task Force (Schatzberg, Scully, Kupfer, & Regier, Reference Schatzberg, Scully, Kupfer and Regier2009). The controversy became so fierce that one psychiatric observer claimed that it put ‘the Hatfield-McCoy feud to shame’ (Psychiatric Times blog, quoted in Horwitz, Reference Horwitz2021). If they did not manage to derail the publication of DSM-5, Spitzer and Frances did succeed in delaying its appearance and in important ways undermining its legitimacy.

Exacerbating the problems provoked by these public controversies, Kupfer and Regier, who had been chosen to head the task force, exercised little leadership, leaving it to the various work groups to decide how to proceed. This laissez-faire approach stood in stark contrast to the control Spitzer and Frances had exercised over the development of DSM III, IIIR, and IV, and at times threatened to dissolve the work into chaos. As one participant complained, ‘I get aggravated with Kupfer and Regier sometimes, where I want to say, ‘For God sakes, you have to tell us how many dimensions we can have.’ I mean these are things where you really need someone to make the decision about what the parameters are so that you can work. These guys are just way too open and flexible for us’ (Quoted in Horwitz, Reference Horwitz2021). Some opted to loosen the criteria for particular diagnoses (most notably the group working on major depressive disorders), while others took the opposite tack. It was a situation that provoked alarm at the highest levels of the APA. Concerned with the lack of progress on the manual, and the disorganization and the controversy surrounding its work, the Board of Trustees of the APA appointed an oversight committee to oversee the work groups, and then, a year later, ‘a Scientific Review Committee’ that was independent of the DSM revision structure to review all the proposed changes to the manual and make recommendations directly to the APA President and Board of Trustees (Horwitz, Reference Horwitz2021; Kendler, Reference Kendler2013a; Lieberman & Ogas, Reference Lieberman and Ogas2015: 278–80).

These developments help to explain the lengthy gestation of DSM-5. But a more serious problem remained. The ambitious plan to shift from a ‘tick the boxes’ approach to diagnosis to a system rooted in a biological understanding of mental illness quickly foundered because the necessary etiological understanding of the various forms of serious mental disorder simply did not exist. Even the more limited attempt to substitute a dimensional for a categorical approach to defining mental disorders proved chimaerical, with the partial exception of the move to redefine autism spectrum disorder. Clinicians feared that their treatment of patients with milder forms of mental disturbance would no longer receive insurance reimbursement (Horwitz, Reference Horwitz2021: Ch. 6). When the Assembly of the APA convened at the association's annual meeting in May 2012, the numerically dominant clinicians unanimously voted to consign all talk of dimensions to an appendix of the manual (Assembly of the American Psychiatric Association, 2012; Whooley & Horwitz, Reference Whooley, Horwitz, Paris and Philips2013; Zachar & First, Reference Zachar and First2015).

When the decision to create the fifth edition of the DSM had first been mooted in the late 1990s, it came in an atmosphere of great optimism about the practical payoffs of the heightened emphasis on the biology of mental illness (Carlsson, Reference Carlsson1990). President Reagan's election in 1980 had prompted an abrupt shift away from social psychiatry and towards a focus on biology. Social psychiatry, after all, made awkward connections between such things as poverty, inequality, and migration and mental illness, and threatened to disclose the negative effects that undermining public psychiatry had produced. Biological research avoided such politically unwelcome findings. Reagan's successor's proclamation that the 1990s were ‘the decade of the brain’ (Bush, Reference Bush1990) was reflected in a sharp increase in funding for research on genetics and neuroscience. Fueled also by the funds provided by the pharmaceutical industry, the centre of gravity in American psychiatry had shifted rapidly away from psychoanalysis, and the expectation was that the billions of dollars flowing into genetics and neuroscience would soon translate into a greater understanding of the aetiology of the major psychoses, and rapid clinical advances. The advent of polymerase chain reaction (PCR) (which provides the technical capacity to make millions and billions of copies of a very small sample of DNA), and major advances in imaging technology, not to mention the decoding of the human genome and the hope that further advances in psychopharmacology were on the horizon, fed expectations that psychiatry was on the brink of resting its diagnosis and clinical interventions on a more secure scientific foundation.

Those expectations have mostly been disappointed. Interest in the genetics of mental illness had fallen into abeyance in the years after the Second World War, its associations with the Nazis' policy of murdering the mentally ill rendering it toxic, not to mention being out of step with the post-war Freudian hegemony in American psychiatry. But the rising emphasis on the biology of mental illness had spawned a new interest in genetics in the 1970s and 1980s. Twin and family studies had suggested a high degree of heritability for mental illness. The arrival of PCR, licensed in 1989, and the sequencing of the human genome, announced in 2003, seemed to promise that the genetic basis of serious mental disorder – previously something that could only be inferred – would soon be demonstrated directly and unambiguously.

That has not happened. Claims to have discovered the genetic basis of schizophrenia have repeatedly failed the test of replication (For example, St Claire et al. Reference St Clair, Blackwood, Muir, Baillie, Hubbard, Wright and Evans1989; Crowe et al., Reference Crowe, Black, Wesner, Andreasen, Cookman and Roby1991; Detera-Wadleigh et al., Reference Detera-Wadleigh, Goldin, Sherrington, Encio, de Migual, Berretini and Gerson1989; Johnson et al., Reference Johnson, Border, Melroy-Greif, de Leeuw, Ehringer and Keller2017; Sherrington et al., Reference Sherrington, Brynjolfsson, Petersson, Potter, Dudleston, Barraclough and Gurling1988; and for documentation of the methodological problems that lay behind and then undermined such claims, see Sullivan, Reference Sullivan2008). There is increasing scientific consensus that ‘despite our wishing it were so, individual gene variants of large effect appear to have a small to non-existent role in the aetiology of major psychiatric disorders’ (Kendler, Reference Kendler2013b: 1065). Repeatedly, researchers prioritised candidate genes that plausibly looked as though they might explain the genetic roots of schizophrenia and major depression. But none of those proposed linkages has survived close scrutiny (Border et al., Reference Border, Johnson, Evans, Smolen, Berely, Sullivan and Keller2019; Farrell et al., Reference Farrell, Werge, Owen, Ophoff, O'Donovan, Corvin and Sullivan2015; Johnson et al., Reference Johnson, Border, Melroy-Greif, de Leeuw, Ehringer and Keller2017). It was not just that the maximal claim – that schizophrenia, for example, was a Mendelian disorder – was quickly shown to be false, but that even an alternative hypothesis, that ‘a substantial proportion of the [hypothesized] genetic signal could have been concentrated in a few large-effect genes’ was soon rejected (Kendler, Reference Kendler2013a, Reference Kendlerb: 1059).

In place of candidate genes, genome-wide association studies (GWAS) have been employed across a whole spectrum of psychiatric disorders, encompassing tens of thousands of patients. Unfortunately, hopes that these would uncover clear pictures of underlying biology of major mental disorders have quickly faded. Instead, the data show that hundreds of genetic variants may (or may not) contribute to the diagnosis of a particular case. Each of these is individual of small effect, and may be present without giving rise to the disease. They constitute polygenic risk factors predispose for mental illness, but so far they account for only a tiny percentage of the variance (e.g. Cross-Disorder Psychiatric Genetics Consortium, 2013; Major Depressive Working Group, 2013; Schizophrenia Working Group, 2014). In the words of the Cross-Disorder Group of the Psychiatric Genomics Consortium (2013), ‘the effect sizes of the genome-wide significant loci are individually quite small and the variance they [collectively] account for is insufficient for predictive or diagnostic usefulness.’ It was results like these that led Uhler and Rutter (Reference Uhler and Rutter2012) to conclude that ‘molecular genetic studies of psychiatric disorders have done a lot to find very little. In fact, in the era of genome-wide association studies, psychiatric disorders have distinguished themselves from most types of physical illness by the absence of strong genetic associations.’

More recent collaborative work on schizophrenia has expanded the number of genetic loci that may be associated with schizophrenia to as many as 270, and by incorporating this expanded array, one can statistically account for around 7.7% of the observed variance. (Legge et al., Reference Legge, Santoro, Peryasamy, Okewole, Arsalan and Kowalec2021). What is concerning, however, is that each of these potential variations is individually of small effect, and many carriers of these genetic variations fail to exhibit signs of mental disorder. Again, this suggests that these are polygenic risk factors for mental illness, not differences that inevitably or even probably lead to schizophrenia or affective disorders. Genes, it seems, are not fate, and the thousands of alleles that contribute a small additional risk of illness do not operate ‘in a simple deterministic manner’ (Hyman, Reference Hyman2021: 14–15). Developmental and environmental factors must play a crucial role in whether the ‘nudge’ of these alleles manifests itself in mental disorder, which suggests that the over-emphasis on the biology of mental disorder has been a strategic mistake.

On a related front, though advances in basic neuroscience have been considerable, their contributions to the understanding of the aetiology of major mental illness have been slight, and their clinical usefulness nugatory. Steven Hyman, who headed NIMH from 1996 to 2001, reinforced its emphasis on genetics and neuroscience, and in the early stages of revising the DSM, actively encouraged the APA to incorporate findings from these disciplines into the new version of the manual (Greenberg, Reference Greenberg2013: 60). Hyman was soon disillusioned. Having once hoped that ‘we might soon identify causal mutations,’ it transpired that even with a focus on ‘high density’ families, ‘where schizophrenia or bipolar disorders appeared to be transmitted with tragic certainty,’ the results were meagre and contradictory, and were associated with ‘a plethora of other disconcerting observations.’ Chastened, he acknowledged that ‘the genetic, epigenetic, and other environmental risks of psychopathology are etiologically complex and heterogeneous’ (Hyman, Reference Hyman2012). As for incorporating the findings of neuroscience into the fifth edition of the DSM, by 2007 he had concluded that it might only be possible for a small sub-set of mental disorders – in reality, not even that was in reach – and acknowledged that ‘It is probably premature to bring neurobiology into the formal classification of mental disorders’ (Hyman, Reference Hyman2007)

Hyman's successor as director of NIMH, Thomas Insel, was equally convinced, and remains convinced, that biology is the key to understanding and ultimately treating mental disorder. His funding priorities reflected his intellectual convictions, and he had vast sums at his disposal. Shortly after he stepped down as director to − very briefly – work for Google, he summarized what all this funding had produced: ‘I spent 13 years at NIMH really pushing on the neuroscience and genetics of mental disorders, and when I look back on that I realize that while I think I succeeded in getting lots of really cool papers published by cool scientists at fairly large cost − I think $20 billion − I don't think we moved the needle in reducing suicide, reducing hospitalizations, improving recovery for the tens of millions of people who have mental illness’ (Regalado, Reference Regalado2015). Small wonder that there have been complaints that NIMH has focused ‘exclusively on basic research rather than clinical research that could help people currently affected by diseases such as schizophrenia and bipolar disorder’ (Torrey & Yolken, Reference Torrey and Yolken2020; see also Lewis-Fernandez et al., Reference Lewis-Fernandez, Rotheram, Betts, Greenman, Essock, Escober and Iversen2016).

As we enter the third decade of the twenty-first century, the causes of the major forms of mental illness remain an enigma, the product, it seems increasingly obvious, of a complex of biological and social factors. The biological monism that has dominated American psychiatry for 40 years and more has been unable to solve questions of causation. That should not come as a surprise. There is mounting evidence for the importance of social factors in the genesis of major mental disturbance (Howes & Murray, Reference Howes and Murray2014; Morgan, Charalambides, Hutchinson, & Murray, Reference Morgan, Charalambides, Hutchinson and Murray2010; Murray, Reference Murray2017). And there are growing doubts about whether such entities as schizophrenia and the heterogeneous array of disorders the DSM lumps together as major depression will survive as diagnoses (Brockington, Reference Brockington1992; Guloksuz & van Os, Reference Guloksuz and van Os2018). Ironically, genetic research has served to heighten doubts about the separability of these diagnoses. Rather than revealing one set of vulnerabilities for schizophrenia and others for manic depressive disorder or major depressive disorder, the risks it identified seemed to overlap to a considerable extent (Gandal et al., Reference Gandal, Haney, Parishak, Leppa, Ramaswami, Hartl and White2018). Biology, it seems, predisposes some to a heightened liability to mental disorder, but that liability is not disorder-specific, as the researchers into psychiatric genetics used to assume. Hence Robin Murray's conclusion that the idea that schizophrenia is a distinct disorder ‘has been fatally undermined.’ Quite soon, he suggests, ‘the term schizophrenia will be confined to history, like ‘dropsy’’ (Murray, Reference Murray2017: 256).

The attempt to replace the ‘tick the boxes’ approach to diagnosis that the third edition of the DSM had inaugurated in 1980 thus ended in failure. Rather than tying diagnoses to biology and moving to a dimensional rather than a categorical view of mental illness, the architects of DSM 5 were forced once more to base their elephantine system on symptoms. What Freudians had once dismissed as the surface manifestations of underlying pathology continued to be the touchstone by which mental disorders were to be defined and distinguished from one another, and from mental health.

Two major sources of support for this symptom-based approach to diagnosis have been the insurance industry and the pharmaceutical houses. For the former, the DSM approach provided a diagnostic checklist to which they could apply treatment profiles of a limited sort – greatly preferable to the almost interminable treatments offered by psychoanalysis. For the pharmaceutical industry, stable diagnostic categories enabled the testing of new drugs, and the elastic nature of the manual's categories allowed the creation of segmented markets for new pills, chemicals creating illnesses rather than the other way round. Facing cuts to their budget and the threat of more, NIMH found it useful to endorse the scientific-seeming diagnosis system to ward off attacks motivated by its previous support of psychosocial research. Inadvertently, another federal agency also helped to reify the new system. The FDA's decision to treat mental diseases as having the same form as physical diseases led them to require the pharmaceutical houses to test and advertise their products as specific remedies for specific diseases. Clinicians had little choice but to validate the DSM, since they had to use it to get paid, and it has become the defining feature of American psychiatry since 1980. It did at least provide some sense of order when confronting the diverse manifestations of psychiatric disorders, and patients welcomed being able to put a name to their troubles. DSM diagnoses provided reassurance that their doctors had seen other cases like theirs and comforted them with the sense that psychiatrists knew about their symptoms and had treatments for them.

But the DSM model is deeply flawed. It is no longer clear that its diagnoses provide accurate guidance about treatment or prognosis, particularly as symptoms are often unstable over time and co-morbidity is rife (Hyman, Reference Hyman2021: 19–20). In many ways, its division of mental illnesses into hundreds of categories and its emphasis on ‘reducing complex phenomena to distinct, putatively well-bounded classes [has] exacted high costs on psychiatric research and patient care’ (Hyman, Reference Hyman2012: 18).

Even the promise to create professional consensus on diagnosis is under threat. Spitzer had introduced Kappa as a statistical measure of agreement between clinicians that eliminated the concordance that could simply occur by chance. He used it to measure inter-rater agreement in field trials of his new manual to document the heightened reliability the DSM III produced. When the DSM-5 Task Force conducted its own field trials, it used the same statistic. When they reported their findings, Allen Frances immediately cited them as further evidence of the defects of their work. Comparing the two sets of data, he pointed out that Kappa fell from 0.81 in the DSM III trials for schizophrenia to only 0.46 in the new trials, and for major affective disorders the results were worse: kappa here declined from 0.80 to 0.25 (Frances, Reference Frances2012).

At first blush, that seems a devastating critique. Defenders of the DSM-5, however, rightly pointed out that there were crucial differences in the methods used in 1980 and in 2012. Spitzer used two interviewers who had been highly trained in the use of the new manual, and had them examine the same patient at the same time, an approach that by design increased inter-rater agreement. The newer trials operated very differently, using interviewers with much briefer training and having them make their assessments separately (between 4 h and two days apart) (Kraemer, Kupfer, Clarke, Narrow, & Regier, Reference Kraemer, Kupfer, Clarke, Narrow and Regier2012; Regier et al., Reference Regier, Narrow, Clarke, Kraemer, Kuramoto, Kuhl and Kupfer2013). One can argue that such a test of DSM-5 more accurately portrays how it would work in the hands of most clinicians, and it is certainly true that direct comparisons of the DSM III and DSM 5 Kappas should not be made. That said, even those in charge of the 2012 trials acknowledged that the degree of agreement they found for major depression was ‘questionable,’ and for schizophrenia, the results were marginally better but scarcely cause for celebration (Jones, Reference Jones2012; Lieblich et al., Reference Lieblich, Castle, Pantelis, Hopwood, Young and Everall2015).

More fundamentally still, to attempt to diagnose illness using patient symptoms resembles the approach of the eighteenth, not twenty-first-century medicine. Just before DSM 5 finally appeared, Thomas Insel denounced it on precisely these grounds. ‘In the rest of medicine,’ he complained, ‘[the DSM approach] would be equivalent to creating diagnostic systems based on the nature of chest pain or the quality of fever…symptom-based diagnosis, once common to other areas of medicine, has been largely replaced in the past half-century as we have understood that symptoms alone rarely indicate the best course of treatment.’ The DSM's, Insel noted, were often called Psychiatry's Bible even though they were no more than dictionaries, but ‘biology never read that book.’ Most psychiatrists ‘actually believe [that the diseases they diagnose using the DSM] are real. But there's no reality. These are just constructs. There is no reality to schizophrenia or depression’ (Lieberman & Insel, Reference Lieberman and Insel2013).

One sees what motivated such a statement (and it must have been greeted with glee by the Scientologists), but the phrasing was distinctly unfortunate. The labels may need to go (with who knows what consequences for psychiatry's reputation), yet the distress and pathology those traditional labels seek to capture will not disappear with them. Insel's conclusions, however, promptly drew support from Steven Hyman, now directing the Stanley Institute for Psychiatric Research at Harvard. In his words, ‘[DSM 5 is] totally wrong in a way its authors couldn't have imagined…What they produced was an absolute scientific nightmare’ (Bellick & Carey, Reference Bellick and Carey2013). The two men's pronouncements drew massive coverage in both professional and lay media, and feeble attempts to paper over the cracks (Lieberman & Insel, Reference Lieberman and Insel2013) could not disguise the damage they had inflicted on the DSM's credibility.

Physicians are expected to define disease, to understand what causes it, and to provide relief and if possible cures to those suffering from the pathologies they aim to treat. American psychiatry adopted DSM III to shore up its claims to diagnose accurately when its ability to do so had come under sharp attack. Over time, trust in that system has eroded, and its frail foundations are ever more obvious. As to the origins of mental pathology, where once the American professional elite embraced Freudian ideas, for nearly a half-century it has bet on biology, and the wager has mostly turned up snake's eyes.

What, then, of therapeutics? Can psychiatry at least claim progress on that front? Though the Freudian dominance of American psychiatry persisted into the 1970s, its foundations were crumbling. The psychopharmacological revolution that commenced in the early 1950s emerged in the state mental hospital system that the psychoanalytic elite studiously avoided, but ultimately it transformed psychiatric practice and public perceptions of mental illness. Thorazine (Largactil) and the other phenothiazines that soon entered the marketplace were an accidental discovery, as were the antidepressant medications that began to appear on the scene in the late 1950s. Along with the minor tranquillizers such as Miltown and later Valium, they marked a radical shift in the response to a host of mental illnesses, minor and profound. During the closing decades of the twentieth century, American psychiatrists largely ceded the psychotherapies to clinical psychologists and social workers who accepted the lower reimbursement rates offered by American medical insurance companies, and drugs came to form the lynchpin of psychiatric practice (Mojtabai & Olfson, Reference Mojtabai and Olfson2011; Scull, Reference Scull2011a, Reference Scullb, Reference Scull, Fontaine and Pooley2021a).

Thorazine and its competitors did indeed help severely disturbed patients. They decreased the hallucinations and delusions that, along with distortions of perception and peculiar thoughts, are the most conspicuous manifestations of psychosis. Those forms of symptomatic relief were real and dramatic, and their importance should not be minimized. Revolutionary in their way, they continue to underpin psychiatry's faith in psychopharmacology to this day. But, as gradually became apparent, the symptomatic relief was far from universal. Many patients failed to respond to the drugs, and even among those that did, many paid a heavy price in side effects (Crane, Reference Crane1973; Montcrieff, Reference Montcrieff2013).

Many patients given the drugs became pathologically restless and unable to keep still, pacing up and down, exhibiting symptoms of extreme anxiety, often extending to panic and even violence and thoughts of suicide. Akathisia, as this syndrome was dubbed, sometimes persisted for months after the drugs were discontinued. More serious still was tardive dyskinesia, a syndrome that emerged only in time, and sometimes was masked as long as the patient remained on antipsychotics. But in cases of long-term treatment, it afflicted between 20% and 60% of patients to varying degrees, and is often irreversible (Jeste, Caligiuri, & Paulsen, Reference Jeste, Caligiuri and Paulsen1995).

Remarkably, during the first three decades of antipsychotic prescription, these serious problems were ignored or minimized by many of the psychiatric profession. The drugs' ability to control the florid symptomatology of schizophrenia initially outweighed any concern over these side effects. The widely cited NIMH (1964) study of efficacy and safety claimed that unwanted side effects were ‘generally mild or infrequent’ while endorsing the ‘anti-schizophrenic’ properties of antipsychotics. Four years later, Nathan Kline, referred to by some as ‘the father of psychopharmacology’ and once a serious candidate for a Nobel Prize (Healy, Reference Healy2004: 125), suggested that these movement disorders were common in schizophrenia and that tardive dyskinesia was ‘not of great clinical significance’ (Kline, Reference Kline1968: 51). Reporting for the first US task force on tardive dyskinesia, Daniel X Freedman was similarly dismissive. His group reported that prevalence rates were low – 3–6% – and the ‘unavoidable price to be paid for the benefits of prolonged neuroleptic therapy’ (Freedman, Reference Freedman1973: 463).

By the 1980s, however, concern was mounting over these problems (Baldesserini, Reference Baldessarini1980; Gardos and Cole, Reference Gardos and Cole1980). Drug companies sought to alleviate by marketing a new array of drugs, dubbed atypical or second-generation antipsychotics in the 1990s. Excitement about the second generation of antipsychotic drugs has subsided in recent years as it has become apparent that they are largely a marketing ploy, offering few advantages over their predecessors, and in many cases bringing a new array of iatrogenic illnesses in their wake, including massive weight gain, and a heightened risk of diabetes and heart disease (Geddes, Freemantle, Harrison, & Bebbington, Reference Geddes, Freemantle, Harrison and Bebbington2000; Jones et al., Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst and Murray2006; Tyrer & Kendall, Reference Tyrer and Kendall2009; Leucht et al., Reference Leucht, Cipriani, Spineli, Mavridis, Orey, Richter and Davis2013; Young, Taylor, & Lawrie, Reference Young, Taylor and Lawrie2015). Among most atypicals, the incidence of tardive dyskinesia has dropped by 50% or 60%, a welcome development though the problem remains a serious one and ‘many clinicians may have developed a false sense of security when prescribing these medications’ (Lorca, Reference Lorca2002; Cornett, Novitch, Kaye, Kata, & Kaye, Reference Cornett, Novitch, Kaye, Kata and Kaye2017; Kim, Macmaster, & Schwartz, Reference Kim, Macmaster and Schwartz2014; Kinon et al., Reference Kinon2015). But the newer non-extrapyramidal side-effects are extremely serious, and afflict many of those taking the drugs.

First-generation antipsychotics proved largely ineffective in treating the less dramatic, but in many ways more devastating deficits that are characteristic of schizophrenia: the blunted affect, the poverty of speech, the absence of spontaneity and initiative, the failure to connect with others, the anhedonia, or apparent inability to experience or feel pleasure. Cumulatively, these have catastrophic effects on people's quality of life and their ability to function independently. Unfortunately, the new drugs also left these deficits largely untouched. These were uncomfortable realities that psychiatrists preferred to ignore, emphasizing instead, to themselves as well as to outsiders, the gains that the antipsychotics brought in their train. Many patients beg to disagree, judging by those who drop out of clinical trials designed to test the efficacy of antipsychotics – between two thirds and more than four-fifths of those in one large government-sponsored trial (Lieberman et al., Reference Lieberman, Stroup, McEvoy, Swartz, Rosenheck, Perkins and Hsiao2005).

The market for antidepressants is considerably larger than for antipsychotics. Recent statistics indicate that as many as 12% of Americans over the age of 12 use these medications, and between 1999 and 2010, those who had been taking these pills for 2 years or more rose from 3% to 7% of the population. Yet the evidence for the efficacy of these widely prescribed drugs is murky, and the degree of clinical improvement they offer over placebo is surprisingly small (Jakobsen et al., Reference Jakobsen, Karakam, Schou, Hellmuth, Stallknecht, Leth-Møller and Gluud2017; Khan & Brown, Reference Khan and Brown2015; Moncrieff, Wessely, & Hardy, Reference Moncrieff, Wessely and Hardy1998). Those improvements, as with antipsychotics, have to be weighed against the side effects these drugs often produce, which include sexual dysfunction, weight gain, nausea, apathy, and sleep disturbance, problems that may not disappear when antidepressants are discontinued (Bahrick, Reference Bahrick2008; Ferguson, Reference Ferguson2001; Kennedy & Rizvi, Reference Kennedy and Rizvi2009; Rosen, Lane, & Menza, Reference Rosen, Lane and Menza1999). Besides, making the necessary cost−benefit analysis presumes that one can take the published data on antidepressants at face value, and there are substantial reasons to doubt that one should (Healy, Reference Healy1997, Reference Healy2012).

As with antipsychotics, the data on most clinical trials is owned by drug companies, who selectively mine it and are known to suppress findings they find commercially inexpedient (Jureidini, McHenry, & Mansfield, Reference Jureidini, McHenry and Mansfield2008; Spielman & Parry, Reference Spielman and Parry2010; Turner, Matthews, Linardatos, Tell, & Rosenthal, Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). They have found willing allies among academic researchers, who lend their names to ghost-written papers that are composed by in-house writers (DeAngelis & Fontanarosa, Reference DeAngelis and Fontanarosa2008; McHenry & Jureidini, Reference McHenry and Jureidini2008; Sismondo, Reference Sismondo2007; Wislar, Flanagin, Fontanarosa, & DeAngelis, Reference Wislar, Flanagin, Fontanarosa and DeAngelis2011) and who in more than a few instances have been willing to promote off-label uses for profitable pills in return for substantial financial rewards in the form of research funds and honoraria (Harris, Reference Harris2008a, Reference Harrisb; Harris & Carey, Reference Harris and Carey2008; Kaiser, Reference Kaiser2009; Thacker, Reference Thacker2011; Whitaker & Cosgrove, Reference Whitaker and Cosgrove2015). One indication of just how serious drug company misconduct has been is the massive fines that major pharmaceutical houses have been forced to pay when particular instances of misconduct have surfaced, damages that have in some instances cost them billions of dollars. Such lying and misrepresentation have not been confined to the psychiatric arena, of course, (e.g. Berenson, Harris, Meier, and Pollack, Reference Berenson, Harris, Meier and Pollack2004), but have been particularly common and egregious with psychiatric drugs (Langefeld & Steinman, Reference Langefeld and Steinman2009; Staton, Reference Staton2014; Thomas, Reference Thomas2013; Thomas & Schmidt, Reference Thomas and Schmidt2013; Whitaker & Cosgrove, Reference Whitaker and Cosgrove2015). To be sure, company profits have greatly exceeded their fines, but the reputational damage these suits have inflicted may have played some role in discouraging further corporate investment in psychiatric research.

There is still another way in which the United States is failing those with serious forms of mental illness. For a century and more, most unusually for a society that treats medical care as a commodity to be bought and sold in the marketplace, those with severe mental illness were supported at public expense. Most psychiatrists were paid by the state and practised in mental hospitals paid for by taxpayers. For all their flaws – and they were legion – these institutions constituted a recognition of the incapacities that mental illness brings in its train, and the impossible financial and other burdens it places on individuals and families. More than a half-million patients thronged the wards of these state hospitals in the 1950s. Thereafter, at an accelerating pace, mental hospital populations declined, and institutions began to close. By 1980, those that remained led “a lingering existence as demoralized and impoverished facilities” (Mollica, Reference Mollica1981). In the years since, public psychiatry has essentially vanished from the American scene.

Deinstitutionalization was hailed by both politicians and leading psychiatrists as a grand reform, the replacement of now-discredited ‘total institutions’ (Goffman, Reference Goffman1961) by community care. But treatment in the community has turned out to be a shell game with no pea. All too many of the chronically mentally ill have simply been abandoned to their fate. The sidewalk psychotic has become a familiar figure on city streets and in shelters for the homeless. In twenty-first-century America, the Los Angeles county jail, the Cook County jail in Chicago, and the jail on Rikers Island in New York have become the single largest set of institutions ‘treating’ the mentally ill (Lyon, Reference Lyon2019; Torrey, Kennard, Eslinger, Lamb, & Pavle, Reference Torrey, Kennard, Eslinger, Lamb and Pavle2010).

Psychiatrists were not the prime movers behind the emptying of the mental hospitals and the consignment of many mental patients to the gutter and to prison. Nor, as several scholars have shown, was the psycho-pharmaceutical revolution the primary driver of the changes that occurred (Grob, Reference Grob1991; Gronfein, Reference Gronfein1985a; Scull, Reference Scull1976; Segal & Aviram, Reference Segal and Aviram1978). Rather, deinstitutionalization was the product of political choices made in the aftermath of the limited growth of the welfare state, and the rise of a neoliberal consensus that abhorred long-term provision of state aid to dependent populations (Aviram, Syme, & Cohen, Reference Aviram, Syme and Cohen1976; Grob, Reference Grob1991: 239–272; Gronfein, Reference Gronfein1985b; Kirk & Thierren, Reference Kirk and Thierren1975; Lerman, Reference Lerman1982; Rose, Reference Rose1979; Scull, Reference Scull1977, Reference Scull2021b).

What is at first blush surprising, however, is the failure of American psychiatry to object to the closure of the very institutions that had given birth to the profession in the first place, or, once the consequences of privatizing mental health care became apparent, to demand the resources that genuine care in the community requires. Such arguments, however, ignore the dramatic changes that had taken place in American psychiatry in the aftermath of the Second World War. Where once institutional psychiatrists had dominated the profession, that dominance had already disappeared by the end of the 1950s. The overwhelming bulk of the profession had abandoned the institutional practice for the much more lucrative and attractive office-based psychiatry (Grob, Reference Grob1991; Marmor, Reference Marmor1975; Scull, Reference Scull2011a, Reference Scullb). This new professional elite had little interest in the impoverished, clinically challenging patients who thronged the wards of the state hospitals, patients who were a stark reminder of the limits of psychiatry's ability to treat severe forms of mental illness. Faced in later years by national politicians of both political stripes committed to ‘welfare reform,’ an Orwellian euphemism for retrenchment of the social safety net, and state governments that possessed neither the political will nor the resources needed to address the problems at hand, psychiatry's complaisance is understandable, though not praiseworthy (Schram, Reference Schram and Cahill2018; Wacquant, Reference Wacquant2009). The consequences for many of the most gravely mentally ill have been disastrous.