‘There's a tension in the 2018 Exoplanet Science Strategy document’

On p. 10 of the 2018 National Academies Exoplanet Science Strategy document (NASEM, 2018), ‘Expect the unexpected’ is described as a general principle of the exoplanet field. In the Exoplanet Science Strategy public briefing which accompanied the publication of this document, Scott Gaudi – co-chair of the strategy committee – reiterates: ‘One thing I am quite sure of, now having spent more than twenty years in this field of exoplanets … expect the unexpected.’ This is motivated by the fact that there has been an ‘avalanche of unexpected discoveries’ (op. cit., p. 1) in the field in the past 20 years, and this trend is expected to continue. Gaudi also likes to say, ‘Mother Nature is more imaginative than we are’ (Goldsmith, Reference Goldsmith2018, Prologue). The implication is that we must remain open-minded about what we will find out there in the universe.

It is easy to say these things, but not so easy to adopt scientific strategies that do justice to them. Reading through the Exoplanet Science Strategy document, it is hard to see how the statement on p. 10 – ‘expect the unexpected’ – influences anything that comes thereafter. The document is (as one might well expect) saturated with strategy decisions based very firmly on our theoretical expectations. Questions are repeatedly raised concerning such issues as the types of star most likely to host planets with atmospheres, the conditions determining whether planets will be tidally locked to their host star, the spectral signatures most likely to be indicative of life, and so on. Our answers to questions such as these directly influence high-stakes strategy decisions concerning which (types of) telescope to build, which instruments to put on our satellites, which stars to focus our attention on, which planets to rank highest in our priorities for extended study, and so on; and in each and every case, our theoretical expectations provide the driving force. When we later encounter paragraphs explicitly putting weight on our ‘expectations’ (e.g. p. 50), the phrase ‘expect the unexpected’ seems long forgotten.

One obvious response to this is that we should be concerned: we are not ‘expecting the unexpected’ when our exoplanet science strategies are steeped in our expectations, and this might have serious consequences for progress: we are liable to miss things precisely because they fall outside our expectations. Well-known psychology experiments demonstrate the difficulty we sometimes have when it comes to ‘the unexpected’. In the famous ‘gorilla in the room’ experiment (Simons and Chabris, Reference Simons and Chabris1999), roughly 50% of people miss the person dressed in a gorilla costume when their attention is taken up by something else (counting bounces of a basketball). Other experiments in psychology lead to similar findings. Subjects faced with anomalous playing cards, such as a red four of spades, fail to recognize that they are anomalous, putting them into pre-existing conceptual boxes (Bruner and Postman, Reference Bruner and Postman1949). In other words, they see what they expect instead of what is actually presented to them.

If these examples seem too far removed from scientific practice, there are similar lessons throughout the history of science. The ozone hole was originally missed because ozone readings below a certain level were not taken seriously (Oreskes and Conway, Reference Oreskes and Conway2010, p. 119ff.). In work closely related to this paper, Cleland (Reference Cleland2019a; Reference Clelandb, Ch. 8) provides several examples from the biological sciences illustrating this general phenomenon, where ‘assumptions can blind biologists to the significance of puzzling empirical findings’ (Cleland, Reference Cleland2019b, p. 182). Her first example (2019a, p. 723f) concerns how the discovery of ribozymes was delayed by almost a year and a half, owing to the fact that their possibility departed significantly from expectations. Cleland (Reference Cleland2019b, p. 178) usefully clarifies: ‘It is not so much that such beliefs alter our actual perceptions as that our expectations about a phenomenon, coupled with limitations of our instruments, influence how closely we attend to it.’Footnote ¹

Exoplanet science is hardly immune to this phenomenon: Mayor and Queloz (Reference Mayor and Queloz1995) discovered the first exoplanet orbiting a sun-like star (51 Peg b) in 1995, but others could have spotted it (or a similar planet) sooner if only they had been inclined to check their data for a large planet with an unthinkably short orbital period – 51 Peg b orbits its star in 4 days. Gordon Walker's group had used the same method of detection (the radial velocity/wobble method) as Mayor and Queloz for 12 years, between 1980 and 1992, but as Boss (Reference Boss2009, p. 21) notes, ‘Walker did not search his data for such a short-period orbit. The shortest orbital period he considered was 40 days, which must have seemed ridiculously short at the time.’ Walker (Reference Walker, Garwin and Lincoln2003, p. 318) himself distinguishes two separate problems: ‘we had all been looking in the wrong place’, and ‘our programs had been ill designed to sample such periods’. At the same time (late 80s/early 90s), Marcy and Butler were also using this method to search for planets, and fell into the same trap as Walker. A Jupiter-mass planet in a 4-day orbit was ‘absurd’ (Marcy and Butler, Reference Marcy and Butler1998, p. 67). They admit, in hindsight, that they were misled by their expectations, subject to ‘detection biases’ (p. 60). Indeed, when Mayor and Queloz announced the discovery of 51 Peg b in 1995, Marcy and Butler quickly realised that they already had in hand data corresponding to two further planets (70 Vir b and 47 Uma b). They could find the signatures of these planets in their data only when they knew what they were looking for.

Such examples support Hacking's (Reference Hacking1983, p. 179) claim that ‘noticing is theory-loaded’, and perhaps even Kuhn's (Reference Kuhn1970, p. 24) stronger claim that ‘those [phenomena] that will not fit the box are often not seen at all’. If this is right, the status quo is problematic, because we have good grounds for thinking we are missing things and will continue to miss things: there could be significant phenomena we will miss for decades, so long as our theories provide us with expectations (biases) which serve to block them from our view, or which lead us to dismiss them too quickly as ‘uninteresting’.

‘No there isn't – nothing to worry about’

But perhaps these concerns are not really warranted. Scientific communities are diverse, with different types of individual, different teams, different values, different talents and so on. The ‘gorilla in the room’ is not missed by everybody, because some individuals suffer from so-called ‘inattentional blindness’ less than others. In science, we actually only need one scientist to see something unexpected, and, if it really is significant, very soon the whole community will know about it. In other words, even if 99.99% of the scientific community are somehow ‘blind’ to something, it still does not get missed by that community. In addition, individual scientific teams that originally miss something often correct their mistake after a relatively short delay (in the grand scheme of things). Cleland's own example of the discovery of ribozymes involved a ‘delay’ of less than 18 months, for example – and it is no delay at all if one thinks that scientific teams should explore options in line with expectations before they explore unorthodox options.Footnote ²

Consider also the specific examples that motivated Gaudi to say ‘expect the unexpected’. Arguably, these unexpected phenomena were never going to be missed, at least not for long. Drawing on NASEM (2018) and the corresponding public briefing, three examples motivating Gaudi's remark are:

1. (i) the abundance of a type of planet (super-Earths/mini-Neptunes) in our Galaxy that does not feature in our own Solar System (e.g. COROT-7b);

2. (ii) planets that are hotter than some stars (e.g. KELT-9b);

3. (iii) planets found in the ‘wrong’ place, given our theoretical expectations of where they would have originally formed (e.g. 51 Peg b).

These were all highly unexpected, and yet as soon as we had the technology such phenomena were (quite obviously) always going to reveal themselves to us. Thus the mantra of the inattentional blindness literature, ‘we often miss what we don't expect to see’, just does not apply.

One reason for this is that we design our missions and instruments in a way that allows for ‘the unexpected’. The Hubble space telescope, for example, has observed many thousands of unexpected phenomena, including phenomena (including exoplanet phenomena) that formed absolutely no part of the theoretical landscape when the telescope was designed. The Kepler mission was similarly open-minded to what it might find; it focused indiscriminately on a rather random portion of the night sky, continuously observing 150,000 stars, looking for exoplanet transits. It was not somehow restricted to only notice phenomena that are expected. Other missions (e.g. WFIRST) will similarly be open to what they will find. Indeed, one might argue that the NASEM (2018) strategy document does accommodate ‘expecting the unexpected’, for example when it stresses the importance of mission flexibility in both design and operation (p. 83).

What of the ‘theory ladenness of observation’ familiar to philosophers of science (e.g. Psillos Reference Psillos1999, p. 78)? A case can be made that it has often been exaggerated. It cannot be said that the theories we have affect everything we see; if that were true, it would be hard to make sense of the fact that scientists just do see phenomena that seriously challenge their preferred theories (cf. Cleland, Reference Cleland2019b, p. 177). Indeed, many scientists of the past have been primarily talented as investigators of phenomena, adopting a method of investigation that is theory-neutral to a very significant degree. Hacking (Reference Hacking1983, p. 157ff.) makes this point using David Brewster as an example. He writes, ‘Brewster firmly held the ‘wrong’ theory while creating the experimental phenomena that we can understand only with the ‘right’ theory, the very theory that he vociferously rejected’, and emphasizes that Brewster was merely ‘trying to find out how light behaves’. (ibid.). Brewster was very successful, discovering new facts about light including birefringence in bodies under stress. His observations of light behaviour were not affected by the theory of light that he explicitly preferred (the Newtonian, corpuscular theory).

This all suggests that we will continue to uncover surprises in the future, just as we have in the past, without needing to change our scientific methods in any way.

Two ways ‘the unexpected’ might get missed

The fact that we have uncovered unexpected phenomena in the past (the ozone hole; ribozymes; Gaudi's three examples) does not go far to supporting the conclusion that we will always uncover unexpected phenomena. The fact that telescope concepts (e.g. Hubble and Kepler) are somewhat ‘open-minded’ vis-à-vis what they will find, and have uncovered some unexpected phenomena, does not go far to showing that we are not missing some/many other important phenomena. Two ways in which the space science community might miss something stand out.

First, consider the competition for time when it comes to our best satellites. The NASEM strategy document states the issue clearly in a discussion of JWST, the James Webb Space Telescope:

In addition, only ~20% of JWST time will be devoted to exoplanet science (ibid.), and the teams that built JWST's instruments as well as select members of the Science Working Group get a certain amount of guaranteed time. The competition for the time left over is incredibly intense.

One thing is trivial: nobody doubts that many things will get missed for a long time, simply because there are too many things to look at. In the past 10 years more than 3650 exoplanets have been confirmed – more than one a day. With missions such as TESS, we expect this trend to continue (Huang et al., Reference Huang, Shporer, Dragomir, Fausnaugh, Levine, Morgan, Nguyen, Ricker, Wall, Woods and Vanderspek2018). Perhaps the best we can do is select the most exciting targets (e.g. planets in the habitable zone of the TRAPPIST-1 system; Gillon et al., Reference Gillon, Triaud, Demory, Jehin, Agol, Deck, Lederer, Wit, Burdanov, Ingalls, Bolmont, Leconte, SN, Selsis, Turbet, Barkaoui, Burgasser, MR, SJ, Chaushev, CM, Delrez, CS, DL, EJ, VV, Almleaky, Benkhaldoun, Magain and Queloz2017), and move on to other interesting targets as and when time allows, always keeping an eye on the newly confirmed planets coming out of the raw data.

But how do we decide which targets are the highest priorities? The NASEM (2018) strategy document notes that we must try to ‘rank exoplanet targets’ (p. 51). How will such a ranking be achieved? Quite clearly, our theoretical expectations will play a central role in determining any ranking. For example, if our highest priority is to find extraterrestrial life, and we characterize the habitable zone in a generous way, certain planets will be ranked higher than they would with a more conservative habitable zone (cf. Joshi and Haberle, Reference Joshi and Haberle2012). We should absolutely expect our theoretical expectations to change in all sorts of ways over the next 20 years (say). When it comes to the ‘avalanche of unexpected discoveries’ in the past 20 years, we should not be thinking only about concrete empirical surprises (e.g. the abundance of super-Earths/mini-Neptunes in the Galaxy). We should also be thinking about very significant theoretical surprises.

Theoretical surprises in the field of exoplanet science have been numerous in the past 20 years, some spurred by empirical discoveries and others a result of relatively ‘pure’ theoretical research. An obvious theoretical reassessment was demanded by the fact that planets have been found in the ‘wrong’ place; we now think that planet migration is common (not rare), and models of such migration are a major part of theory. Other significant theoretical turnarounds include: (i) a reassessment of the habitability of terrestrial planets orbiting in the habitable zone of dwarf stars (e.g. Luger and Barnes, Reference Luger and Barnes2015), and (ii) a reassessment of various suggested ‘biosignatures’ (e.g. Schwieterman et al., Reference Schwieterman, Meadows, Domagal-Goldman, Deming, Arney, Luger, Harman, Misra and Barnes2016).Footnote ³ Many other examples could be mentioned; e.g. NASEM (2018, p. 48f.): ‘It was previously thought that cores of 10 M_E were necessary to initiate runaway gas accretion…’. Nobody doubts that such theoretical turnarounds will continue, and this in turn will affect any priority ranking of exoplanet targets. Thus we expect that at least some planets ranked low down any proposed list right now will turn out to be highly significant. We could see this significance quite soon, were we to point our best telescopes at them. But as things are, we will not point our best telescopes at them any time soon. In fact, some such planets will probably never get looked at: newly discovered planets will keep being slotted into the top 20% of our ranking, meaning that we will never get around to analysing planets in the bottom 80% of our ranking with our best instruments. Not unless theory changes, and tells us to bump some planets up the list.

Can we shrug our shoulders and say that, if exciting planets are far down the list such that they will never get looked at, we just need to trust in theory, and wait for the theoreticians to tell us why those planets really should be high-priority targets? But consider, for example, the seasonal variation of methane and oxygen levels on Mars (Moores et al., Reference Moores, Atreya, Gough, Martinez, Meslin, CL, SK, PR, CE and CR2019): theoreticians were (very probably) never going to predict this phenomenon from pure theory, leading us to look for it. We had to discover it empirically, and only then did theoreticians get busy developing relevant theory.Footnote ⁴ Similarly, we should expect that at least some planets down the list will be highly significant for theoretical reasons not yet in our possession, and which will not be in our possession unless we get empirical data to spur on the relevant theory.

This is one way in which we should expect to miss the unexpected: planets currently not expected to be especially interesting targets, given current theory, will get missed because we will never look at them. We would look at them if we had different theoretical expectations, but we may never develop the relevant theory unless we look at them (a chicken-and-egg situation). The relationship between the discovery of 51 Peg b and the development of planet migration theory is just such a case: the theoretical development is a consequence of the discovery, and the discovery would have come much sooner had we already had the theory. In another such case, the discovery might never come without the relevant theory in hand.

There is also a second way in which we might miss the unexpected, having to do with which projects we choose to fund in the first place. What type of project is likely to get funded? An important consideration has to do with the expected payoff for the investment of resources, especially in a worst-case-scenario. For example, missions such as TESS get funded because, even if our expectations are too optimistic, we still expect to get a huge amount of valuable data, including the discovery of a great many new exoplanets. The mission is a ‘safe bet’. Suppose this is in competition with another mission, primarily designed to explore a relatively unknown parameter space. The more unknown the parameter space, the riskier the mission is perceived to be: it is likely to be thought of (disparagingly) as a high-risk ‘fishing expedition’. An investor (funding body) has a simple choice between a near-zero-risk investment and a high-risk investment.Footnote ⁵

At any one time, certain minority groups in the community are researching non-mainstream – and thus intellectually risky – theoretical ideas. For example, some astrobiologists are researching non-mainstream biosignatures, e.g. signatures based on life that is not carbon-based. If we really do ‘expect the unexpected’, we should want to support such research, since we should expect that it really does have a non-negligible chance of paying off (and in a big way). But such research is perceived as ‘high risk, high reward’, and the term ‘high risk’ (much more than ‘high reward’) is going to be especially salient to potential funders. This is essentially because ‘bad is stronger than good’ (Baumeister et al., Reference Baumeister, Bratslavsky, Finkenauer and Vohs2001). Indeed, as recently as 2009 NASA rejected missions based solely on a risk assessment (Schingler et al., Reference Schingler, Marshall, MacDonald, Lupisella and Lewis2009) – high-risk projects were rejected. An aversion to ‘risky’ projects is also in evidence in the public briefingFootnote ⁶ accompanying the publication of NASEM (2018); Dave Charbonneau (one of the two co-chairs of the committee) says:

Obviously, when he says that this minority sub-community ‘is looking for alternate biosignatures’ he means with pen, paper and computer simulations! They will not be looking for relevant biosignatures with high-spec instrumentation any time soon.

Thus, there are two different ways in which ‘the unexpected’ might be missed for a very long time, or even indefinitely: (i) we have many targets we expect to be interesting, and these take up all the mission time (e.g. for JWST), meaning that the huge number of low-priority targets never get looked at, and (ii) we have many projects we expect to be fruitful, and these take up all the available funding, meaning that ‘high-risk’ projects based on non-mainstream theory never get funded.

How to ‘expect the unexpected’ – adopt a balanced portfolio?

If we take these threats seriously, what can be done? One thought is that we simply have to base our resource allocation strategy on the best theoretical ideas we have right now, and we just have to accept that sometimes we will miss something unexpected, potentially for a long time. But we need not be so defeatist. We might instead be inspired by a popular financial investment strategy, where we adopt a ‘balanced portfolio’, striking a balance between higher risk (but also higher reward) investments, and lower risk (but lower reward) investments. Such an emphasis on balance is well known in the literature on the division of cognitive labour in scientific communities. For example, Weisberg and Muldoon (Reference Weisberg and Muldoon2009) provide an analysis indicating that, ‘a mixed strategy where some scientists are very conservative and others quite risk taking leads to the maximum amount of epistemic progress in the scientific community’ (p. 227). Unfortunately, Weisberg and Muldoon's model applies poorly in a context where the risk-takers' research is inhibited by journal editors, funding bodies and/or other community dynamics.Footnote ⁷ That is, we need to ensure that risk-takers are allowed to be risk-takers in the contemporary scientific context, where scientific communities are integrated to a greater extent than ever before, and scientists cannot be accurately modelled as isolated individuals (contra Strevens, Reference Strevens2003).Footnote ⁸

Thus, we want not merely a balance of low/medium/high-risk research appetites in the scientific community, but in addition a community dynamic where resources are allocated to reflect such a balance of appetites. Evidence that there is a mismatch between research appetites and resource allocation (see Fig. 1) comes in two forms:

1. (i) research into the relevant community dynamics (Haufe, Reference Haufe2013; Stanford, Reference Stanford2019);

2. (ii) concrete examples of unfortunately inhibited ‘high-risk’ research from the history of science, including:

   • • continental drift theory 1915–1960;

   • • the bacterial theory of peptic ulcer disease 1954–1985 – see e.g. Zollman (Reference Zollman2010).

Fig. 1. The appetite-resources mismatch: at the extremes there is a mismatch between the percentage of the community with an appetite to pursue a given line of research, and the percentage of resources made available for that line of research.

To give just one line of argument (cf. Stanford, Reference Stanford2019, Section 1.3): if only 5% of the community think a hypothesis is worth pursuing, then if one such researcher applies for funding, it is likely that the peer-reviewers and members of the funding committee will consider it a hypothesis that is not worth pursuing. If the researcher is lucky and one or two members of the funding committee consider the hypothesis worth pursuing, it is still very unlikely that the project will be funded; it will inevitably be competing with projects that every member of the funding committee considers eminently worthy of funding. Stanford is not thinking about the space science context, but consider the following from a NASA white paper (Schingler et al., Reference Schingler, Marshall, MacDonald, Lupisella and Lewis2009):

If this is right, and non-mainstream research judged to be high risk by 95% (say) of the relevant scientific community is significantly inhibited, then the community is raising a barrier between its practices and ‘the unexpected’, especially when we think about particularly big surprises that are most likely out there waiting to be discovered (Fig. 2).

Fig. 2. A simple model of the frequency and magnitude of surprises in science.

This all adds to the case that we cannot be laid-back about resource allocation; the status quo is not optimum. And why would it be? As Kitcher (Reference Kitcher1990, p. 22) remarks: ‘[I]t would be highly surprising if the existing social structures of science … were to be vindicated by an optimality analysis.’ Scientists – as human beings more generally – are naturally risk averse. Kitcher (p. 7) cites Wegener's theory of continental drift and notes, ‘we may be grateful to the stubborn minority who continue to advocate problematic ideas’.Footnote ¹⁰ The greater the integration of science, the more power the conservatives have to inhibit the research appetite of the small minority of risk-takers. Greater integration also increases the chance of a problematic ‘bandwagon effect’; cf. Zollman (Reference Zollman2010) considering the case of peptic ulcer disease: ‘Palmer's study was too influential. […] It was the widespread acceptance of Palmer's result which led to the premature abandonment of the diversity in scientific effort present a few years earlier.’ (p. 21). And the international scientific community is today more integrated than at any time in its history.

How would the ‘balanced portfolio’ proposal apply to a major new telescope such as JWST? We might adopt a policy to dedicate roughly 5% of its exoplanet time to high-risk targets, and roughly 20% of its time to targets judged to be medium-risk, still leaving 75% of its time for mainstream, low-risk targets. Of course, there would be some difficult decisions to make, such as which projects to discount completely on the grounds that they are too risky for serious consideration. But these are hardly insuperable challenges.Footnote ¹¹ Once the truly unpalatable proposals are eliminated, we might choose between the other available ‘high-risk’ proposals at random. Or, if we really want to remove all theoretical expectations, we could forget about scientists' proposals, and devote one day in a hundred to a ‘spin the bottle day’ for JWST where a target planet which would otherwise never get looked at is chosen more or less at random.Footnote ¹²

It is one thing to apply this ‘balanced portfolio’ thinking to observation targets; quite another to apply it to funding decisions generally. Space missions, and instruments, are often hugely expensive, so giving 5% of resources to a non-mainstream research program might not go very far. There is also the issue of how to judge between all the different non-mainstream options. Should 5% of resources be spread out between many different high-risk research programs? Or are we to choose the least risky high-risk project for the full 5% of funding? Or the most risky? There are bound to be some high-risk projects (e.g. life on Titan) where a dedicated satellite/mission or dedicated instrumentation would be needed – it would not be a simple case of pointing our current telescope in a different direction. One option is for the relevant scientific sub-community to save up money for a great many years, until such a time as the required funds are available. Alternatively, the sub-community in question could choose to devote these funds to theoreticians, with the thought that, once the theory is sufficiently robust, the corresponding proposals would move from ‘high risk’ to ‘medium risk’, thus opening up new sources of funding.

Factoring in ‘low-risk, high-reward’ exploratory science

A ‘spin the bottle day’ for JWST may strike the reader as highly controversial. Many different scientists around the world have spent years researching precisely what they might do with JWST if they could just get their hands on it for a few days. We can imagine how these scientists would react to a ‘spin the bottle day’ for JWST (even just one day in a hundred) where random high-risk targets are selected instead of their own carefully chosen ones. A ‘balanced portfolio’ strategy takes time away from well-thought-through targets, preferring targets that have not been thought through at all. Scientists have sometimes said that they like the idea of the community adopting a balanced portfolio; what could be wrong with ‘balance’? But whilst it is easy to agree to this in the abstract, it is harder to swallow when one makes the suggestion concrete, as when we propose spending precious JWST time on high-risk targets.Footnote ¹³ It might be urged that some such high-risk targets will deliver unexpectedly high rewards. But what of the thought that we will already get plenty of high rewards from the low-risk targets?

Here we meet a disanalogy with financial investment ‘balanced portfolio’ strategies. Financial investors almost never have an opportunity for low-risk, high-reward investments; their sole motivation for pursuing high-risk options lie with the fact that this is (usually) the only way to get high rewards. If they could get comparable high rewards with low-risk options, they would never take those high risks. But in the contemporary space science context, we can see before us low-risk, high-reward projects.

Take, for example, the LUVOIR telescope concept (cf. NASEM, 2018, pp. 80 and 135). This is very low risk, since we know for sure (ignoring ‘technical risk’) that it would deliver all sorts of step-change scientific returns. Most significant of all, it would be able to directly image exoplanets, perhaps even imaging a ‘pale blue dot’ exoplanet similar to Earth that has a decent chance of being inhabited with at least primitive life. Taking everything into account, LUVOIR (though expensive) is a clear ‘very low risk, very high reward’ project. A direct image of an Earth-like exoplanet would immediately become one of the most influential images of all time, making the front page of every media outlet in the world.Footnote ¹⁴ There is no higher reward (within grasp) in the field of exoplanets, and we do not need anything like a ‘high-risk’ project to go after this extraordinary return. Similarly with many other low-risk, high-reward mission concepts being placed in front of NASA and other space agencies right now.

This might be called the ‘low hanging fruit argument’. If the apples hanging low are just as juicy as those higher up, we have no reason to put any effort into reaching those apples that are difficult to reach. In the context of contemporary space science, this would mean leaving high-risk projects behind, at least for now – not even devoting 5% of resources to them. This seems backward when we think about cases where inhibiting the non-mainstream/high-risk projects turned out to be a big mistake. However, not all scientific contexts are alike. Those historical cases where the minority view was unfortunately inhibited are cases where scientists were adjudicating between competing theories. In the current space science context, scientists are not adjudicating between opposing theories in anything like the same way. Instead, exoplanet science is sensibly characterized as exploratory science – the exploration of a huge range of phenomena that has recently become accessible. We know enough about the relevant parameter space to know that major scientific rewards are out there, in abundance, waiting to be claimed.

We can make this more concrete by considering observation target options for JWST. Three of the debated options are:

1. i. Gas giant planets and mini-Neptunes. JWST would be good at analysing these planets, and could look at a relatively large number of them, such that a good cache of results would be guaranteed. In the process, we would expect to learn a lot about the formation of planetary systems. Specific targets could be GJ 436b and GJ 1214b (NASEM, 2018, p. 86; Kempton et al., Reference Kempton, Bean, Louie, Deming, Koll, Mansfield, López-Morales, MR, RT, Ballard, Barclay, JK, NE, TG, Berta-Thompson, Birkby, LA, Charbonneau, NB, Crossfield, Val-Borro, Doyon, Dragomir, Gaidos, Heng, Hu, Kane, Kreidberg, Mallonn, CV, Narita, V, EV, Rauscher, Seager, EL, DK, Sozzetti, KG, JA and von Essen2018).

2. ii. Rocky planets, such as those within the TRAPPIST-1 system. This would be much more time-intensive; far fewer planets could be analysed in the same amount of time. However, many scientists are excited to learn more about the TRAPPIST-1 system (Lustig-Yaeger et al., Reference Lustig-Yaeger, Meadows and Lincowski2019).

3. iii. K2-22b, a disintegrating rocky planet orbiting its parent star once every 9h. JWST would be able to analyse the disintegration (via transmission spectroscopy of the planet's dust tail), thereby providing information on the interior of the planet. This may be a rare opportunity to ‘see’ the interior of a rocky planet (Bodman et al., Reference Bodman, Wright, Desch and Lisse2018).

These options are all low risk, in the sense that it is vanishingly unlikely that they would deliver no results of interest. They all offer potentially quite high rewards: the information we would expect to acquire in each case would be extremely valuable. If we are pushed to decide between them, one reasonable answer seems to be that there just is no right answer here. Instead we are faced with a mere difference of tastes, preferences, or values. In which case the decision might be made by a simple ‘show of hands’ (or a proxy).

However, option (ii) does stand out, in that it offers the possibility of a very high reward in a way (i) and (iii) do not. If we turn to the planets in the TRAPPIST-1 system, there is a non-negligible chance that one or more of these planets is inhabited. The probability of this is judged by many to be quite low, given the aggressive activity of the TRAPPIST-1 host star (past and present), and the fact that these planets are probably tidally locked to the star. However, relevant theory is tentative, and the majority of scientists agree that we can put only limited trust in any given modelFootnote ¹⁵; we really just need to go and look at the planets and find out. Certainly, we know that the TRAPPIST-1 system contains small rocky planets in the ‘classical’ habitable zone, suitable for surface liquid water.

Thus, option (ii) is more clearly ‘low risk, high reward’ than options (i) and (iii), chiefly because when we think about the ‘high rewards’ that are reasonable to hope for from options (i)–(iii), option (ii) stands out. Whilst exoplanet scientists do indeed have different values and preferences, one thing unites them: the search for extraterrestrial life. Scientists are united when it comes to the high value of discovering (even primitive) life.

The promise of this particular high reward from option (ii) could disappear very quickly. JWST may discover relatively quickly that none of the TRAPPIST-1 planets have substantial atmospheres (never mind interesting atmospheres), and are all no more habitable than modern-day Mars. Such a discovery would immediately reduce the value of pursuing an analysis of these planets. Thus a balanced approach to options (i)–(iii) might go as follows:

Note how such a strategy goes firmly against the ‘spin the bottle’ idea mentioned earlier. With these options before us, it just is not reasonable to spend any JWST time (even 5%) on targets we currently have no good reason to think are interesting.

Developments at NASA since 2009, and a proposal

A concern remains: according to a NASA white paper (Schingler et al., Reference Schingler, Marshall, MacDonald, Lupisella and Lewis2009), as recently as 2009 proposals were rejected if they were judged to be high risk, regardless of any other considerations. The foregoing argument suggests that this was a good strategy: high-risk proposals are not worth looking at, since we can get high rewards from low (and of course medium) risk proposals. But surely throwing all high-risk proposals in the bin was too crude?

Developments at NASA since 2009 indicate that this has indeed been judged as too crude. First, a policy was put in place to simply flag high-risk high-impact proposals for further scrutiny. Then, in a 2017 review of NASA's Planetary Science Division, the National Academies of Sciences, Engineering, and Medicine explicitly put forth the following recommendation:

This led to a survey of peer reviewers, for 1577 proposals submitted to the 2017 Research Opportunities in Space and Earth Science (ROSES) funding competition. Reviewers were provided with definitions of ‘high impact’ and ‘high risk’, and asked to flag those proposals coming out as both high risk and high impact. Ten per cent of proposals were flagged, and out of these 35% were selected for funding, compared with an overall selection rate of 24%. This seemed to show that there is in fact no problem with the funding of high-risk high-impact projects within NASA; Paul Hertz – Director of the Astrophysics Division at the Science Mission Directorate – stated in his 22 October 2018, NASA Astrophysics Update that this ‘disproves the myth’ that there is a problem with the funding of such projects.Footnote ¹⁶

Given all the reasons for supposing that there would be natural reticence to both submit and fund such projects, together with the point made above that many projects can be high impact without needing to be high risk, this seems like an odd result.Footnote ¹⁷ Has something gone wrong? The definition of ‘risk’ used for the 2017 survey (and still in place in January 2020) was:

Reviewers had to select: (i) great extent, (ii) some extent, or (iii) little or none, with ‘great extent’ corresponding to ‘high risk’. But this definition is too easy to meet.Footnote ¹⁸ There are many projects which test ‘significant hypotheses’, and for which there is ‘scant precedent’, but where, for theoretical reasons, we fully expect the project to deliver exciting results. And a project we fully expect to deliver exciting results is not a high-risk project. Consider for instance the WFIRST microlensing survey and the Rosetta mission to land a module (Philae) on a comet. A reviewer might think they both fit the definition, since they both ‘test novel or significant hypotheses’, and for both there is ‘scant precedent’. However, although there is indeed ‘scant precedent’ for these projects, they are both low risk, since on theoretical grounds we fully expect(ed) significant results (supposing nothing goes wrong technically, and of course the Philae project did not go to plan; it was arguably risky in the technical sense, but that should be irrelevant).

Thus the worry is that (far) less than 10% of the submitted proposals were truly high risk, because reviewers judged them to meet the given definition for ‘high risk’ whereas, in fact, there was little (intellectual) risk involved. This hypothesis could be tested, by looking at the 54 relevant projects (34% of 10% of 1577 submitted proposals), and seeing how many of them ultimately fail(ed) to deliver significant scientific returns. If they are all truly high risk, then a good number of them should fail. Alternatively, one could test the definition by handing descriptions of 20 past projects (say) to a number of space scientists, and asking them the following questions: (i) Did this project test novel or significant hypotheses? (ii) Was there ‘scant precedent’ for this project? If they answer ‘yes, to a great extent’ for projects we are sure should not come out as ‘high risk’ in the intellectual sense, then that would reveal a problem with the definition.

The point here is not to argue that there really is a problem with funding high-risk high-impact projects within space science. Rather, the point is that we should not fool ourselves that such projects are being regularly submitted and funded if they are not. If they are not being regularly submitted and funded, there might be good reasons for that. For example, as noted in the previous section, given all the exciting fully mainstream targets for JWST, we would not expect scientists to request funding for a high-risk, theoretically fringe target and, if such a funding proposal were made, it would be very reasonable to reject it. To put it even more starkly, if 0% of target proposals for JWST are high risk, or if 0% of high-risk proposals get funded, that need not indicate a problematic reticence to fund high-risk high-impact projects. Rather, it might be the only reasonable thing to do in a context where there are several low-risk high-impact options.

Should we go back, then, to the situation within NASA prior to 2009, where all high-risk proposals were immediately rejected? Surely not. It has been here argued that scientific communities sometimes do struggle to support high-risk projects, even to the extent that projects crucial for scientific progress and the pursuit of truth (e.g. continental drift theory) were seriously inhibited for many years. What we really want is a system that allows high-risk projects to be fairly judged alongside low- and medium-risk projects.

Schingler et al. (Reference Schingler, Marshall, MacDonald, Lupisella and Lewis2009) briefly proposes ‘a mission selection system’ based on the following Mission Evaluation Metric:

$$V{\rm } = {\rm }S{\rm } \times {\rm }P_{\rm S}/{\$},$$

where S = science value, P _S = Probability of Success of the mission and $ is the cost. This is a good initial model in the sense that it allows for various balances between risk, reward and cost to come out with the same score. It also allows for high-risk high-reward proposals to score higher than low-risk high-reward proposals, if the cost of the former is small compared with the latter. Thus, just because there are low-risk high-reward options, that does not mean that all high-risk proposals should be immediately dismissed.

This proposal is certainly just a starting point. Suppose that various projects put forward for funding are scored according to the given Mission Evaluation Metric; we might not want to simply award funding to the highest scorers, since that is consistent with a situation where one incredibly expensive project takes all the funding, and there could reasonably be objections to that. In addition, the highest scoring proposals could all be ‘high risk’, and this would be worrying, since one would then expect a good number of the projects to fail, and that would be embarrassing from a public and political point of view. It is important that relatively few missions are seen to ‘fail’, since there is a great tendency to perceive such projects as a waste of money:

If many high-scoring proposals were high risk, it would seem reasonable to adopt a balanced portfolio of low/medium/high-risk projects, even if this would mean funding some low-risk projects that scored lower than some high-risk projects. However, this is merely hypothetical since, as already noted, in the contemporary space science context we might well find that no high-risk proposals are competitive, even when they offer high rewards.

A note on the definition of ‘high risk’: In the Schingler et al. (Reference Schingler, Marshall, MacDonald, Lupisella and Lewis2009) model, ‘high risk’ is clearly associated with ‘low probability of success of the mission’. This is an improvement on the ‘scant precedent’ definition used recently within NASA, but still by no means ideal. A project to explore a little-known parameter space where the scientific return cannot be predicted in advance ought to come out as ‘high risk’ – the project might deliver little of much scientific interest.Footnote ¹⁹ And yet, the mission would still have been successful in the sense that it successfully explored the parameter space it set out to explore. What we really need is a definition sensitive to the probability that there will be few/no significant scientific returns.Footnote ²⁰ My suggestion, then, is that a project should be labelled ‘high risk’ just when the probability of a significant scientific return is significantly below average. In any future survey comparable to the ROSES-2017 survey, I recommend the following question be asked:

It would be very interesting to compare the results of a new survey, asking this question, with the ROSES-2017 survey.

Bias reduction strategies in the search for extraterrestrial life

This paper has articulated ways in which theoretical biases can and do hinder scientific progress. We need concrete, implementable strategies for minimizing such biases. The primary strategy offered here is for funding bodies to be especially sensitive to the value of funding a balanced portfolio of projects, including high-risk projects so long as:

1. (i) they are not too expensive,

2. (ii) the rewards are potentially transformative, and

3. (iii) there is not a low- or medium-risk route to the same/similar rewards.

The previous section showed the care that is necessary when we define the term ‘high risk’; most importantly, a significant percentage of high-risk projects are expected to fail, in the sense that the final scientific payoff is of limited significance compared with initial ambitions. We must learn to accept such ‘failures’; the only way to avoid them is to never fund high-risk projects.

At several points already in this paper, I have referenced Cleland (Reference Cleland2019a, Reference Clelandb) who is also motivated by the fact that we need bias reduction strategies in the search for extraterrestrial life. Her own strategy is based on two specific concerns vis-à-vis the way astrobiologists approach phenomena: (i) they are too wedded to definitions of life, and (ii) they are slow to recognize when a phenomenon is anomalous, in the sense that it strongly resists any (good) explanation ‘within a framework of widely accepted scientific beliefs’ (2019b, p. 180). Thus, she advocates that scientists employ a suite of tentative (as opposed to defining) criteria for life in order to ‘identify phenomena that resist classification as living or nonliving as worthy of further investigation for novel life’ (2019a, p. 722). The strategy is designed to ‘[expedite] the recognition that a phenomenon is anomalous’ (2019b, p. 178).

Cleland's strategy and my own are starkly different, but not in the sense that they conflict; they are fully complementary. Cleland is motivated by examples where scientists' theoretical biases bring them to misinterpret, or at least too hastily interpret, a given phenomenon; she provides a list of such examples in Cleland (Reference Cleland2019a, Section 2) and Cleland (Reference Cleland2019b, Section 8.2). What is central to all of Cleland's examples is the way scientists are too quick to fit a puzzling phenomenon into the standard theoretical framework, and are too slow to recognize that what is in front of them is a genuine anomaly, potentially requiring important theoretical adjustments. As she writes in Cleland (Reference Cleland2019b, p. 178), ‘My focus in this chapter is … on expediting the recognition that a phenomenon is anomalous,’ and all of her examples fit with that explicit focus.

Cleland's focus is different from the focus of this paper, and this serves to illustrate a very important point: theoretical biases are damaging to scientific progress in more than one way. My own focus throughout has been on cases where an unexpected phenomenon is completely missed, as in the case of the gorilla costume example, the anomalous playing card example, and the hot Jupiter example. Thus, theoretical biases hinder progress in at least these two ways:

1. (i) by blocking or delaying the discovery of unexpected phenomena;

2. (ii) by blocking or delaying the recognition that a given phenomenon is anomalous.

There is no reason to suppose that there is just one way to respond to these different barriers to progress. Cleland's strategy is more focused on the way scientists approach the concept of ‘life’, how they think about biosignatures, and the difference between a merely ‘puzzling’ phenomenon and a genuine anomaly. My strategy is more focused on funding-body decisions, and the distribution of intellectual risk across the community. It is also broader, in the sense that it is not restricted to the search for life, but concerns the funding of scientific projects more generally. Neither strategy demands a major overhaul of the status quo. But if both strategies are implemented this will be a significant step in the right direction, removing barriers to progress caused by theoretical biases, and conducive to the most surprising, and thus most significant, scientific discoveries.